spark运行错误解决

最新推荐文章于 2024-04-15 20:35:30 发布

24koby

最新推荐文章于 2024-04-15 20:35:30 发布

阅读量416

点赞数

分类专栏： spark

spark 专栏收录该内容

15 篇文章 1 订阅

订阅专栏

问题：”java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
解决；Vmoptions中添加-Dspark.testing.memory=1073741824

问题：用spark连接mysql出现Exception in thread “main” java.lang.ClassNotFoundException: com.mysql.jdbc.Driver

解决：idea中resource中引入mysql驱动包并use as library添加path中

问题：java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.InputSplit.getLocationInfo()[Lorg/apache/hadoop/mapred/SplitLocationInfo;
由于缺少下面的依赖包

<dependency>
	<groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-mapreduce-client-core</artifactId>
    <version>2.7.3</version>
<dependency>

Error:(48, 45) value toDS is not a member of Seq[Person]
val caseClassDS = Seq(Person(“asd”,43)).toDS()错误
将case class Person()移到main方法外