yarn-cluste问题

java.lang.IllegalArgumentException: System memory 238551040 must be at least 4.718592E8. Please use a larger 
heap size.

解决  --driver-memory 1g  提升driver的内存

报错Stack trace: ExitCodeException exitCode=127: ， 不知道什么错误

通过 yarn logs -applicationId application_1541057451280_0013|more 查看日志
 ERROR ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Pleas
e check earlier log output for errors. Failing the application.


解决方法：资源不能分配过大,或者没有把.setMaster("local[*]")去掉， 
所以设置driver-menory 512M
数据成功插入数据库中， 但是任务的状态仍然是失败的。


Application application_1541057451280_0017 failed 2 times due to AM Container for appattempt_1541057451280_0017_000002 exited with  exitCode: -1000
For more detailed output, check the application tracking page: http://chb0-179005:8088/cluster/app/application_1541057451280_0017 Then click on links to logs of each attempt.
Diagnostics: File does not exist: hdfs://chb0-179004:8020/user/root/.sparkStaging/application_1541057451280_0017/__spark_conf__1430427675917075383.zip
java.io.FileNotFoundException: File does not exist: hdfs://chb0-179004:8020/user/root/.sparkStaging/application_1541057451280_0017/__spark_conf__1430427675917075383.zip
参考https://bbs.csdn.net/topics/392169982
设置spark.yarn.jars  




spark.yarn.driver.memoryOverhead 
spark.yarn.executor.memoryOverhead 





spark.executor.extraClassPath=hdfs://chb0-179004/spark_libs/*.jar
spark.driver.extraClassPath=hdfs://chb0-179004/spark_libs/*.jar


spark.executor.extraLibraryPath {{spark_hadoop_lib_native}}:hdfs://chb0-179004:8020/apps/spark/source/lib/*
spark.driver.extraLibraryPath {{spark_hadoop_lib_native}}:hdfs://chb0-179004:8020/apps/spark/source/lib/*

spark.yarn.jar 
spark.yarn.jars hdfs://chb0-179004/spark_libs/*.jar
spark.yarn.archive



18/11/03 14:49:10 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (3840 M
B per container)

发现 memoryStore分配比较多
18/11/03 15:36:45 INFO MemoryStore: MemoryStore started with capacity 1259.8 MB

18/11/03 15:36:47 INFO BlockManagerMasterEndpoint: Registering block manager localhost:46292 with 1259.8 MB RAM, BlockManagerId(driver, l
ocalhost, 46292)

18/11/03 15:36:53 ERROR ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output f
or errors. Failing the application.