spark on yarn 的一些问题、hbase关闭问题

spark on yarn 报错:

• When running with master 'yarn-client' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.

在spark-env.sh中  export  HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

•  Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.

如果spark.yarn.archive 或者 spark.yarn.jars这两个参数都没有配置，spark就会把$SPARK_HOME/jars/所有的jar上传到分布式缓存中，造成提交缓慢。

先将spark的jar上传到hdfs,在spark的conf的spark-default.conf 
添加配置 spark.yarn.archive=hdfs://ns/spark/jars

• 运行spark-submit提交job到yarn,一直不停地Accepted不报错:

19/02/20 02:17:39 INFO yarn.Client: Application report for application_1550627305102_0006 (state: ACCEPTED)
19/02/20 02:17:40 INFO yarn.Client: Application report for application_1550627305102_0006 (state: ACCEPTED)
19/02/20 02:17:41 INFO yarn.Client: Application report for application_1550627305102_0006 (state: ACCEPTED)
19/02/20 02:17:42 INFO yarn.Client: Application report for application_1550627305102_0006 (state: ACCEPTED)
19/02/20 02:17:43 INFO yarn.Client: Application report for application_1550627305102_0006 (state: ACCEPTED)
19/02/20 02:17:44 INFO yarn.Client: Application report for application_1550627305102_0006 (state: ACCEPTED)
19/02/20 02:17:45 INFO yarn.Client: Application report for application_1550627305102_0006 (state: ACCEPTED)
19/02/20 02:17:46 INFO yarn.Client: Application report for application_1550627305102_0006 (state: ACCEPTED)
19/02/20 02:17:47 INFO yarn.Client: Application report for application_1550627305102_0006 (state: ACCEPTED)
 

 yarn资源分配问题.

• 运行spark程序总是看到很多INFO的日志信息,可以修改spark的日志配置文件log4j.properties,(学习不建议修改,这里面还是能看到很多有用信息的)

# Set everything to be logged to the console
log4j.rootCategory=INFO, console   将这行里的INFO改为WARN.

log4j.rootCategory=WARN, console

 

hbase关闭问题:

• stopping hbasecat: /tmp/hbase-mango-master.pid: No such file or directory  

 原因是，默认情况下pid文件保存在/tmp目录下，/tmp目录下的文件很容易丢失（重启后基本就会删除），

解决办法：在hbase-env.sh中修改pid文件的存放路径
# The directory where pid files are stored. /tmp by default.  
 export HBASE_PID_DIR=/var/hadoop/pids 
 

• 执行stop-hbase.sh 后，一直出现点点点点......长时间无法关闭

尝试手动强制关闭regionserver，

hbase-daemons.sh stop regionserver

hbase-daemons.sh stop master

stop-hbase.sh