一、spark job日志介绍
spark中提供了log4j的方式记录日志。可以在$SPARK_HOME/conf/下,将 log4j.properties.template 文件copy为 log4j.properties 来启用log4j配置。但这个配置为全局配置,不能单独配置某个job的运行日志。在Spark的conf目录下,把log4j.properties.template修改为log4j.properties,原来的内容如下:(log4j的用法和配置,请参考另一篇文档)
# Set everything to be logged to the console
log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN
# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
二、spark job独立配置的log4j的方法
现在我们介绍spark (streaming) job独立配置的log4j的方法,通过查看官方文档,要为应用主程序(即driver端)或执行程序使(即executor端)自定义log4j配置,需要两步就完成了,下面具体说明。第一步:上传自定义 log4j-driver.properties和log4j-executor.properties
第二步:添加-Dlog4j的配置。使用 --conf参数。
用于驱动程序:spark.driver.extraJavaOptions = -Dlog4j.configuration = <配置文件的位置>
用于执行者:spark.executor.extraJavaOptions= -Dlog4j.configuration = <配置文件的位置>
注意:driver端还可以使用spark-submit的--driver-java-options参数去配置。
方案一:使用 spark-submit的 --files 参数将自定义的配置文件上传到应用程序的文件列表中。
spark-submit--class com.hm.spark.Application
--master yarn --deploy-mode cluster
--driver-cores 1 --driver-memory 1G
--num-executors 2 --executor-cores 1 --executor-memory 1G
--driver-java-options "-Dlog4j.configuration=log4j-driver.properties"
--conf spark.executor.extraJavaOptions="-Dlog4j.configuration=log4j-executor.properties"
--files /home/hadoop/spark-workspace/log4j-driver.properties,/home/hadoop/spark-workspace/log4j-executor.properties
/home/hadoop/spark-workspace/my-spark-etl-assembly-1.0-SNAPSHOT.jar
注意,这里我没有使用spark.driver.extraJavaOptions参数去配置,而是使用spark-submit的--driver-java-options参数进行设置的。方案二:不使用 spark-submit的 --files 参数上传文件,直接使用文件。
spark-submit
--class com.hm.spark.Application
--master yarn --deploy-mode cluster
--driver-cores 1 --driver-memory 1G
--num-executors 2 --executor-cores 1 --executor-memory 1G
--driver-java-options "-Dlog4j.configuration=file:/home/hadoop/spark-workspace/log4j-driver.properties "
--conf spark.executor.extraJavaOptions="-Dlog4j.configuration=file:/home/hadoop/spark-workspace/log4j-executor.properties"
/home/hadoop/spark-workspace/my-spark-etl-assembly-1.0-SNAPSHOT.jar
注意:如果使用文件, file: 则应明确提供配置文件的,并且文件需要在所有节点上本地存在。
在shell脚本中写的格式
#!/bin/bash
base_path=/data/service/spark-workspace
version=1.0-SNAPSHOT
jar_name=spark-etl_2.11-${version}-assembly.jar
class_name=com.spark.etl.Application
# client or cluster
deploy_mode=cluster
cmd="spark-submit --class ${class_name} --master yarn --deploy-mode ${deploy_mode} --driver-cores 1 --driver-memory 2G --num-executors 4 --executor-cores 2 --executor-memory 4G --driver-java-options \"-Dlog4j.configuration=log4j-driver.properties\" --conf spark.executor.extraJavaOptions=\"-Dlog4j.configuration=log4j-executor.properties\" --files ${base_path}/log4j-executor.properties,${base_path}/log4j-dri