spark 程序配置日志级别 通过log4j

spark 如何配置日志打印级别 Log4j.properties文件

 

1、spark自身的打印级别

首先明白如何设置spark本身的日志级别

Edit your conf/log4j.properties file and change the following line:

log4j.rootCategory=INFO, console

to

log4j.rootCategory=ERROR, console

Another approach would be to :

Start spark-shell and type in the following:

import org.apache.log4j.Logger import org.apache.log4j.Level Logger.getLogger("org").setLevel(Level.OFF) Logger.getLogger("akka").setLevel(Level.OFF)

如果是在main方法里

  Logger.getLogger("org").setLevel(Level.INFO);

 

这种方式设置

2、程序自己的打印级别

实际上spark 使用log4j 来作为日志打印 所以可以配置log4j.properties文件 在里面设置一些日志打印级别 类似如下配置 具体配置可以参看log4j的文档 自定义类 可以通过获取类 自己再指定级别

 

 

log4j.rootLogger=INFO,CONSOLE,ROLLING_FILE

###################
# Console Appender
###################
log4j.appender.CONSOLE.Threshold=ERROR
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.Target=System.out
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern= [%p] %d %c - %m%n

########################
# Rolling File
########################
log4j.appender.ROLLING_FILE=org.apache.log4j.RollingFileAppender
log4j.appender.ROLLING_FILE.File=us-eng.log
log4j.appender.ROLLING_FILE.Append=true
log4j.appender.ROLLING_FILE.MaxFileSize=5000KB
log4j.appender.ROLLING_FILE.MaxBackupIndex=100
log4j.appender.ROLLING_FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.ROLLING_FILE.layout.ConversionPattern=%d{yyyy-M-d HH:mm:ss}%x[%5p](%F:%L) %m%n

 

3、spark submit 时

spark-submit \ --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=file:<file path>/log4j.xml" \ --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:<file path>/log4j.xml"

 

也可以使用log4j.properties

把log4j.properties放在一个分布式环境 比如HDFS里

Cluster模式

spark-submit \ --master yarn \ --deploy-mode cluster \ --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties" \ --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties" \ --files "/absolute/path/to/your/log4j.properties" \ --class com.github.atais.Main \ "SparkApp.jar"

 

Client模式

spark-submit \ --master yarn \ --deploy-mode cluster \ --driver-java-options "-Dlog4j.configuration=file:/absolute/path/to/your/log4j.properties \ --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties" \ --files "/absolute/path/to/your/log4j.properties" \ --class com.github.atais.Main \ "SparkApp.jar"

 

 

Notes:

1. Files uploaded to spark-cluster with --files will be available at root dir, so there is no need to add any path in file:log4j.properties.
2. Files listed in --files must be provided with absolute path!
3. file: prefix in configuration URI is mandatory.

 

 

 

 

 

 

 

参考 StackOverFlow

https://stackoverflow.com/questions/27781187/how-to-stop-info-messages-displaying-on-spark-console

https://stackoverflow.com/questions/27781187/how-to-stop-info-messages-displaying-on-spark-console?noredirect=1&lq=1