Spark on Yarn 自定义日志打印及级别

spark 自定义日志打印及级别

在处理spark任务的时候，我们需要经常查看日志，那么日志的级别对我们来说就很重要。但有时候，我们还需要输出一些业务日志，那它就会被spark的大量日志给覆盖掉，但调日志级别，又会使查找问题变的比较困难，今天，我们来解决下这个问题。

日志打印定义

我们首先需要日志配置文件，并在任务提交的脚本中设置成我们自定义的日志文件。这里将分别对两种日志进行配置对比，分别是java自带日志和spark内部日志。

1. 自定义日志文件 ，也可以将spark配置文件中的log4j文件拷贝下并做修改。

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# Set everything to be logged to the console
# DEBUG,INFO,WARN,FATAL,ERROR
log4j.rootLogger=INFO, console
#log4j.rootCategory=INFO
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{
   yy/MM/dd HH:mm:ss} %p %c{
   1}: %m%n


log4j.logger.javaLog=WARN, javaLog
#log4j.logger.javaLog.access=WARN
log4j.appender.javaLog.errorhandler.root-ref=false
log4j.appender.javaLog=org.apache.log4j.ConsoleAppender
log4j.appender.javaLog.target