运行程序时出现日志和警告：WARNING: An illegal reflective access operation has occurredWARNING: Illegal

Beyond_awk

已于 2023-05-02 00:16:57 修改

阅读量2.6k

点赞数 1

文章标签：大数据 spark java scala

于 2023-05-02 00:05:58 首次发布

本文链接：https://blog.csdn.net/weixin_46780030/article/details/130460107

版权

该文指出，当使用JDK11运行Spark时可能会遇到非法反射访问警告。为消除这些警告，可以通过创建log4j.properties文件来设置日志级别为ERROR，以减少不必要的输出信息。特别地，文件中列出了针对特定Spark组件和第三方库的日志级别设置。如果警告依然存在，文章建议进一步采取措施处理。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

问题描述：

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/D:/maven/maven-repo/org/apache/spark/spark-unsafe_2.12/3.0.0/spark-unsafe_2.12-3.0.0.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

……

原因分析：

可能是因为版本过高，只要使用jdk11大多数都有此种情况，

解决方法：

①在项目的resources目录中创建log4j.properties文件

log4j.rootCategory=ERROR, console

log4j.appender.console=org.apache.log4j.ConsoleAppender

log4j.appender.console.target=System.err

log4j.appender.console.layout=org.apache.log4j.PatternLayout

log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Set the default spark-shell log level to ERROR. When running the spark-shell, the

# log level for this class is used to overwrite the root logger's log level, so that

# the user can have different defaults for the shell and regular Spark apps.

log4j.logger.org.apache.spark.repl.Main=ERROR

# Settings to quiet third party logs that are too verbose

log4j.logger.org.spark_project.jetty=ERROR

log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR

log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=ERROR

log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=ERROR

log4j.logger.org.apache.parquet=ERROR

log4j.logger.parquet=ERROR

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support

log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL

log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR