spark错误日志查看方法

最新推荐文章于 2024-06-26 11:02:49 发布

dataastron

最新推荐文章于 2024-06-26 11:02:49 发布

阅读量5.3k

点赞数

分类专栏： Spark 文章标签： spark yarn

本文链接：https://blog.csdn.net/dataastron/article/details/78965573

版权

本文介绍了如何通过YARN查看Spark应用的错误日志，具体操作是使用`yarn logs -applicationId <application_id>`命令，例如：yarn logs -applicationId application_1512098667219_5898489，这对于理解和解决Spark运行时遇到的问题至关重要。

摘要由CSDN通过智能技术生成

查看错误日志yarn logs -applicationId application_1512098667219_5898489

yarn logs -applicationId xxx

执行方式

spark-submit  \
--master  yarn-cluster \
--class com.jd.astron.jrdm.ClusterTest \
--num-executors 1  \
--executor-memory 1g  \
--driver-memory 1g   \
--executor-cores 1 \
--queue bdp_jmart_astron_union.bdp_jmart_astron_formal \
/home/mart_astron/data_dir/astron/sparktest/jrdm_astron_test-1.0-SNAPSHOT.jar

出错日志

[mart_astron@BJLFRZ-Client-9-199 sparktest]$ spark-submit  --master  yarn-cluster --class com.astron.astron.jrdm.ClusterTest --num-executors 1  --executor-memory 1g  --driver-memory 1g   --executor-cores 1 --queue bdp_jmart_astron_union.bdp_jmart_astron_formal /home/mart_astron/data_dir/shida6/sparktest/jrdm_astron_test-1.0-SNAPSHOT.jar
Warning: Master yarn-cluster is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead.
18/01/03 21:07:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/01/03 21:07:06 INFO ConfiguredRMFailoverProxyProvider: Failing over to rm2
18/01/03 21:07:06 INFO Client: Requesting a new application from cluster with 4145 NodeManagers
18/01/03 21:07:06 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (53248 MB per container)
18/01/03 21:07:06 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
18/01/03 21:07:06 INFO Client: Setting up container launch context for our AM
18/01/03 21:07:06 INFO Client: Setting up the launch environment for our AM container
18/01/03 21:07:06 INFO Client: Preparing resources for our AM container
18/01/03 21:07:07 INFO Client: Uploading resource hdfs://ns2/user/spark/lib/spark-2.1.0-hadoop-2.7.1.zip -> hdfs://ns12/user/mart_astron/.sparkStaging/application_1512098667219_5899268/spark-2.1.0-hadoop-2.7.1.zip
18/01/03 21:07:08 INFO Client: Uploading resource file:/home/mart_astron/data_dir/shida6/sparktest/jrdm_astron_test-1.0-SNAPSHOT.jar -> hdfs://ns12/user/mart_astron/.sparkStaging/application_1512098667219_5899268/jrdm_astron_test-1.0-SNAPSHOT.jar
18/01/03 21:07:08 INFO Client: Uploading resource file:/data0/tmp/spark-46f0c288-8591-49eb-8273-7140b3f3e9c5/__spark_conf__7338282222566816143.zip -> hdfs://ns12/user/mart_astron/.sparkStaging/application_1512098667219_5899268/__spark_conf__.zip
18/01/03 21:07:08 INFO Client: ===============================================================================
18/01/03 21:07:08 INFO Client: YARN AM launch context:
18/01/03 21:07:08 INFO Client:     user class: com.astron.astron.jrdm.ClusterTest
18/01/03 21:07:08 INFO Client:     env:
18/01/03 21:07:08 INFO Client:         CLASSPATH -> {
  {PWD}}<CPS>{
  {PWD}}/__spark_conf__<CPS>{
  {PWD}}/__spark_libs__/*<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_H