系统环境
CentOS 7.7.1908
CDH 6.3.1
问题现象
执行spark-submit,具体命令如下:
spark-submit --class xxx --master yarn --deploy-mode cluster --driver-memory 4g --executor-memory 2g --executor-cores 1 --queue thequeue /tmp/xxx.jar
console输出如下:
2020-06-25 17:59:56 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-06-25 17:59:56 INFO RMProxy:133 - Connecting to ResourceManager at master01-prod.cdh/10.0.101.86:8032
2020-06-25 17:59:56 INFO Client:57 - Requesting a new application from cluster with 3 NodeManagers
2020-06-25 17:59:56 INFO Configuration:2662 - resource-types.xml not found
2020-06-25 17:59:56 INFO ResourceUtils:419 - Unable to find 'resource-types.xml'.
2020-06-25 17:59:56 INFO Client:57 - Verifying our application has not requested more than the maximum memory capability of the cluster (13458 MB per container)
2020-06-25 17:59:56 INFO Client:57 - Will allocate AM container, with 4505 MB memory including 409 MB overhead
2020-06-25 17:59:56 INFO Client:57 - Setting up container launch context for our AM
2020-06-25 17:59:56 INFO Client:57 - Setting up the launch environment for our AM container
2020-06-25 17:59:56 INFO Client:57 - Preparing resources for our AM container
2020-06-25 17:59:56 WARN Client:69 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2020-06-25 17:59:57 INFO Client:57 - Uploading resource file:/tmp/spark-44972a1d-fb2d-443b-b6c8-d94ae7d4dc83/__spark_libs__3873912406489385752.zip -> hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0063/__spark_libs__3873912406489385752.zip
2020-06-25 17:59:58 INFO Client:57 - Uploading resource file:/tmp/dopEtlDev.jar -> hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0063/dopEtlDev.jar
2020-06-25 17:59:58 INFO Client:57 - Uploading resource file:/tmp/spark-44972a1d-fb2d-443b-b6c8-d94ae7d4dc83/__spark_conf__8206675857340680660.zip -> hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0063/__spark_conf__.zip
2020-06-25 17:59:58 INFO SecurityManager:57 - Changing view acls to: root
2020-06-25 17:59:58 INFO SecurityManager:57 - Changing modify acls to: root
2020-06-25 17:59:58 INFO SecurityManager:57 - Changing view acls groups to:
2020-06-25 17:59:58 INFO SecurityManager:57 - Changing modify acls groups to:
2020-06-25 17:59:58 INFO SecurityManager:57 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
2020-06-25 17:59:58 INFO HiveConf:188 - Found configuration file jar:file:/tmp/dopEtlDev.jar!/hive-site.xml
2020-06-25 17:59:58 INFO YARNHadoopDelegationTokenManager:57 - Attempting to load user's ticket cache.
2020-06-25 17:59:58 INFO Client:57 - Submitting application application_1591413191159_0063 to ResourceManager
2020-06-25 17:59:58 INFO YarnClientImpl:310 - Submitted application application_1591413191159_0063
2020-06-25 17:59:59 INFO Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 17:59:59 INFO Client:57 -
client token: N/A
diagnostics: AM container is launched, waiting for AM container to Register with RM
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.thequeue
start time: 1593079198932
final status: UNDEFINED
tracking URL: http://master01-prod.cdh:8088/proxy/application_1591413191159_0063/
user: root
2020-06-25 18:00:00 INFO Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:01 INFO Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:02 INFO Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:03 INFO Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:04 INFO Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:05 INFO Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:06 INFO Client:57 - Application report for application_1591413191159_0063 (state: FAILED)
2020-06-25 18:00:06 INFO Client:57 -
client token: N/A
diagnostics: Application application_1591413191159_0063 failed 2 times due to AM Container for appattempt_1591413191159_0063_000002 exited with exitCode: 13
Failing this attempt.Diagnostics: [2020-06-25 18:00:06.105]Exception from container-launch.
Container id: container_1591413191159_0063_02_000001
Exit code: 13
[2020-06-25 18:00:06.106]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data4/yarn/nm/usercache/root/filecache/57/__spark_libs__3873912406489385752.zip/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2020-06-25 18:00:06.106]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data4/yarn/nm/usercache/root/filecache/57/__spark_libs__3873912406489385752.zip/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
For more detailed output, check the application tracking page: http://master01-prod.cdh:8088/cluster/app/application_1591413191159_0063 Then click on links to logs of each attempt.
. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.thequeue
start time: 1593079198932
final status: FAILED
tracking URL: http://master01-prod.cdh:8088/cluster/app/application_1591413191159_0063
user: root
2020-06-25 18:00:06 ERROR Client:73 - Application diagnostics message: Application application_1591413191159_0063 failed 2 times due to AM Container for appattempt_1591413191159_0063_000002 exited with exitCode: 13
Failing this attempt.Diagnostics: [2020-06-25 18:00:06.105]Exception from container-launch.
Container id: container_1591413191159_0063_02_000001
Exit code: 13
[2020-06-25 18:00:06.106]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data4/yarn/nm/usercache/root/filecache/57/__spark_libs__3873912406489385752.zip/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2020-06-25 18:00:06.106]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data4/yarn/nm/usercache/root/filecache/57/__spark_libs__3873912406489385752.zip/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
For more detailed output, check the application tracking page: http://master01-prod.cdh:8088/cluster/app/application_1591413191159_0063 Then click on links to logs of each attempt.
. Failing the application.
Exception in thread "main" org.apache.spark.SparkException: Application application_1591413191159_0063 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1158)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1606)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2020-06-25 18:00:06 INFO ShutdownHookManager:57 - Shutdown hook called
2020-06-25 18:00:06 INFO ShutdownHookManager:57 - Deleting directory /tmp/spark-15956e0f-d264-4042-8968-1a3ba0b696a9
2020-06-25 18:00:06 INFO ShutdownHookManager:57 - Deleting directory /tmp/spark-44972a1d-fb2d-443b-b6c8-d94ae7d4dc83
解决方案
1.查看yarn日志
访问:http://master01-prod.cdh:19888/jobhistory/logs//slave01-prod.cdh:8041/container_1591413191159_0057_02_000001/container_1591413191159_0057_02_000001/root
具体日志如下:
Log Type: prelaunch.err
Log Upload Time: Thu Jun 25 16:40:56 +0800 2020
Log Length: 0
Log Type: prelaunch.out
Log Upload Time: Thu Jun 25 16:40:56 +0800 2020
Log Length: 70
Setting up env variables
Setting up job resources
Launching container
Log Type: stderr
Log Upload Time: Thu Jun 25 16:40:56 +0800 2020
Log Length: 544
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data2/yarn/nm/usercache/root/filecache/42/__spark_libs__2999614218847688045.zip/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Log Type: stdout
Log Upload Time: Thu Jun 25 16:40:56 +0800 2020
Log Length: 5141
Showing 4096 bytes of 5141 total. Click here for the full log.
-25 16:40:55 INFO ApplicationMaster:57 - Waiting for spark context initialization...
arg is not exist and use System currentTimeMillis!
2020-06-25 16:40:55 ERROR ApplicationMaster:94 - User class threw exception: java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:878)
at com.mskj.dopdev.scfp.dm_scfp_num_type_value$.main(dm_scfp_num_type_value.scala:34)
at com.mskj.dopdev.scfp.dm_scfp_num_type_value.main(dm_scfp_num_type_value.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
2020-06-25 16:40:55 INFO ApplicationMaster:57 - Final app status: FAILED, exitCode: 13, (reason: User class threw exception: java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:878)
at com.mskj.dopdev.scfp.dm_scfp_num_type_value$.main(dm_scfp_num_type_value.scala:34)
at com.mskj.dopdev.scfp.dm_scfp_num_type_value.main(dm_scfp_num_type_value.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
)
2020-06-25 16:40:55 ERROR ApplicationMaster:94 - Uncaught exception:
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:447)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:275)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:805)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:804)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:804)
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:878)
at com.mskj.dopdev.scfp.dm_scfp_num_type_value$.main(dm_scfp_num_type_value.scala:34)
at com.mskj.dopdev.scfp.dm_scfp_num_type_value.main(dm_scfp_num_type_value.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
2020-06-25 16:40:55 INFO ApplicationMaster:57 - Deleting staging directory hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0057
2020-06-25 16:40:55 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-06-25 16:40:55 INFO ShutdownHookManager:57 - Shutdown hook called
重点关注:reason: User class threw exception: java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found
初步分析是缺少hive相关的jar
2.检查hive相关jar包
找一台可以正常submit的服务器,对比分析发现spark所需要的jar包都在对应位置,因此排除缺少jar包问题
3.检查spark配置
正常服务器的spark配置:
异常服务器的spark配置:
这下真相大白了,spark-conf的配置不同
4.unlink spark-conf ,创建正确的软链接
5.验证问题是否解决
[root@master02-prod alternatives]# spark-submit --class xxxx --master yarn --deploy-mode cluster --driver-memory 4g --executor-memory 2g --executor-cores 1 --queue thequeue /tmp/xxxx.jar
20/06/25 18:06:55 INFO client.RMProxy: Connecting to ResourceManager at master01-prod.cdh/10.0.101.86:8032
20/06/25 18:06:55 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
20/06/25 18:06:55 INFO conf.Configuration: resource-types.xml not found
20/06/25 18:06:55 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
20/06/25 18:06:55 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (13458 MB per container)
20/06/25 18:06:55 INFO yarn.Client: Will allocate AM container, with 4505 MB memory including 409 MB overhead
20/06/25 18:06:55 INFO yarn.Client: Setting up container launch context for our AM
20/06/25 18:06:55 INFO yarn.Client: Setting up the launch environment for our AM container
20/06/25 18:06:55 INFO yarn.Client: Preparing resources for our AM container
20/06/25 18:06:55 INFO yarn.Client: Uploading resource file:/tmp/dopEtlDev.jar -> hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0064/dopEtlDev.jar
20/06/25 18:06:55 INFO yarn.Client: Uploading resource file:/tmp/spark-410edde9-c2d8-4fd4-9a12-1f7473cc89f5/__spark_conf__1869574925881857964.zip -> hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0064/__spark_conf__.zip
20/06/25 18:06:55 INFO spark.SecurityManager: Changing view acls to: root
20/06/25 18:06:55 INFO spark.SecurityManager: Changing modify acls to: root
20/06/25 18:06:55 INFO spark.SecurityManager: Changing view acls groups to:
20/06/25 18:06:55 INFO spark.SecurityManager: Changing modify acls groups to:
20/06/25 18:06:55 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
20/06/25 18:06:56 INFO conf.HiveConf: Found configuration file file:/etc/hive/conf.cloudera.hive/hive-site.xml
20/06/25 18:06:56 INFO security.YARNHadoopDelegationTokenManager: Attempting to load user's ticket cache.
20/06/25 18:06:56 INFO yarn.Client: Submitting application application_1591413191159_0064 to ResourceManager
20/06/25 18:06:56 INFO impl.YarnClientImpl: Submitted application application_1591413191159_0064
20/06/25 18:06:57 INFO yarn.Client: Application report for application_1591413191159_0064 (state: ACCEPTED)
20/06/25 18:06:57 INFO yarn.Client:
client token: N/A
diagnostics: AM container is launched, waiting for AM container to Register with RM
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.thequeue
start time: 1593079616105
final status: UNDEFINED
tracking URL: http://master01-prod.cdh:8088/proxy/application_1591413191159_0064/
user: root
20/06/25 18:06:58 INFO yarn.Client: Application report for application_1591413191159_0064 (state: ACCEPTED)
20/06/25 18:06:59 INFO yarn.Client: Application report for application_1591413191159_0064 (state: ACCEPTED)
20/06/25 18:07:00 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:00 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: slave03-prod.cdh
ApplicationMaster RPC port: 40442
queue: root.thequeue
start time: 1593079616105
final status: UNDEFINED
tracking URL: http://master01-prod.cdh:8088/proxy/application_1591413191159_0064/
user: root
20/06/25 18:07:01 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:02 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:03 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:04 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:05 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:06 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:07 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:08 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:09 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:10 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:11 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:12 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:13 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:14 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:15 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:16 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:17 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:18 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:19 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:20 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:21 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:22 INFO yarn.Client: Application report for application_1591413191159_0064 (state: FINISHED)
20/06/25 18:07:22 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: slave03-prod.cdh
ApplicationMaster RPC port: 40442
queue: root.thequeue
start time: 1593079616105
final status: SUCCEEDED
tracking URL: http://master01-prod.cdh:8088/proxy/application_1591413191159_0064/
user: root
20/06/25 18:07:22 INFO util.ShutdownHookManager: Shutdown hook called
20/06/25 18:07:22 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-410edde9-c2d8-4fd4-9a12-1f7473cc89f5
20/06/25 18:07:22 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-e847fe28-3bd5-4c74-bd1b-2b8fc7e5c4e6
参考资料
- https://java.wekeepcoding.com/article/11916051/Unable+to+instantiate+SparkSession+with+Hive+support+because+Hive+classes+are+not+found
- https://blog.csdn.net/weixin_43552807/article/details/105209467?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522159307823319725250158297%2522%252C%2522scm%2522%253A%252220140713.130102334…%2522%257D&request_id=159307823319725250158297&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2allsobaiduend~default-1-105209467.ecpm_v1_rank_business_v1&utm_term=Unable+to+instantiate+SparkSes