1、Cause: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
; Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask; nested exception is java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
Question:
### Cause: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
; Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask; nested exception is java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
定位:数据处理只是简单的sql
insert overwrite table dim_transaction_date select cast(rq as string) from ods_securities_tjyr
(1)、登录hive客户端,查询select *正常;select count(*) 或者带有where语句,则失败;报错如下:
Error:
Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'
(2)、set hive.execution.engine= mr; 执行count() 正常。
set hive.execution.engine= spark; 错误重现。
hive.execution.engine; 执行引擎为spark.
(3)、spark2-shell 客户端中,使用spark.sql("select count(*) from.."),执行正常,说明程序执行没有问题。需要将hive的默认执行引擎换成mr。
Solution:
CDH页面在hive配置中,搜索,hive.execution.engine,将默认引擎换成MapReduce。重启
2、Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Question:
警告: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-08-10 15:56:01.447 [job-0] ERROR JobContainer - Exception when job run
java.lang.IllegalArgumentException: Wrong FS: hdfs://cdh1:8020/user/hive/warehouse/client.