hive on spark 出現錯誤,Caused by: org.apache.spark.SparkException: java.util.concurrent.ExecutionException: Exception thrown by job
Caused by: java.lang.ClassNotFoundException: org.antlr.runtime.tree.Tree
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 54 more
2023-10-20 10:21:48,263 INFO cluster.YarnClusterScheduler: Cancelling stage 5
2023-10-20 10:21:48,265 INFO cluster.YarnClusterScheduler: Killing all running tasks in stage 5: Stage cancelled
2023-10-20 10:21:48,265 INFO scheduler.DAGScheduler: ShuffleMapStage 5 (UnionRDD (Map 1 (1), Map 6 (1))) failed in 1.204 s due to Job aborted due to stage failure: Task 0 in stage 5.0 failed 4 times, most recent failure: Lost task 0.3 in stage 5.0 (TID 11, hadoop103, executor 2): UnknownReason
Driver stacktrace:
2023-10-20 10:21:48,266 ERROR client.RemoteDriver: Failed to run job ce5f418f-d59b-48c1-b3f7-96221c0c4e96
java.util.concurrent.ExecutionException: Exception thrown by job
at org.apache.spark.JavaFutureActionWrapper.getImpl(FutureAction.scala:282)
at org.apache.spark.JavaFutureActionWrapper.get(FutureAction.scala:287)
at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:382)
at org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:343)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 5.0 failed 4 times, most recent failure: Lost task 0.3 in stage 5.0 (TID 11, hadoop103, executor 2): UnknownReason
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2023)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1972)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1971)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1971)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2203)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2152)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2141)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
發現使用命令
set hive.auto.convert.join=false;
已經沒有用了,具體的代碼如下:
with customer as (
select kunnr,
adrnr,
mcod3,
name1,
sortl,
stras
from ods_kna1_full
where dt = '2023-10-19'
and is_deleted = 0
)
,customer_append01 as(
select kunnr,
dtype,
fdliy,
sonly,
case
when dtype is not null and fdliy is not null
then concat(dtype, '/', fdliy)
else dtype
end as cmtyp
from ods_zkna1_append01_full
where dt = '2023-10-19'
)
,so1 as (
select kunnr,
vwerk
from ods_knvv_full
where dt = '2023-10-19'
)
,address as (
select addrnumber,
location,
name1,
street,
house_num1
from ods_adrc_full
where dt = '2023-10-19'
)
insert overwrite table dim_it_cutm partition (dt='2023-10-19')
select
customer.kunnr,
customer.name1,
customer.mcod3,
if(addrnumber is null,customer.stras,concat(nvl(address.street,''),' ', nvl(address.house_num1,''))) stras,
customer.adrnr,
so1.vwerk,
address.location,
if( customer_append01.sonly = 'X', 'Yes', sonly) sonly,
customer_append01.cmtyp
from customer
left join customer_append01 on customer.kunnr = customer_append01.kunnr
left join so1 on customer.kunnr = so1.kunnr
left join address on customer.adrnr = address.addrnumber;片
數據量也不大,在prd環境是可以運行成功的,但是在test環境就一直報錯誤,後面想了一下 ods_knvv_full 是一張空表,要不將這張空表去除看看,發現去除之後,可以成功,這是什麼原理?????
with customer as (
select kunnr,
adrnr,
mcod3,
name1,
sortl,
stras
from ods_kna1_full
where dt = '2023-10-19'
and is_deleted = 0
)
,customer_append01 as(
select kunnr,
dtype,
fdliy,
sonly,
case
when dtype is not null and fdliy is not null
then concat(dtype, '/', fdliy)
else dtype
end as cmtyp
from ods_zkna1_append01_full
where dt = '2023-10-19'
)
,so1 as (
select kunnr,
vwerk
from ods_knvv_full
where dt = '2023-10-19'
)
,address as (
select addrnumber,
location,
name1,
street,
house_num1
from ods_adrc_full
where dt = '2023-10-19'
)
insert overwrite table dim_it_cutm partition (dt='2023-10-19')
select
customer.kunnr,
customer.name1,
customer.mcod3,
if(addrnumber is null,customer.stras,concat(nvl(address.street,''),' ', nvl(address.house_num1,''))) stras,
customer.adrnr,
-- so1.vwerk,
null,
address.location,
if( customer_append01.sonly = 'X', 'Yes', sonly) sonly,
customer_append01.cmtyp
from customer
left join customer_append01 on customer.kunnr = customer_append01.kunnr
-- left join so1 on customer.kunnr = so1.kunnr
left join address on customer.adrnr = address.addrnumber;