执行Hive sql时报错：
执行语句和报错详情如下：
hive (thnews_new)>
> insert into table thnews_new.thnews_all
> select date_id.id,date_id.source,date_id.sendTime,date_id.descripe,date_id.subject,date_id.type,context_by_id.context
> from thnews_new.context_by_id as context_by_id join thnews_new.date_id as date_id
> on date_id.id = context_by_id.id;
Automatically selecting local only mode for query
Query ID = root_20190710125148_fd34f996-1eb8-40c3-a3a5-7e04c6edad76
Total jobs = 1
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/soft/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/soft/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
19/07/10 12:51:57 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
19/07/10 12:51:57 WARN conf.Configuration: file:/tmp/root/e96efa4b-a506-4956-8bda-ca74b326b128/hive_2019-07-10_12-51-48_070_3371426343275360779-1/-local-10005/jobconf.xml:an attempt to override final parameter: mapreduce.framework.name; Ignoring.
Execution log at: /tmp/root/root_20190710125148_fd34f996-1eb8-40c3-a3a5-7e04c6edad76.log
2019-07-10 12:51:59 Starting to launch local task to process map join; maximum memory = 518979584
2019-07-10 12:52:03 Dump the side-table for tag: 0 with group count: 39 into file: file:/tmp/root/e96efa4b-a506-4956-8bda-ca74b326b128/hive_2019-07-10_12-51-48_070_3371426343275360779-1/-local-10002/HashTable-Stage-4/MapJoin-mapfile60–.hashtable
2019-07-10 12:52:03 Uploaded 1 File to: file:/tmp/root/e96efa4b-a506-4956-8bda-ca74b326b128/hive_2019-07-10_12-51-48_070_3371426343275360779-1/-local-10002/HashTable-Stage-4/MapJoin-mapfile60–.hashtable (105284 bytes)
2019-07-10 12:52:03 End of local task; Time Taken: 4.079 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there’s no reduce operator
19/07/10 12:52:04 [main]: ERROR mr.ExecDriver: local
Job running in-process (local Hadoop)
19/07/10 12:52:04 [LocalJobRunner Map Task Executor #0]: ERROR exec.MapJoinOperator: Unexpected exception: null
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.getRefKey(MapJoinOperator.java:327)
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:339)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:122)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
at org.apache.hadoop.hive.ql.exec.MapOperator
M
a
p
O
p
C
t
x
.
f
o
r
w
a
r
d
(
M
a
p
O
p
e
r
a
t
o
r
.
j
a
v
a
:
162
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
i
v
e
.
q
l
.
e
x
e
c
.
M
a
p
O
p
e
r
a
t
o
r
.
p
r
o
c
e
s
s
(
M
a
p
O
p
e
r
a
t
o
r
.
j
a
v
a
:
508
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
i
v
e
.
q
l
.
e
x
e
c
.
m
r
.
E
x
e
c
M
a
p
p
e
r
.
m
a
p
(
E
x
e
c
M
a
p
p
e
r
.
j
a
v
a
:
163
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
m
a
p
r
e
d
.
M
a
p
R
u
n
n
e
r
.
r
u
n
(
M
a
p
R
u
n
n
e
r
.
j
a
v
a
:
54
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
m
a
p
r
e
d
.
M
a
p
T
a
s
k
.
r
u
n
O
l
d
M
a
p
p
e
r
(
M
a
p
T
a
s
k
.
j
a
v
a
:
453
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
m
a
p
r
e
d
.
M
a
p
T
a
s
k
.
r
u
n
(
M
a
p
T
a
s
k
.
j
a
v
a
:
343
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
m
a
p
r
e
d
.
L
o
c
a
l
J
o
b
R
u
n
n
e
r
MapOpCtx.forward(MapOperator.java:162) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.LocalJobRunner
MapOpCtx.forward(MapOperator.java:162)atorg.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508)atorg.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)atorg.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)atorg.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)atorg.apache.hadoop.mapred.MapTask.run(MapTask.java:343)atorg.apache.hadoop.mapred.LocalJobRunnerJob
M
a
p
T
a
s
k
R
u
n
n
a
b
l
e
.
r
u
n
(
L
o
c
a
l
J
o
b
R
u
n
n
e
r
.
j
a
v
a
:
243
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
E
x
e
c
u
t
o
r
s
MapTaskRunnable.run(LocalJobRunner.java:243) at java.util.concurrent.Executors
MapTaskRunnable.run(LocalJobRunner.java:243)atjava.util.concurrent.ExecutorsRunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutorKaTeX parse error: Expected 'EOF', got '#' at position 135: … Task Executor #̲0]: FATAL mr.Ex…Job
M
a
p
T
a
s
k
R
u
n
n
a
b
l
e
.
r
u
n
(
L
o
c
a
l
J
o
b
R
u
n
n
e
r
.
j
a
v
a
:
243
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
E
x
e
c
u
t
o
r
s
MapTaskRunnable.run(LocalJobRunner.java:243) at java.util.concurrent.Executors
MapTaskRunnable.run(LocalJobRunner.java:243)atjava.util.concurrent.ExecutorsRunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor
W
o
r
k
e
r
.
r
u
n
(
T
h
r
e
a
d
P
o
o
l
E
x
e
c
u
t
o
r
.
j
a
v
a
:
617
)
a
t
j
a
v
a
.
l
a
n
g
.
T
h
r
e
a
d
.
r
u
n
(
T
h
r
e
a
d
.
j
a
v
a
:
745
)
C
a
u
s
e
d
b
y
:
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
i
v
e
.
q
l
.
m
e
t
a
d
a
t
a
.
H
i
v
e
E
x
c
e
p
t
i
o
n
:
U
n
e
x
p
e
c
t
e
d
e
x
c
e
p
t
i
o
n
:
n
u
l
l
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
i
v
e
.
q
l
.
e
x
e
c
.
M
a
p
J
o
i
n
O
p
e
r
a
t
o
r
.
p
r
o
c
e
s
s
(
M
a
p
J
o
i
n
O
p
e
r
a
t
o
r
.
j
a
v
a
:
426
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
i
v
e
.
q
l
.
e
x
e
c
.
O
p
e
r
a
t
o
r
.
f
o
r
w
a
r
d
(
O
p
e
r
a
t
o
r
.
j
a
v
a
:
837
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
i
v
e
.
q
l
.
e
x
e
c
.
F
i
l
t
e
r
O
p
e
r
a
t
o
r
.
p
r
o
c
e
s
s
(
F
i
l
t
e
r
O
p
e
r
a
t
o
r
.
j
a
v
a
:
122
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
i
v
e
.
q
l
.
e
x
e
c
.
O
p
e
r
a
t
o
r
.
f
o
r
w
a
r
d
(
O
p
e
r
a
t
o
r
.
j
a
v
a
:
837
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
i
v
e
.
q
l
.
e
x
e
c
.
T
a
b
l
e
S
c
a
n
O
p
e
r
a
t
o
r
.
p
r
o
c
e
s
s
(
T
a
b
l
e
S
c
a
n
O
p
e
r
a
t
o
r
.
j
a
v
a
:
97
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
i
v
e
.
q
l
.
e
x
e
c
.
M
a
p
O
p
e
r
a
t
o
r
Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception: null at org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:426) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:122) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97) at org.apache.hadoop.hive.ql.exec.MapOperator
Worker.run(ThreadPoolExecutor.java:617)atjava.lang.Thread.run(Thread.java:745)Causedby:org.apache.hadoop.hive.ql.metadata.HiveException:Unexpectedexception:nullatorg.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:426)atorg.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)atorg.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:122)atorg.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)atorg.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)atorg.apache.hadoop.hive.ql.exec.MapOperatorMapOpCtx.forward(MapOperator.java:162)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508)
… 10 more
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.getRefKey(MapJoinOperator.java:327)
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:339)
… 16 more
2019-07-10 12:52:05,879 Stage-4 map = 0%, reduce = 0%
Ended Job = job_local1546372873_0008 with errors
19/07/10 12:52:05 [main]: ERROR exec.Task: Ended Job = job_local1546372873_0008 with errors
Error during job, obtaining debugging information…
19/07/10 12:52:05 [Thread-296]: ERROR exec.Task: Error during job, obtaining debugging information…
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
19/07/10 12:52:05 [main]: ERROR ql.Driver: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-4: HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
★解决方法：
set hive.auto.convert.join=false;
set hive.vectorized.execution.enabled=false;
set hive.vectorized.execution.reduce.enabled=false;
●set hive.vectorized.execution.enabled=false;
默认关闭，开启后，每次处理数据时会将1024行数据组成一个batch进行处理，而不是一行一行进行处理，这样能够显著提高执行速度。
★利用矢量化（ Vectorization）
矢量化（Vectorization） 通过在一次操作中提取t 1024 行数据提升性能，而不是一次只取一条。它提升了像过滤, 联合, 聚合等等操作的性能。
Vectorization 可以通过在环境中执行如下命令而得到启用。
set hive.vectorized.execution.enabled=true;
set hive.vectorized.execution.reduce.enabled=true;
2万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
