1.问题描述
桶表加载数据的时候报错Cannot run job locally: Number of reducers (= 4) is more than 1。首先创建桶表
create table emp_bu_2(
empno int,
ename string,
job string,
mgr int,
hiredate string,
sal double,
comm double,
deptno int
)CLUSTERED BY(deptno) INTO 4 BUCKETS
row format delimited fields terminated by '\t';
然后加载数据
insert overwrite table emp_bu_2 select * from emp;
报错
hive (hadoop)> insert overwrite table emp_bu_2 select * from emp;
Query ID = hadoop_20181106211010_c7fc887a-874c-42ca-943b-a922afc96c90
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 4
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Cannot run job locally: Number of reducers (= 4) is more than 1
Starting Job = job_1541343004110_0528, Tracking URL = http://hadoop:8088/proxy/application_1541343004110_0528/
Kill Command = /opt/modules/hadoop-2.6.0-cdh5.15.0/bin/hadoop job -kill job_1541343004110_0528
2.原因
reducer数量默认1,不足。
3.解决
增加reducer数量
set mapred.reduce.tasks = 4;