往hive中的表插入数据的时候,不能直接运行成功,报一下错误
错误原因:
第一种,namenode内存空间不够
原因:JVM剩余内存空间不够新job运行所致
错误提示
0: jdbc:hive2://hadoop101:10000> insert into table student values(1002,"zss");
INFO : Compiling command(queryId=root_20210909172055_08738d9c-4ded-4067-a2ac-6a64572ad49b): insert into table student values(1002,"zss")
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Semantic Analysis Completed (retrial = false)
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:col1, type:int, comment:null), FieldSchema(name:col2, type:string, comment:null)], properties:null)
INFO : Completed compiling command(queryId=root_20210909172055_08738d9c-4ded-4067-a2ac-6a64572ad49b); Time taken: 0.425 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=root_20210909172055_08738d9c-4ded-4067-a2ac-6a64572ad49b): insert into table student values(1002,"zss")
WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
INFO : Query ID = root_20210909172055_08738d9c-4ded-4067-a2ac-6a64572ad49b
INFO : Total jobs = 3
INFO : Launching Job 1 out of 3
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Number of reduce tasks determined at compile time: 1
INFO : In order to change the average load for a reducer (in bytes):
INFO : set hive.exec.reducers.bytes.per.reducer=<number>
INFO : In order to limit the maximum number of reducers:
INFO : set hive.exec.reducers.max=<number>
INFO : In order to set a constant number of reducers:
INFO : set mapreduce.job.reduces=<number>
INFO : number of splits:1
INFO : Submitting tokens for job: job_1631178391888_0002
INFO : Executing with tokens: []
INFO : The url to track the job: http://hadoop101:8088/proxy/application_1631178391888_0002/
INFO : Starting Job = job_1631178391888_0002, Tracking URL = http://hadoop101:8088/proxy/application_1631178391888_0002/
INFO : Kill Command = /opt/bagdata/hadoop-3.1.3/bin/mapred job -kill job_1631178391888_0002
INFO : Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
INFO : 2021-09-09 17:21:13,094 Stage-1 map = 0%, reduce = 0%
INFO : 2021-09-09 17:21:53,093 Stage-1 map = 100%, reduce = 100%
ERROR : Ended Job = job_1631178391888_0002 with errors
ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
INFO : MapReduce Jobs Launched:
INFO : Stage-Stage-1: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
INFO : Total MapReduce CPU Time Spent: 0 msec
INFO : Completed executing command(queryId=root_20210909172055_08738d9c-4ded-4067-a2ac-6a64572ad49b); Time taken: 59.838 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2
解决办法1:
set hive.exec.mode.local.auto=true;
错误原因
第二种,yarn资源不足
原因:
该错误是YARN的虚拟内存计算方式导致,上例中用户程序申请的内存为1Gb,YARN根据此值乘以一个比例(默认为2.1)得出申请的虚拟内存的值,当YARN计算的用户程序所需虚拟内存值大于计算出来的值时,就会报出以上错误。调节比例值可以解决该问题。具体参数为:yarn-site.xml中的yarn.nodemanager.vmem-pmem-ratio
解决方法:
调整hadoop配置文件yarn-site.xml中值:
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>2048</value>
<description>default value is 1024</description>
</property>
增加yarn.scheduler.minimum-allocation-mb 数量,从缺省1024改为2048;上述运行问题即刻得到解决