Job Submission failed with exception 'java.io.IOException

在hive上执行如下sql:

select a.product_id, a.merchant_id, b.category_id 
from pm_info a join product b on (a.product_id=b.id) 
where a.is_deleted=0 and a.product_is_gift=0 and a.can_sale=1 and a.can_show=1 and b.is_deleted=0 and b.PRODUCT_IS_GIFT=0;

出现错误,错误如下:

java.io.IOException: Failed to run job : org.apache.hadoop.security.AccessControlException: User pms cannot submit applications to queue root.default
	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:300)
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
	at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:448)
	at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1352)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1138)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:951)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Job Submission failed with exception 'java.io.IOException(Failed to run job : org.apache.hadoop.security.AccessControlException: User pms cannot submit applications to queue root.default)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask

原因分析:执行的SQL语句含有where条件,join连接,就会执行MapReduce任务去执行,1号店新集群的MapReduce含有两个队列,一个是root.default队列,一个是pms队列;如果不指定提交到哪个队列,hadoop就会将此Job提交到默认的root.default队列中,但是集群设定了job由MapReduce的pms队列受理,所以提交到root.default队列的任务通通被打回


解决方案(指定Job提交到哪个MapReduce的队列,在shell中执行):

hive -e "set mapred.job.queue.name=pms;select a.product_id, a.merchant_id, b.category_id 
from pm_info a join product b on (a.product_id=b.id) 
where a.is_deleted=0 and a.product_is_gift=0 and a.can_sale=1 and a.can_show=1 and b.is_deleted=0 and b.PRODUCT_IS_GIFT=0;"

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值