hive中创建子表并插入数据过程初始化MR报错解决方法

最新推荐文章于 2024-04-23 18:52:22 发布
最新推荐文章于 2024-04-23 18:52:22 发布
阅读量1.1k 收藏
点赞数
分类专栏： java Nginx 文章标签： java Nginx Hive
java 同时被 2 个专栏收录
203 篇文章 1 订阅
订阅专栏
Nginx
32 篇文章 0 订阅
订阅专栏

本文继成上一篇通过hive分析nginx日志文章，详情参考下面链接：

http://www.cnblogs.com/wcwen1990/p/7066230.html

 

接着来：

 

创建业务子表：

drop table if exists chavin.nginx_access_log_comm;
create table if not exists chavin.nginx_access_log_comm(
host STRING,
time STRING,
request STRING,
referer STRING
)
row format delimited fields terminated by '\t'
STORED AS orc tblproperties("orc.compress"="SNAPPY");
insert into chavin.nginx_access_log_comm select host,time,request,referer from chavin.nginx_access_log;

结果报错如下：

hive (default)> insert into chavin.nginx_access_log_comm select host,time,request,referer from chavin.nginx_access_log;
Query ID = root_20170623203838_35656104-9f46-4801-9dbd-1d5685de8187
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1498218985196_0009, Tracking URL = http://db01:8088/proxy/application_1498218985196_0009/
Kill Command = /opt/cloudera/parcels/CDH-5.9.2-1.cdh5.9.2.p0.3/lib/hadoop/bin/hadoop job  -kill job_1498218985196_0009
Hadoop job information for Stage-1: number of mappers: 4; number of reducers: 0
2017-06-23 20:38:10,702 Stage-1 map = 0%,  reduce = 0%
2017-06-23 20:38:32,584 Stage-1 map = 100%,  reduce = 0%
Ended Job = job_1498218985196_0009 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1498218985196_0009_m_000002 (and more) from job job_1498218985196_0009

Task with the most failures(4): 
-----
Task ID:
   task_1498218985196_0009_m_000002

URL:
   http://db01:8088/taskdetails.jsp?jobid=job_1498218985196_0009&tipid=task_1498218985196_0009_m_000002
-----
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: Error in configuring object
     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:449)
     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:415)
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1912)
     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:606)
     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
     ... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
     at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
     at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
     at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
     ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:606)
     at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
     ... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
     at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:147)
     ... 22 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.contrib.serde2.RegexSerDe not found
     at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:323)
     at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:333)
     at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:116)
     ... 22 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.contrib.serde2.RegexSerDe not found
     at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
     at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:140)
     at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:297)
     ... 24 more


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 4   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
hive (default)>

从日志看，貌似执行mr因为缺少包而导致的故障。
解决办法：向hive命令行中添加hive-contrib-1.1.0-cdh5.9.2.jar包：

hive (chavin)> add jar /opt/cloudera/parcels/CDH/lib/hive/lib/hive-contrib-1.1.0-cdh5.9.2.jar;
Added [/opt/cloudera/parcels/CDH/lib/hive/lib/hive-contrib-1.1.0-cdh5.9.2.jar] to class path
Added resources: [/opt/cloudera/parcels/CDH/lib/hive/lib/hive-contrib-1.1.0-cdh5.9.2.jar]

再次运行加载数据命令：

hive (chavin)>insert into chavin.nginx_access_log_comm select host,time,request,referer from chavin.nginx_access_log;
Query ID = root_20170623203333_d244452b-9fae-4131-826a-428620219dbe
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1498218985196_0007, Tracking URL = http://db01:8088/proxy/application_1498218985196_0007/
Kill Command = /opt/cloudera/parcels/CDH-5.9.2-1.cdh5.9.2.p0.3/lib/hadoop/bin/hadoop job  -kill job_1498218985196_0007
Hadoop job information for Stage-1: number of mappers: 4; number of reducers: 0
2017-06-23 20:33:59,442 Stage-1 map = 0%,  reduce = 0%
2017-06-23 20:34:19,200 Stage-1 map = 13%,  reduce = 0%, Cumulative CPU 50.65 sec
2017-06-23 20:34:20,239 Stage-1 map = 25%,  reduce = 0%, Cumulative CPU 53.72 sec
2017-06-23 20:34:21,333 Stage-1 map = 42%,  reduce = 0%, Cumulative CPU 56.75 sec
2017-06-23 20:34:22,368 Stage-1 map = 55%,  reduce = 0%, Cumulative CPU 62.82 sec
2017-06-23 20:34:24,451 Stage-1 map = 63%,  reduce = 0%, Cumulative CPU 68.45 sec
2017-06-23 20:34:28,984 Stage-1 map = 88%,  reduce = 0%, Cumulative CPU 88.58 sec
2017-06-23 20:34:31,049 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 91.97 sec
MapReduce Total cumulative CPU time: 1 minutes 31 seconds 970 msec
Ended Job = job_1498218985196_0007
Stage-4 is filtered out by condition resolver.
Stage-3 is selected by condition resolver.
Stage-5 is filtered out by condition resolver.
Starting Job = job_1498218985196_0008, Tracking URL = http://db01:8088/proxy/application_1498218985196_0008/
Kill Command = /opt/cloudera/parcels/CDH-5.9.2-1.cdh5.9.2.p0.3/lib/hadoop/bin/hadoop job  -kill job_1498218985196_0008
Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 0
2017-06-23 20:34:38,770 Stage-3 map = 0%,  reduce = 0%
2017-06-23 20:34:44,973 Stage-3 map = 100%,  reduce = 0%, Cumulative CPU 1.46 sec
MapReduce Total cumulative CPU time: 1 seconds 460 msec
Ended Job = job_1498218985196_0008
Loading data to table chavin.nginx_access_log_comm
Table chavin.nginx_access_log_comm stats: [numFiles=1, numRows=2538166, totalSize=6856514, rawDataSize=1598858202]
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 4   Cumulative CPU: 91.97 sec   HDFS Read: 1000717778 HDFS Write: 6859275 SUCCESS
Stage-Stage-3: Map: 1   Cumulative CPU: 1.46 sec   HDFS Read: 6989009 HDFS Write: 6856514 SUCCESS
Total MapReduce CPU Time Spent: 1 minutes 33 seconds 430 msec
OK
host    time    request    referer
Time taken: 55.218 seconds

问题解决，但是该解决方法只能对本次Hive会话有用，Hive使用命令exit退出后再进入依旧会出现该问题。

mojolang
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
spark或mr引擎插入的数据，hive表查询数据为0
02-28
spark或mr引擎插入的数据，hive表查询数据为0
执行mr时出错,卡住了Number of reduce tasks is set to 0 since there‘s no reduce operator
weixin_46321794的博客
07-25 6849
在hive中执行插入的hql语句后: hive> insert overwrite table t_user > select myLog(data,'en'),myLog(data,'name'),myLog(data,"password") from tmp; mr运行许久未响应: 经检查发现是没有启动yarn…启动yarn后问题解决
Hive在执行插入数据等job任务出错
fengjinghong的博客
05-30 5784
1、完全分布式中hive的执行模式 参考博客 问题描述： Hive在执行job任务时候卡死并报出信息：Number of reduce tasks is set to 0 since there’s no reduce operator（翻译：由于没有减少操作符，减少任务数被设置为0），如图： 这样的情况下，Hive 已经无法继续执行，只能退出重启hive(但是依然不能执行)。这是因为虽然输入文件很小，但是需要加载的文件很大，根本吃不消。 解决问题： Hive支持任务执行选择本地模式(local mode
hive的一些报错及解决方法
12-18
hive的一些报错及解决方法：连接报错、执行脚本卡住、内存溢出报错、表死锁问题
配置hive元数据到Mysql中的全过程记录
01-19
在hive的安装目录下，进入conf目录，创建一个hive-site.xml文件 根据官方文档配置参数，拷贝数据到hive-site.xml文件中 https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin 注意：先创建一个metastore数据库，字符集要用latin1 将mysql驱动复制到hive的lib目录里。（这有关上一篇安装MySQL的内容） 解压驱动包 将mysql驱动复制到hive的lib目录里 启动hive（别忘了先启动hdfs和yarn~~） start-dfs.s
创建动态分区时Number of reduce tasks is set to 0 since there‘s no reduce operator
werewolf2的博客
10-19 9752
创建动态分区时卡住不动，显示Number of reduce tasks is set to 0 since there‘s no reduce operator 原因：yarn集群的资源管理器没有启动成功 解决：start-yarn.sh 插入动态分区时：contains non-partition columns (state=42000,code=40000) 原因：应该是语句写错了 解决：重写了分区语句又对了 ...
hive生产实践问题（一）在使用Hive Client跑job时，一直提示job被kill，
xuxu96
03-31 3758
然后观察YARN的WebUI进行查看，如图： 然后观察Hive Client的控制台输出，如下： Launching Job 1 out of 3 Number of reduce tasks is set to 0 since there’s no reduce operator Starting Job = job_1552895066408_0001, Tracking URL = htt...
Hive基本操作
zhou920786312的博客
03-16 996
Hive基本操作 建表语法 CREATE [EXTERNAL] TABLE [IF NOT EXISTS] table_name [(col_name data_type [COMMENT col_comment], ...)] [COMMENT table_comment] [PARTITIONED BY (col_name data_type [COMMENT c...
Hive：简单查询不启用Mapreduce job而启用Fetch task
An342647823的专栏
03-14 2899
转载：http://blog.csdn.net/wypblog/article/details/17410315 谢谢分享。 如果你想查询某个表的某一列，Hive默认是会启用MapReduce Job来完成这个任务，如下： 01 02 03 04 05 06 07 08 09 10 11 12
hive2.1.1中orc格式读取报数组越界错误解决方法
05-24
hive2.1.1orc格式读取报数组越界错误，替换jar包。hive-exec-2.1.1-cdh6.3.2.jar、hive-orc-2.1.1-cdh6.3.2.jar。分发各个服务器即可。
hive on spark mr 数据开发常见问题解决
11-20
hive工作常见问题解决收集开发人员在Hive日常开发过程中难免遇到各种各样的hive报错，这些报错信息很多时间并没有形成汇总的知识库，每次遇到问题都会重复查资料，效率非常低 现在总结一些常见的知识库，方便大家查看并分析： 最近工作的公司，我们常常见的错误 有 shuffle in fetcher#6 at GC overhead limit exceeded Unknown primitive type VOID Output column number expected to be 0 when isRepeating from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask 空文件 等等
hadoop 中hive查询表时出现无法使用mayreduce job任务 错误
gyunling的专栏
10-09 1936
hadoop中使用hive+mysql 对表进行查询时，只能全表查询，不能对查询显示部分列。希望大虾们能给出解决办法，具体如下：    1 使用 select * from student;           正常  hive> select * from student; OK 1    lisi    computer    034 2    zhangsan    compu
hive报错汇总
weixin_30394251的博客
06-16 160
1、需要注意的是，要在namenode（超级用户）上操作，貌似是 hive> insert into table record_partition partition(trancation_date) select * from record; WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available i...
hive的调优以及数据倾斜
qq_41848129的博客
08-19 154
调优： fectch抓取(select,filter,limit) (这个适合老版本，因为老版本hive默认是minimal，现在新版本都是默认more) hive>set hive.fetch.task.conversion=none;//转为mr hive> select * from emp; Query ID = ya...
hive基本用法实例
kingxuexi的博客
02-20 600
转自：http://blog.fens.me/hadoop-hive-intro/ Hive的基本使用 1. 进入hive控制台 ~ cd /home/cos/toolkit/hive-0.9.0 ~ bin/hive shell Logging initialized using configuration in file:/home/cos/toolkit/hive-0.
【异常】Hive作业异常kill ：Hadoop MapReduce Error - /bin/bash: /bin/java: is a directory异常
qq_29726945的博客
12-21 368
问题描述 在使用Hive Client跑job时，一直提示job被kill，然后观察YARN的WebUI进行查看，如图： 然后观察Hive Client的控制台输出，如下： Launching Job 1 out of 3 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_...
hive 数据导入导出
eyeofeagle的博客
09-23 575
1， 数据源端 hive表数据导出到本地目录 - - -> scp 本地目录到远程主机 hive> select * from t; OK t.id t.name 1 a 2 a2 3 b Time taken: 0.451 seconds, Fetched: 3 row(s) hive> insert overwrite local directory '/root/t.da...
mybatisPlus使用MetaObjectHandler对字段进行更新
最新发布
m0_56356631的博客
04-23 221
都是符合我们的预期的。
hive创建分区表并插入数据
05-12
创建分区表： ``` CREATE TABLE table_name ( col1 data_type, col2 data_type, ... ) PARTITIONED BY (partition_col data_type); ``` 其中，`table_name` 为表名，`col1`、`col2` 等为列名，`data_type` 为数据类型，`partition_col` 为分区列名。 例如，创建一个分区表 `sales`，包含列 `date`、`product`、`price`，并按照 `date` 列进行分区： ``` CREATE TABLE sales ( date DATE, product STRING, price DOUBLE ) PARTITIONED BY (year INT, month INT, day INT); ``` 插入数据： ``` INSERT INTO TABLE table_name PARTITION (partition_col1=value1, partition_col2=value2, ...) VALUES (value1, value2, ...); ``` 其中，`table_name` 为表名，`partition_col1`、`partition_col2` 等为分区列名，`value1`、`value2` 等为对应的分区值，`value1`、`value2` 等为对应的列值。 例如，插入一条数据到 `sales` 表中，日期为 `2021-01-01`，产品为 `apple`，价格为 `10.5`： ``` INSERT INTO TABLE sales PARTITION (year=2021, month=1, day=1) VALUES ('2021-01-01', 'apple', 10.5); ```

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值