运行了一个简简单单的分区表的sql,结果报错,百度谷歌都不管用,特此记录一下。
sql如下:
insert overwrite table dwd_nshop.dwd_nshop_actlog_launch partition (bdp_day='20191102')
select
customer_id,
device_num,
device_type,
os,
os_version,
manufacturer,
carrier,
network_type,
area_code,
from_unixtime(cast(ct/1000 as int),'HH') as launch_time_segment,
ct
from
ods_nshop.ods_nshop_01_useractlog
where bdp_day='20191102'
and `action`='02'
报错记录:
Number of reduce tasks is set to 0 since there's no reduce operator
java.io.FileNotFoundException: File does not exist: hdfs://mini1:9000/opt/hive/lib
opt/hive/lib下面是放jar包的,这里是意思说缺少了jar包?而且还是去hdfs上找?
接下来修改了hive-site.xml
<property>
<name>hive.aux.jars.path</name>
<value>file:///opt/hive/lib/</value>
</property>
加了个路径在本地找
紧接着报错:
Caused by: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found
其实这缺少的jar包(hive-hcatalog-core-2.1.1.jar)我昨天就加在了/opt/hive/lib/
这东西原本是在$HIVE_HOME/hcatalog/share/hcatalog/hive-hcatalog-core-2.1.1.jar
接下来就引用到了hive的第三方jar包,意思就是file:///opt/hive/lib/虽然设置成了hive.aux.jars.path
但是,如果在hive目录下面mkdir 一个auxlib那hive找jar包运行mr时就自动去这个目录下找
然后呢把jar包放进去($HIVE_HOME/hcatalog/share/hcatalog/ 这个目录下所有的jar包)
直接运行成功了
总结:
如果想把你想放的jar包都放在默认的lib下其实也是可以的,但是为了方便管理,这样也是可行的,其实在hive-env.sh内
也可以设置默认的第三方jar包的路径,我自己没有设置,如果以后遇到了上面的类似报错,就试试吧。
# Folder containing extra ibraries required for hive compilation/execution can be controlled by:
# export HIVE_AUX_JARS_PATH=