最近在将hive中的临时表数据insert overwrite到ods表中去的时候,设置为lzo的压缩格式,报错如下:
Caused by: java.lang.RuntimeException: native-lzo library not available
针对"native-lzo library not available" 异常即lzo安装的异常。
查了很多帖子,最终整合了解决方法。
1.将 HADOOP-LZO-0.4.20放到 每个机器的 /HADOOP-2.7.2/SHARE/HADOOP/COMMON/ 目录下
将编译后得到的hadoop-lzo-0.4.20.jar放到我的三台机器安装了hadoop的share/hadoop/common目录下。
2.修改各个HADOOP节点的HADOOP的配置文件:
cd /opt/module/hadoop-2.7.2/etc/hadoop
vi hadoop-env.sh
# Extra Java CLASSPATH elements. Optional.
export HADOOP_CLASSPATH="<extra_entries>:$HADOOP_CLASSPATH:${HADOOP_HOME}/share/hadoop/common"
#export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/opt/module/hadoop-2.7.2/lib/native:/opt/glibc-2.14/lib
export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/opt/module/hadoop-2.7.2/lib/native