greenplum 数据库gphdfs 外部表报错，请求支持。（已解决）

最新推荐文章于 2022-11-19 00:00:00 发布

朱图图

最新推荐文章于 2022-11-19 00:00:00 发布

阅读量3.3k

点赞数 1

分类专栏： greenplum 文章标签： greenplum 外部表 gphdfs

本文链接：https://blog.csdn.net/xzh19870715/article/details/82899709

版权

greenplum 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

问题说明：

尝试做hive到greenplum的数据迁移，故想到用gphdfs直接访问hdfs下的数据，但是出现报错，查找良久没有找到解决方法，所以来请教各位大神。麻烦各位指点。

环境：

greeplum ：5.1.10

hadoop：hdp2.4

java：1.8.0_91

外部表建表脚本：
CREATE EXTERNAL TABLE gplink.dim_certype_test (
certype VARCHAR ( 32 ),
certype_name VARCHAR ( 32 ),
chosign VARCHAR ( 32 ))
LOCATION ( 'gphdfs://GXGSBigDataHA/apps/hive/warehouse/dma.db/dim_certype/000000_0' ) format 'TEXT';

查询报错：

postgres=# select * from gplink.dim_certype_test;

ERROR: external table gphdfs protocol command ended with error. Error: A JNI error has occurred, please check your installation and try again (seg0 slice1 20.20.20.14:40000 pid=49535)
DETAIL:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/TaskAttemptContext
   at java.lang.Class.getDeclaredMethods0(Native Method)
   at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
   at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
   at java.lang.Class.getMethod0(Class.java:3018)
   at java.lang.Class.getMethod(Class.java:1784)
   at sun.launcher.LauncherHelper.validateMain
Command: execute:source $GPHOME/lib//hadoop/hadoop_env.sh;java $GP_JAVA_OPT -classpath $CLASSPATH com.emc.greenplum.gpdb.hdfsconnector.HDFSReader $GP_SEGMENT_ID $GP_SEGMENT_COUNT TEXT hdp-gnet-1.2.0.0 'gphdfs://GXGSB_0' '000000104300044000000104300044000000104300044' 'certype,certype_name,chosign,'
External table dim_certype_test, file gphdfs://GXGSBigDataHA/apps/hive/warehouse/dma.db/dim_certype/000000_0

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------

************************************************************************************************************************************************

解决方法：

查找官方文档：

If needed, ensure that the CLASSPATH environment variable generated by the $GPHOME/lib/hadoop/hadoop_env.sh file on every Greenplum Database host contains the path to JAR files that contain Java classes that are required for gphdfs.

For example, if gphdfs returns a class not found exception, ensure the JAR file containing the class is on every Greenplum Database host and update the$GPHOME/lib/hadoop/hadoop_env.sh file so that the CLASSPATH environment variable created by file contains the JAR file.

原因是hadoop_env.sh 里面没有包含java的jar包包含在内，这里我处理的方法是修改hadoop_env.sh，将jar添加进去。添加以下语句到hadoop_env.sh里面。

if [ -d "$JAVA_HOME/lib" ]; then
for f in $JAVA_HOME/lib/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
fi

if [ -d "$JAVA_HOME/jre/lib" ]; then
for f in $JAVA_HOME/jre/lib/*.jar; do
CLASSPATH=${CLASSPATH}:$f;
done
fi