flume sink hive 启动命令中常见异常

常见问题1

Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/streaming/RecordWriter

解决办法一:

将hive中相关的jar复制到$FLUME_HOME/lib下

cp /home/xiaobin/soft/apache-hive-0.14.0-bin/hcatalog/share/hcatalog/*.jar  $FLUME_HOME/lib/
 
cp /home/xiaobin/soft/apache-hive-0.14.0-bin/hive-*.jar $FLUME_HOME/lib/
 
cp antlr-2.7.7.jar ~/soft/apache-flume-1.7.0-bin/lib/
 
cp antlr-runtime-3.4.jar ~/soft/apache-flume-1.7.0-bin/lib/

解决办法二:

指定classpath

--classpath "/home/work/soft/apache-hive-0.14.0-bin/hcatalog/share/hcatalog/*":"/home/work/soft/apache-hive-0.14.0-bin/lib/*"

常见问题二

Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://192.168.1.115:9083', database='default', table='kafkauser', partitionVals=[20170826] }
    at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
    at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:343)
    at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:295)
    at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:253)
    ... 3 more
Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://192.168.1.115:9083', database='default', table='kafkauser', partitionVals=[20170826] }
    at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:383)
    at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:86)
    ... 6 more
Caused by: java.util.concurrent.TimeoutException
    at java.util.concurrent.FutureTask.get(FutureTask.java:205)
    at org.apache.flume.sink.hive.HiveWriter.timedCall(HiveWriter.java:434)
    at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:376)
    ... 7 more

这个问题是目录权限问题,解决办法如下

hadoop dfs -chmod 777 /tmp/hive
chmod 777 /tmp/hive

常见问题三

NullPointerException Non-local session path expected to be non-null

配置文件错误（仔细检查）