storm+kafka整合异常处理

[摘要：1 拷贝kafka依附jar包到storm lib [root@hdmaster libs]# cp kafka_2.10-0.8.2.1.jar /opt/apache-storm-0.9.5/lib/ [root@hdmaster libs]# cp scala-library-2.10.4.ja] 

1 拷贝kafka依赖jar包到storm lib
[root@hdmaster libs]# cp kafka_2.10-0.8.2.1.jar /opt/apache-storm-0.9.5/lib/
[root@hdmaster libs]# cp scala-library-2.10.4.jar /opt/apache-storm-0.9.5/lib/
[root@hdmaster libs]# cp metrics-core-2.2.0.jar /opt/apache-storm-0.9.5/lib/
[root@hdmaster libs]# cp zkclient-0.3.jar /opt/apache-storm-0.9.5/lib/
[root@hdmaster libs]# cp log4j-1.2.16.jar /opt/apache-storm-0.9.5/lib/
[root@hdmaster libs]# cp slf4j-api-1.7.6.jar /opt/apache-storm-0.9.5/lib/
[root@hdmaster libs]# cp jopt-simple-3.2.jar /opt/apache-storm-0.9.5/lib/
2 执行storm jar命令
报错信息：
Exception in thread "main" java.lang.NoClassDefFoundError: storm/kafka/BrokerHosts
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2615)
        at java.lang.Class.getMethod0(Class.java:2856)
        at java.lang.Class.getMethod(Class.java:1668)
        at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException: storm.kafka.BrokerHosts
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

原因：
server端没有storm-kafka相关jar包
从本地maven库找到拷过去


5372 [Thread-11-kafka-reader] ERROR backtype.storm.util - Async loop died!
java.lang.NoClassDefFoundError: org/apache/curator/RetryPolicy
        at storm.kafka.KafkaSpout.open(KafkaSpout.java:85) ~[storm-kafka-0.9.5.jar:0.9.5]
        at backtype.storm.daemon.executor$fn__6579$fn__6594.invoke(executor.clj:522) ~[storm-core-0.9.5.jar:0.9.5]
        at backtype.storm.util$async_loop$fn__459.invoke(util.clj:461) ~[storm-core-0.9.5.jar:0.9.5]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[na:1.7.0_79]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_79]
        at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_79]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_79]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[na:1.7.0_79]
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[na:1.7.0_79]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_79]
        ... 5 common frames omitted

从本地maven库找到curator-client-2.7.1.jar curator-framework-2.7.1.jar拷过去
java.lang.NoClassDefFoundError: com/google/common/base/Preconditions
        at org.apache.curator.ensemble.fixed.FixedEnsembleProvider.<init>(FixedEnsembleProvider.java:39) ~[curator-client-2.7.1.jar:na]
        at org.apache.curator.framework.CuratorFrameworkFactory$Builder.connectString(CuratorFrameworkFactory.java:193) ~[curator-framework-2.7.1.jar:na]
        at org.apache.curator.framework.CuratorFrameworkFactory.newClient(CuratorFrameworkFactory.java:94) ~[curator-framework-2.7.1.jar:na]
        at storm.kafka.ZkState.newCurator(ZkState.java:45) ~[storm-kafka-0.9.5.jar:0.9.5]
        at storm.kafka.ZkState.<init>(ZkState.java:61) ~[storm-kafka-0.9.5.jar:0.9.5]
        at storm.kafka.KafkaSpout.open(KafkaSpout.java:85) ~[storm-kafka-0.9.5.jar:0.9.5]
        at backtype.storm.daemon.executor$fn__6579$fn__6594.invoke(executor.clj:522) ~[storm-core-0.9.5.jar:0.9.5]
        at backtype.storm.util$async_loop$fn__459.invoke(util.clj:461) ~[storm-core-0.9.5.jar:0.9.5]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
Caused by: java.lang.ClassNotFoundException: com.google.common.base.Preconditions
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[na:1.7.0_79]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_79]
        at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_79]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_79]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[na:1.7.0_79]
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[na:1.7.0_79]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_79]
需要guava-11.0.2.jar，从hadoop home下的share=common-lib找的

又报zookeeper错误，将zookeeper-3.4.6.jar放进去
又报下列错误：
5481 [Thread-13-kafka-reader] ERROR backtype.storm.util - Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/topics/my-test-topic5/partitions
        at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:81) ~[storm-kafka-0.9.5.jar:0.9.5]
        at storm.kafka.trident.ZkBrokerReader.<init>(ZkBrokerReader.java:42) ~[storm-kafka-0.9.5.jar:0.9.5]
        at storm.kafka.KafkaUtils.makeBrokerReader(KafkaUtils.java:57) ~[storm-kafka-0.9.5.jar:0.9.5]
        at storm.kafka.KafkaSpout.open(KafkaSpout.java:87) ~[storm-kafka-0.9.5.jar:0.9.5]
        at backtype.storm.daemon.executor$fn__6579$fn__6594.invoke(executor.clj:522) ~[storm-core-0.9.5.jar:0.9.5]
        at backtype.storm.util$async_loop$fn__459.invoke(util.clj:461) ~[storm-core-0.9.5.jar:0.9.5]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
Caused by: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/topics/my-test-topic5/partitions
        at storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:94) ~[storm-kafka-0.9.5.jar:0.9.5]
        at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:65) ~[storm-kafka-0.9.5.jar:0.9.5]
        ... 7 common frames omitted
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/topics/my-test-topic5/partitions
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1590) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:214) ~[curator-framework-2.7.1.jar:na]
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:203) ~[curator-framework-2.7.1.jar:na]
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.7.1.jar:na]
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:200) ~[curator-framework-2.7.1.jar:na]
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:191) ~[curator-framework-2.7.1.jar:na]
        at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:38) ~[curator-framework-2.7.1.jar:na]
        at storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:91) ~[storm-kafka-0.9.5.jar:0.9.5]
        ... 8 common frames omitted

原因：KeeperErrorCode = NoNode for /brokers
配置kafka时，如果使用zookeeper create /kafka创建了节点，kafka与storm集成时new ZkHosts(zks) 需要改成 new ZkHosts(zks,”/kafka/brokers”),不然会报java.lang.RuntimeException: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/topics/my-replicated-topic5/partitions。
storm-kafka插件默认kafka的 zk_path如下：
public class ZkHosts implements BrokerHosts {
private static final String DEFAULT_ZK_PATH = “/brokers”;
改一下代码：

 BrokerHosts brokerHosts = new ZkHosts(zks,"/kafka/brokers"); //或者把/kafka写到zks字符串中，测试过是一样的效果

又报下列错误：

12565 [Thread-9-kafka-reader] ERROR backtype.storm.daemon.executor - 
java.lang.NoClassDefFoundError: com/google/common/base/Preconditions
at org.apache.curator.ensemble.fixed.FixedEnsembleProvider.<init>(FixedEnsembleProvider.java:39) ~[curator-client-2.4.0.jar:na]
at org.apache.curator.framework.CuratorFrameworkFactory$Builder.connectString(CuratorFrameworkFactory.java:176) ~[curator-framework-2.4.0.jar:na]
at org.apache.curator.framework.CuratorFrameworkFactory.newClient(CuratorFrameworkFactory.java:91) ~[curator-framework-2.4.0.jar:na]
at storm.kafka.ZkState.newCurator(ZkState.java:45) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
at storm.kafka.ZkState.<init>(ZkState.java:61) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
at storm.kafka.KafkaSpout.open(KafkaSpout.java:85) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.executor$fn__3371$fn__3386.invoke(executor.clj:529) ~[storm-core-0.9.6.jar:0.9.6]
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:461) ~[storm-core-0.9.6.jar:0.9.6]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_24]

还是缺少包,在Maven下找到com/google/common/base/Preconditions对应的包为guava-13.0.jar。

加入storm的lib目录。

 最后以storm jar stormnew-1.0-SNAPSHOT.jar MyKafkaTopology hdmaster成功提交
 
 看worker日志，有报错信息
 Caused by: java.lang.ClassNotFoundException: storm.kafka.KafkaSpout
 找了下lib包里看到storm-kafka jar里就有这个类，可能因为是后来拷贝进去的没被识别，于是重启了nimbus和supervisor
 然后新错误：
 java.lang.NoClassDefFoundError: org/apache/kafka/common/utils/Utils at kafka.consumer.SimpleConsumer$$anonfun$disconnect$1.apply(SimpleConsumer.scala:49) at kafka.consumer.SimpleConsumer$$anonfun$disconnect$1.apply(SimpleConsumer.scala:49) at kafka.utils.Logging$class.debug(Logging.scala:52) at kafka.consumer.SimpleConsumer.debug(SimpleConsumer.scala:30) at kafka.consumer.SimpleConsumer.disconnect(SimpleConsumer.scala:49) at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:82) at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:68) at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:127) at kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79) at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:77) at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:67) at storm.kafka.PartitionManager.<init>(PartitionManager.java:83) at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) at storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:135) at backtype.storm.daemon.executor$fn__6579$fn__6594$fn__6623.invoke(executor.clj:565) at backtype.storm.util$async_loop$fn__459.invoke(util.clj:463) at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745)
 
 storm程序对应java类里的log.info写在各个节点的worker.log中

感谢关注 Ithao123精品文库频道，ithao123.cn是专门为互联网人打造的学习交流平台，全面满足互联网人工作与学习需求，更多互联网资讯尽在 IThao123!