Storm与Hbase使用问题汇总

Storm与Hbase使用问题汇总

Storm版本:0.9.0.1

Hbase版本:0.96.2-hadoop2

java版本:1.6

 

1.storm能提交任务到nimbus,但是supervisor上无法执行,log日志一直出现:

b.s.d.supervisor [INFO] ${JOBID} still hasn't started

解决:删除lib/目录下的log4j-over-slf4j-${version}.jar包,替换为log4j-${version}.jar包.

原因:推测时storm自带的log日志jar包与作业自带的log 日志jar包冲突.

 

2.storm运行作业时出现类无法初始化的问题.

解决:storm对spout或者bolt对象的使用是先初始化,再传输到工作线程.因此需要类以及类成员都具有序列化的特性.因此,对于句柄,自定义类等成员的初始化不能放在类初始化函数中,而应该放在prepare函数中.

 

3.向storm提交的任务在某supervisor节点上响应超时,分配到该节点的所有任务都无法成功启动.

解决:storm默认加载,安装目录$STORM_HOME以及$STORM_HOME/lib下的jar包作为classpath,如果在这些目录下错误的放置打包有log4j,zookeeper等文件的jar包,那么supervisor能正常启动,当时无法接nimbus上的任务jar包.因此要保证storm以及lib目录下没有多余的jar包.

 

4.向storm提交存取Hbase操作的任务,出现以下错误:

Fri May 16 17:37:50 CST 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@16b0337b, java.io.IOException: Could not set up IO Streams

Fri May 16 17:37:51 CST 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@16b0337b, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: dev-hadoop2/10.100.2.93:60020

Fri May 16 17:37:51 CST 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@16b0337b, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: dev-hadoop2/10.100.2.93:60020

Fri May 16 17:37:52 CST 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@16b0337b, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: dev-hadoop2/10.100.2.93:60020

Fri May 16 17:37:54 CST 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@16b0337b, java.io.IOException: Could not set up IO Streams

Fri May 16 17:37:58 CST 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@16b0337b, java.io.IOException: Could not set up IO Streams

Fri May 16 17:38:08 CST 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@16b0337b, java.io.IOException: Could not set up IO Streams

Fri May 16 17:38:18 CST 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@16b0337b, java.io.IOException: Could not set up IO Streams

Fri May 16 17:38:28 CST 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@16b0337b, java.io.IOException: Could not set up IO Streams

Fri May 16 17:38:38 CST 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@16b0337b, java.io.IOException: Could not set up IO Streams

解决:这是hbase上的一个bug:HBASE-10432. 参考网页

https://issues.apache.org/jira/browse/HBASE-10432

升级到0.96.2版本后,问题解决.

 

5.向storm提交存取Hbase操作的任务出现以下错误:

java.lang.NoSuchMethodError:org.apache.hadoop.net.NetUtils.getInputStream(Ljava/net/Socket;)Lorg/apache/hadoop/net/SocketInputWrapper;at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:862) ~[stormjar.jar:na]

解决:storm的classpath中含有hadoop1.1.2版本的jar包,去掉后解决.当前使用的hbase版本时hadoop2,与hadoop1存在不兼容的问题.

 

6.在storm集群上执行时,出现未使用的代码无法通过验证的问题,:

 未解决....在注释掉问题代码运行了一段时间后问题也没有在出现.