backtype.storm.event [ERROR] Error when processing event

storm节点supervisor服务异常停止挂了backtype.storm.event [ERROR] Error when processing event

报错信息：

2016-12-16T03:12:06.109+0800 [Thread-3] backtype.storm.daemon.supervisor [INFO] Downloading code for storm id uid-uuid-7-1486070698 from /var/tmp/storm/workdir/nimbus/stormdist/uid-uud-7-1486070698

2016-12-16T03:16:12.297+0800 [Thread-3] backtype.storm.event [ERROR] Error when processing event
org.apache.thrift7.transport.TTransportException: null
        at org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[storm-core-0.9.6.jar:0.9.6]
        at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84) ~[storm-core-0.9.6.jar:0.9.6]
        at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129) ~[storm-core-0.9.6.jar:0.9.6]
        at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) ~[storm-core-0.9.6.jar:0.9.6]
        at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84) ~[storm-core-0.9.6.jar:0.9.6]
        at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) ~[storm-core-0.9.6.jar:0.9.6]
        at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) ~[storm-core-0.9.6.jar:0.9.6]
        at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) ~[storm-core-0.9.6.jar:0.9.6]
        at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69) ~[storm-core-0.9.6.jar:0.9.6]
        at backtype.storm.generated.Nimbus$Client.recv_beginFileDownload(Nimbus.java:401) ~[storm-core-0.9.6.jar:0.9.6]
        at backtype.storm.generated.Nimbus$Client.beginFileDownload(Nimbus.java:388) ~[storm-core-0.9.6.jar:0.9.6]
        at backtype.storm.utils.Utils.downloadFromMaster(Utils.java:254) ~[storm-core-0.9.6.jar:0.9.6]
        at backtype.storm.daemon.supervisor$fn__5150.invoke(supervisor.clj:484) ~[storm-core-0.9.6.jar:0.9.6]
        at clojure.lang.MultiFn.invoke(MultiFn.java:241) ~[clojure-1.5.1.jar:na]
        at backtype.storm.daemon.supervisor$mk_synchronize_supervisor$this__5070.invoke(supervisor.clj:374) ~[storm-core-0.9.6.jar:0.9.6]
        at backtype.storm.event$event_manager$fn__2465.invoke(event.clj:40) ~[storm-core-0.9.6.jar:0.9.6]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
2016-12-16T03:16:12.306+0800 [Thread-3] backtype.storm.util [ERROR] Halting process: ("Error when processing an event")
java.lang.RuntimeException: ("Error when processing an event")
        at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.6.jar:0.9.6]
        at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
        at backtype.storm.event$event_manager$fn__2465.invoke(event.clj:48) [storm-core-0.9.6.jar:0.9.6]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
2016-12-16T03:16:12.308+0800 [Thread-5] backtype.storm.daemon.supervisor [INFO] Shutting down supervisor 108295b3-0156-46e8-ac50-490d8243ff4e

直接启动supervisor服务，一会又停止了，
2016-12-16T10:37:47.496+0800 [Thread-3] backtype.storm.event [ERROR] Error when processing event
org.apache.thrift7.transport.TTransportException: null
        at org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[storm-core-0.9.6.jar:0.9.6]
        at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84) ~[storm-core-0.9.6.jar:0.9.6]
        at org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129) ~[storm-core-0.9.6.jar:0.9.6]
        at org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) ~[storm-core-0.9.6.jar:0.9.6]
        at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84) ~[storm-core-0.9.6.jar:0.9.6]
        at org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) ~[storm-core-0.9.6.jar:0.9.6]
        at org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) ~[storm-core-0.9.6.jar:0.9.6]
        at org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) ~[storm-core-0.9.6.jar:0.9.6]
        at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69) ~[storm-core-0.9.6.jar:0.9.6]
        at backtype.storm.generated.Nimbus$Client.recv_beginFileDownload(Nimbus.java:401) ~[storm-core-0.9.6.jar:0.9.6]
        at backtype.storm.generated.Nimbus$Client.beginFileDownload(Nimbus.java:388) ~[storm-core-0.9.6.jar:0.9.6]
        at backtype.storm.utils.Utils.downloadFromMaster(Utils.java:254) ~[storm-core-0.9.6.jar:0.9.6]
        at backtype.storm.daemon.supervisor$fn__5150.invoke(supervisor.clj:484) ~[storm-core-0.9.6.jar:0.9.6]
        at clojure.lang.MultiFn.invoke(MultiFn.java:241) ~[clojure-1.5.1.jar:na]
        at backtype.storm.daemon.supervisor$mk_synchronize_supervisor$this__5070.invoke(supervisor.clj:374) ~[storm-core-0.9.6.jar:0.9.6]

解决办法：
建议先停止topology，之后再删除节点的storm.local.dir: "/var/tmp/storm/workdir/"（根据自己的配置）目录下的Supervisor和workers，这两个目录，然后重启动。
rm -rf /var/tmp/storm/workdir/*

如果还是启动节点的不了supervisor，处理方式
1、停止topology
2、停止storm服务
3、将nimbus的服务器上的/var/tmp/storm/workdir/*删除下（inbox、stormdist）
4、删除节点的storm.local.dir: "/var/tmp/storm/workdir/"（根据自己的配置）目录下的Supervisor和workers

正常情况就启动了。

解释一下：

storm.local.dir：nimbus与supervisor守护进程需要一个文件夹来存储少量的状态数据（比如jar包、配置等）。该配置决定了nimbus和supervisor将信息存储在哪。该目录必须存在，且对当前用户有适当的操作权限，这个目录的内容必须在集群运行的过程中一直存在，所以避免在/tmp目录下创建。

问题原因可能是代码问题参考：

http://90hadoop.com/2016/05/28/stormsupervisor-yi-chang-tui-chu/

其他参数可参考：

http://blog.csdn.net/liuxinghao/article/details/50577799