最近升级调试一个storm任务,出了一个很奇怪的internal exception,堆栈如下:
backtype.storm.daemon.worker - Received invalid messages for unknown tasks. Dropping...
backtype.storm.daemon.worker - Received invalid messages for unknown tasks. Dropping...
backtype.storm.daemon.worker - Received invalid messages for unknown tasks. Dropping...
2015-01-28 18:25:52 [Thread-11-worker-receiver-thread-0] WARN backtype.storm.daemon.worker - Received invalid messages for unknown tasks. Dropping...
2015-01-28 18:29:12 [Thread-12-disruptor-worker-transfer-queue] ERROR backtype.storm.util - Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException: Client is being closed, and does not take requests any more
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:127) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:96) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:81) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.disruptor$consume_loop_STAR_$fn__1681.invoke(disruptor.clj:95) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:457) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_38]
Caused by: java.lang.RuntimeException: Client is being closed, and does not take requests any more
at backtype.storm.messaging.netty.Client.send(Client.java:183) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.utils.TransferDrainer.send(TransferDrainer.java:54) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4610$fn__4611.invoke(worker.clj:330) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4610.invoke(worker.clj:328) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.disruptor$clojure_handler$reify__1668.onEvent(disruptor.clj:59) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:124) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
... 6 common frames omitted
2015-01-28 18:29:12 [Thread-12-disruptor-worker-transfer-queue] ERROR backtype.storm.util - Halting process: ("Async loop died!")
java.lang.RuntimeException: ("Async loop died!")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:319) [storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
at backtype.storm.disruptor$consume_loop_STAR_$fn__1679.invoke(disruptor.clj:93) [storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:467) [storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_
backtype.storm.daemon.worker - Received invalid messages for unknown tasks. Dropping...
backtype.storm.daemon.worker - Received invalid messages for unknown tasks. Dropping...
2015-01-28 18:25:52 [Thread-11-worker-receiver-thread-0] WARN backtype.storm.daemon.worker - Received invalid messages for unknown tasks. Dropping...
2015-01-28 18:29:12 [Thread-12-disruptor-worker-transfer-queue] ERROR backtype.storm.util - Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException: Client is being closed, and does not take requests any more
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:127) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:96) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:81) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.disruptor$consume_loop_STAR_$fn__1681.invoke(disruptor.clj:95) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:457) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_38]
Caused by: java.lang.RuntimeException: Client is being closed, and does not take requests any more
at backtype.storm.messaging.netty.Client.send(Client.java:183) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.utils.TransferDrainer.send(TransferDrainer.java:54) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4610$fn__4611.invoke(worker.clj:330) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4610.invoke(worker.clj:328) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.disruptor$clojure_handler$reify__1668.onEvent(disruptor.clj:59) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:124) ~[storm-core-0.9.3-rc1.jar:0.9.3-rc1]
... 6 common frames omitted
2015-01-28 18:29:12 [Thread-12-disruptor-worker-transfer-queue] ERROR backtype.storm.util - Halting process: ("Async loop died!")
java.lang.RuntimeException: ("Async loop died!")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:319) [storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
at backtype.storm.disruptor$consume_loop_STAR_$fn__1679.invoke(disruptor.clj:93) [storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:467) [storm-core-0.9.3-rc1.jar:0.9.3-rc1]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_