Mac Track

你有一个特别擅长的方向吗? 特别熟悉?有丰富的经验?

Twitter Storm入门实战

开始实战,确定你已经安装所有需要的软件:

====================

启动zookeeper

shawny@localhost:~/software/zookeeper-3.3.3/bin$ ./zkServer.sh start
JMX enabled by default
Using config: /home/shawny/software/zookeeper-3.3.3/bin/../conf/zoo.cfg
Starting zookeeper ... 
STARTED

===========
启动storm nimbus (如果启动失败则查看zookeeper上的storm znode是否没有清除干净,如果非空则手动删除storm znode下面的所有子节点)

shawny@localhost:~/software/storm-0.5.5-SNAPSHOT$ ./bin/storm nimbus
Running:  java -server -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Xmx1024m -Dlogfile.name=nimbus.log -Dlog4j.configuration=storm.log.properties -cp /home/shawny/software/storm-0.5.5-SNAPSHOT/storm-0.5.5-SNAPSHOT.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-jetty-adapter-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/junit-3.8.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-exec-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/hiccup-0.3.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5-20081211.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-lang-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jline-0.9.94.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/json-simple-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/tools.macro-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jvyaml-1.0.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-logging-1.1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/core.incubator-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-io-1.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpcore-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-core-0.3.10.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-api-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/libthrift7-0.7.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-contrib-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/zookeeper-3.3.2.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-fileupload-1.2.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpclient-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clj-time-0.3.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-servlet-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/joda-time-1.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jzmq-2.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-util-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clout-0.4.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-log4j12-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/log4j-1.2.16.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/compojure-0.6.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-codec-1.3.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/log4j:/home/shawny/software/storm-0.5.5-SNAPSHOT/conf backtype.storm.daemon.nimbus 

===========
启动storm supervisor

shawny@localhost:~/software/storm-0.5.5-SNAPSHOT$ ./bin/storm  supervisor
Running:  java -server -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Xmx1024m -Dlogfile.name=supervisor.log -Dlog4j.configuration=storm.log.properties -cp /home/shawny/software/storm-0.5.5-SNAPSHOT/storm-0.5.5-SNAPSHOT.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-jetty-adapter-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/junit-3.8.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-exec-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/hiccup-0.3.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5-20081211.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-lang-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jline-0.9.94.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/json-simple-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/tools.macro-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jvyaml-1.0.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-logging-1.1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/core.incubator-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-io-1.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpcore-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-core-0.3.10.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-api-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/libthrift7-0.7.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-contrib-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/zookeeper-3.3.2.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-fileupload-1.2.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpclient-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clj-time-0.3.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-servlet-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/joda-time-1.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jzmq-2.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-util-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clout-0.4.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-log4j12-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/log4j-1.2.16.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/compojure-0.6.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-codec-1.3.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/log4j:/home/shawny/software/storm-0.5.5-SNAPSHOT/conf backtype.storm.daemon.supervisor 

==========
启动storm UI

shawny@localhost:~/software/storm-0.5.5-SNAPSHOT$ bin/storm ui
Running:  java -server -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Xmx768m -Dlogfile.name=ui.log -Dlog4j.configuration=storm.log.properties -cp /home/shawny/software/storm-0.5.5-SNAPSHOT/storm-0.5.5-SNAPSHOT.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-jetty-adapter-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/junit-3.8.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-exec-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/hiccup-0.3.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5-20081211.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-lang-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jline-0.9.94.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/json-simple-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/tools.macro-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jvyaml-1.0.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-logging-1.1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/core.incubator-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-io-1.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpcore-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-core-0.3.10.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-api-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/libthrift7-0.7.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-contrib-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/zookeeper-3.3.2.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-fileupload-1.2.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpclient-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clj-time-0.3.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-servlet-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/joda-time-1.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jzmq-2.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-util-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clout-0.4.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-log4j12-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/log4j-1.2.16.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/compojure-0.6.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-codec-1.3.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/log4j:/home/shawny/software/storm-0.5.5-SNAPSHOT:/home/shawny/software/storm-0.5.5-SNAPSHOT/conf backtype.storm.ui.core

=================
通过“http://localhost:8080/”查看storm信息

Cluster Summary
Nimbus uptime Supervisors Used slots Free slots Total slots Running tasks
4m 10s 1 0 4 4 0
Topology summary
Name Id Status Uptime Num workers Num tasks
Supervisor summary
Host Uptime Slots Used slots
localhost 3m 1s 4 0

=============
创建Storm_test工程

shawny@localhost:~/workspace/java/Storm_test$ tree
.
|-- bin
|   |-- RandomSentenceSpout.class
|   |-- resources
|   |   |-- splitsentence.py
|   |   |-- storm.py
|   |   `-- storm.rb
|   |-- storm.yaml
|   |-- WordCountTopology.class
|   |-- WordCountTopology$SplitSentence.class
|   `-- WordCountTopology$WordCount.class
|-- multilang
|   `-- resources
|       |-- splitsentence.py
|       |-- storm.py
|       `-- storm.rb
|-- src
|   |-- RandomSentenceSpout.java
|   `-- WordCountTopology.java
`-- storm_test.jar

NOTE:
1) 程序写好后打成jar包,注意export后面的library handling选择第二个
2) ~/.storm下面需要有storm.yaml, 里面包含storm的配置信息

===========
提交jar包给storm

shawny@localhost:~/software/storm-0.5.5-SNAPSHOT$ ./bin/storm  jar ~/workspace/java/Storm_test/storm_test.jar WordCountTopology test
Running: export STORM_JAR=/home/shawny/workspace/java/Storm_test/storm_test.jar; java -client -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib  -cp /home/shawny/software/storm-0.5.5-SNAPSHOT/storm-0.5.5-SNAPSHOT.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-jetty-adapter-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/junit-3.8.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-exec-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/hiccup-0.3.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5-20081211.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-lang-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jline-0.9.94.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/json-simple-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/tools.macro-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jvyaml-1.0.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-logging-1.1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/core.incubator-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-io-1.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpcore-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-core-0.3.10.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-api-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/libthrift7-0.7.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-contrib-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/zookeeper-3.3.2.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-fileupload-1.2.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpclient-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clj-time-0.3.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-servlet-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/joda-time-1.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jzmq-2.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-util-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clout-0.4.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-log4j12-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/log4j-1.2.16.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/compojure-0.6.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-codec-1.3.jar:/home/shawny/workspace/java/Storm_test/storm_test.jar:/home/shawny/.storm:/home/shawny/software/storm-0.5.5-SNAPSHOT/bin WordCountTopology test
0    [main] INFO  backtype.storm.StormSubmitter  - Jar not uploaded to master yet. Submitting jar...
25   [main] INFO  backtype.storm.StormSubmitter  - Uploading topology jar /home/shawny/workspace/java/Storm_test/storm_test.jar to assigned location: /tmp/storm/nimbus/inbox/stormjar-a37d88eb-f4c1-4fa0-8621-27c00a8beaf3.jar
472  [main] INFO  backtype.storm.StormSubmitter  - Successfully uploaded topology jar to assigned location: /tmp/storm/nimbus/inbox/stormjar-a37d88eb-f4c1-4fa0-8621-27c00a8beaf3.jar
473  [main] INFO  backtype.storm.StormSubmitter  - Submitting topology test in distributed mode with conf {"topology.workers":3,"topology.debug":true}
747  [main] INFO  backtype.storm.StormSubmitter  - Finished submitting topology: test

====================
查看storm logs

shawny@localhost:~/software/storm-0.5.5-SNAPSHOT/logs$ tree
.
|-- nimbus.log
|-- supervisor.log
|-- ui.log
|-- worker-6700.log
|-- worker-6701.log
`-- worker-6702.log

===============
kill 提交的topology (test)

shawny@localhost:~/software/storm-0.5.5-SNAPSHOT$ ./bin/storm kill test
Running:  java -client -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib  -cp /home/shawny/software/storm-0.5.5-SNAPSHOT/storm-0.5.5-SNAPSHOT.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-jetty-adapter-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/junit-3.8.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-exec-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/hiccup-0.3.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5-20081211.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-lang-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jline-0.9.94.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/json-simple-1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/tools.macro-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jvyaml-1.0.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-logging-1.1.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/core.incubator-0.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/servlet-api-2.5.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-io-1.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpcore-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-core-0.3.10.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-api-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/libthrift7-0.7.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clojure-contrib-1.2.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/zookeeper-3.3.2.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-fileupload-1.2.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/httpclient-4.0.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clj-time-0.3.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/ring-servlet-0.3.11.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/joda-time-1.6.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jzmq-2.1.0.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/jetty-util-6.1.26.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/clout-0.4.1.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/slf4j-log4j12-1.5.8.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/log4j-1.2.16.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/compojure-0.6.4.jar:/home/shawny/software/storm-0.5.5-SNAPSHOT/lib/commons-codec-1.3.jar:/home/shawny/.storm:/home/shawny/software/storm-0.5.5-SNAPSHOT/bin backtype.storm.command.kill_topology test
0    [main] INFO  backtype.storm.command.kill-topology  - Killed topology: test
阅读更多
版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/macyang/article/details/7028993
个人分类: 实时计算
想对作者说点什么? 我来说一句

没有更多推荐了,返回首页

不良信息举报

Twitter Storm入门实战

最多只允许输入30个字

加入CSDN,享受更精准的内容推荐,与500万程序员共同成长!
关闭
关闭