环境:
- centos 7
- storm 1.0.0
- jdk 1.8
- python 2.7.5
- zookeeper 3.4.8
安装环境依赖:
这里,假设你 已经安装好了jdk、Python
安装zookeeper:
- 下载zookeeper http://www.apache.org/dyn/closer.cgi/zookeeper/
- 解压并cp到 /usr/local下,并建立软连接
- 创建tmp文件夹,用于存放zookeeper的data数据;然后修改zoo_simple.cfg为 zoo.cfg
- 编辑 zoo.cfg,把datadir修改为 /usr/local/zookeeper/tmp
- 启动zookeeper
详细shell脚本如下:
# 复制到usr/local目录下 [root@localhost hager]# mv zookeeper-3.4.8 /usr/local/ # 软连 [root@localhost local]# ln -s zookeeper-3.4.8/ zookeeper #修改权限 其中hager表示你的账户名 [root@localhost local]# sudo chown -R hager ./zookeeper #创建文件夹 [root@localhost zookeeper]# mkdir tmp #修改配置 [root@localhost zookeeper]# cp conf/zoo_sample.cfg conf/zoo.cfg #启动服务 [root@localhost bin]# ./zkServer.sh start
安装storm:
- 下载storm:http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.0.0/apache-storm-1.0.0.tar.gz
- 解压tar -zxvf apache-storm-1.0.0.tar.gz
- cp到/usr/local下,并建立软连接
- 运行授权
- 修改storm.yaml文件
- 运行服务
详细shell如下:
[root@localhost local]# ln -s apache-storm-1.0.0/ apache-storm [root@localhost local]# sudo chown -R hager ./apache-storm [root@localhost apache-storm]# cd conf/
########### These MUST be filled in for a storm configuration # storm.zookeeper.servers: # - "server1" # - "server2" # # nimbus.seeds: ["host1", "host2", "host3"]
# 因为是在单机本机运行,所以把server和seeds都设置为本机IP storm.zookeeper.servers: - "127.0.0.1" nimbus.seeds: ["127.0.0.1"]
简单配置后就可以启动 Storm 了。执行如下命令启动 nimbus 后台进程:
[root@localhost bin]# ./storm nimbus
Running: /usr/java/jdk1.8.0_72/bin/java -server -Ddaemon.name=nimbus -Dstorm.options= -Dstorm.home=/usr/local/apache-storm-1.0.0 -Dstorm.log.dir=/usr/local/apache-storm-1.0.0/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/apache-storm-1.0.0/lib/storm-core-1.0.0.jar:/usr/local/apache-storm-1.0.0/lib/kryo-3.0.3.jar:/usr/local/apache-storm-1.0.0/lib/reflectasm-1.10.1.jar:/usr/local/apache-storm-1.0.0/lib/asm-5.0.3.jar:/usr/local/apache-storm-1.0.0/lib/minlog-1.3.0.jar:/usr/local/apache-storm-1.0.0/lib/objenesis-2.1.jar:/usr/local/apache-storm-1.0.0/lib/clojure-1.7.0.jar:/usr/local/apache-storm-1.0.0/lib/disruptor-3.3.2.jar:/usr/local/apache-storm-1.0.0/lib/log4j-api-2.1.jar:/usr/local/apache-storm-1.0.0/lib/log4j-core-2.1.jar:/usr/local/apache-storm-1.0.0/lib/log4j-slf4j-impl-2.1.jar:/usr/local/apache-storm-1.0.0/lib/slf4j-api-1.7.7.jar:/usr/local/apache-storm-1.0.0/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-1.0.0/lib/servlet-api-2.5.jar:/usr/local/apache-storm-1.0.0/lib/storm-rename-hack-1.0.0.jar:/usr/local/apache-storm/conf -Xmx1024m -Dlogfile.name=nimbus.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/usr/local/apache-storm-1.0.0/log4j2/cluster.xml org.apache.storm.daemon.nimbus
启动 nimbus 后,终端被该进程占用了,不能再继续执行其他命令了。因此我们需要另外开启一个终端,然后执行如下命令启动 supervisor 后台进程:
[root@localhost bin]# ./storm supervisor
Running: /usr/java/jdk1.8.0_72/bin/java -server -Ddaemon.name=supervisor -Dstorm.options= -Dstorm.home=/usr/local/apache-storm-1.0.0 -Dstorm.log.dir=/usr/local/apache-storm-1.0.0/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/apache-storm-1.0.0/lib/storm-core-1.0.0.jar:/usr/local/apache-storm-1.0.0/lib/kryo-3.0.3.jar:/usr/local/apache-storm-1.0.0/lib/reflectasm-1.10.1.jar:/usr/local/apache-storm-1.0.0/lib/asm-5.0.3.jar:/usr/local/apache-storm-1.0.0/lib/minlog-1.3.0.jar:/usr/local/apache-storm-1.0.0/lib/objenesis-2.1.jar:/usr/local/apache-storm-1.0.0/lib/clojure-1.7.0.jar:/usr/local/apache-storm-1.0.0/lib/disruptor-3.3.2.jar:/usr/local/apache-storm-1.0.0/lib/log4j-api-2.1.jar:/usr/local/apache-storm-1.0.0/lib/log4j-core-2.1.jar:/usr/local/apache-storm-1.0.0/lib/log4j-slf4j-impl-2.1.jar:/usr/local/apache-storm-1.0.0/lib/slf4j-api-1.7.7.jar:/usr/local/apache-storm-1.0.0/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-1.0.0/lib/servlet-api-2.5.jar:/usr/local/apache-storm-1.0.0/lib/storm-rename-hack-1.0.0.jar:/usr/local/apache-storm/conf -Xmx256m -Dlogfile.name=supervisor.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/usr/local/apache-storm-1.0.0/log4j2/cluster.xml org.apache.storm.daemon.supervisor
启动stormUI服务,这样就可以通过浏览器访问到stormUI界面;http://localhost:8080/index.html
[root@localhost bin]# ./storm ui
Running: /usr/java/jdk1.8.0_72/bin/java -server -Ddaemon.name=ui -Dstorm.options= -Dstorm.home=/usr/local/apache-storm-1.0.0 -Dstorm.log.dir=/usr/local/apache-storm-1.0.0/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/apache-storm-1.0.0/lib/storm-core-1.0.0.jar:/usr/local/apache-storm-1.0.0/lib/kryo-3.0.3.jar:/usr/local/apache-storm-1.0.0/lib/reflectasm-1.10.1.jar:/usr/local/apache-storm-1.0.0/lib/asm-5.0.3.jar:/usr/local/apache-storm-1.0.0/lib/minlog-1.3.0.jar:/usr/local/apache-storm-1.0.0/lib/objenesis-2.1.jar:/usr/local/apache-storm-1.0.0/lib/clojure-1.7.0.jar:/usr/local/apache-storm-1.0.0/lib/disruptor-3.3.2.jar:/usr/local/apache-storm-1.0.0/lib/log4j-api-2.1.jar:/usr/local/apache-storm-1.0.0/lib/log4j-core-2.1.jar:/usr/local/apache-storm-1.0.0/lib/log4j-slf4j-impl-2.1.jar:/usr/local/apache-storm-1.0.0/lib/slf4j-api-1.7.7.jar:/usr/local/apache-storm-1.0.0/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-1.0.0/lib/servlet-api-2.5.jar:/usr/local/apache-storm-1.0.0/lib/storm-rename-hack-1.0.0.jar:/usr/local/apache-storm-1.0.0:/usr/local/apache-storm/conf -Xmx768m -Dlogfile.name=ui.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/usr/local/apache-storm-1.0.0/log4j2/cluster.xml org.apache.storm.ui.core
遇到问题:
Q1、如何把nimbus、supervisor、UI安装成后台运行;
A:以下是启动Storm各个后台进程的方式:
-
- Nimbus: 在Storm主控节点上运行"bin/storm nimbus >/dev/null 2>&1 &"启动Nimbus后台程序,并放到后台执行;
- Supervisor: 在Storm各个工作节点上运行"bin/storm supervisor >/dev/null 2>&1 &"启动Supervisor后台程序,并放到后台执行;
- UI: 在Storm主控节点上运行"bin/storm ui >/dev/null 2>&1 &"启动UI后台程序,并放到后台执行,启动后可以通过http://{nimbus host}:8080观察集群的worker资源使用情况、Topologies的运行状态等信息。
- (解析:dev启动一个记录日子文件,名为null, 2是标准输出,&1是错误输出, &结束)
[root@localhost bin]# ls flight.bash storm storm.cmd storm-config.cmd storm-local storm.py
# 启动nimbus [root@localhost bin]# ./storm nimbus >/dev/null 2>&1 & [1] 49953 # 启动supervisor
[root@localhost bin]# ./storm supervisor >/dev/null 2>&1 & [2] 49978 # 启动 stormUI
[root@localhost bin]# ./storm ui >/dev/null 2>&1 & [3] 50051 [root@localhost bin]# jps 49953 nimbus 2822 jenkins.war 50167 Jps 49978 supervisor 50157 config_value 2846 Bootstrap 4879 QuorumPeerMain [root@localhost bin]#
-
注意事项:
- Storm后台进程被启动后,将在Storm安装部署目录下的logs/子目录下生成各个进程的日志文件。
- 经测试,Storm UI必须和Storm Nimbus部署在同一台机器上,否则UI无法正常工作,因为UI进程会检查本机是否存在Nimbus链接。
- 为了方便使用,可以将bin/storm加入到系统环境变量中。然后就可以很方便的执行一些命令了。比如storm version、storm list 、storm help......
-
export JAVA_HOME=/usr/java/jdk1.8.0_72
export M2_HOME=/opt/apache-maven
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export ZOOKEEPER_HOME=/usr/local/zookeeper-3.4.8
export STORM_HOME=/usr/local/apache-storm-1.0.0
export PATH=$PATH:$JAVA_HOME/bin:$M2_HOME/bin:$STORM_HOME/bin
Q2、如果8080端口被占用,只是更改storm.yaml中的ui.port并不能彻底修改好,比如8000(logviewer.port),6699这些日志查看端口;
测试:
通过jps 进行检查服务是否正常:
[root@localhost hager]# jps 7328 nimbus 8331 Jps 7630 supervisor 4879 QuorumPeerMain
上传业务jar包到topology:
1、启动Storm Topology
bin/storm jar {jarPath} {mainPath} {args}
[root@localhost bin]# ./storm jar /usr/local/apache-storm/examples/storm-starter/storm-starter-topologies-1.0.0.jar storm.starter.StatefulTopology StatefulTopoArgs1
其中,jar包存在路径为:/usr/local/apache-storm/examples/storm-starter/storm-starter-topologies-1.0.0.jar
main函数入口为:storm.starter.StatefulTopology
args : StatefulTopoArgs1
topologyname: word-count
Topology summary
Name Id Owner Status Uptime Num workers Num executors Num tasks Replication count Assigned Mem (MB) Scheduler Info
word-count word-count-2-1461901281 root ACTIVE 1m 36s 1 8 8 1 832
那么这个word-count 是哪里来的呢?其实是在向集群提交拓扑时,指定的名称:
if (args != null && args.length > 0) {
// 如果指定参数,那么就执行远程集群 conf.setNumWorkers(1); StormSubmitter.submitTopology("word-count", conf, builder.createTopology()); } else {
// 否则,就执行本地集群,然后sleep后就会关闭集群 conf.setMaxTaskParallelism(1); LocalCluster cluster = new LocalCluster(); cluster.submitTopology("word-count-local", conf, builder.createTopology()); Thread.sleep(10000L); cluster.shutdown(); }
具体可以参考官方教程:http://storm.apache.org/releases/1.0.0/Running-topologies-on-a-production-cluster.html
2、结束topology
因为topology是永不停止的,除非你kill掉。所以,如果想停止topology,就需要手动kill掉。
./storm kill {topologyname}
{topologyname}为Topology提交到Storm集群时指定的Topology任务名称。比如上面的StatefulTopology,topologyname是 word-count
运行结果:
// 上传图片报错,回头再补吧。
知识扩展:
Storm集群中有两类节点:主控节点(Master Node)和工作节点(Worker Node)。其中,主控节点只有一个,而工作节点可以有多个。
nimbus:位于主节点上,主要负责storm集群内部分发代码,对节点分配任务,并监视主机故障。
supervisor:Supervisor监听其主机上已经分配的主机的作业,启动和停止Nimbus已经分配的工作进程。
问题汇总:
Q1、org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts ["127.0.0.1"]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:90)
A:zookeeper中把/storm节点数据删除,然后重启zookeeper;(此方法太过于暴力,不知道有没有别的更好的办法 ,zookeeper相关命令参考http://blog.csdn.net/xiaolang85/article/details/13021339/)
./zkcli.sh
参考:
http://blogread.cn/it/article/6570?f=sr 基础知识
http://www.cnblogs.com/panfeng412/archive/2012/11/30/how-to-install-and-deploy-storm-cluster.html
http://dblab.xmu.edu.cn/blog/install-storm/
http://www.cnblogs.com/Lawson/p/3815688.html
问题参考:
https://github.com/hortonworks-gallery/iotdemo-service/commit/02ae6087847afab17e027c10ff9f1c539b7af520