JStorm部署

机器规划

172.16.10.210：jstorm nimbus master，jstorm supervisor
172.16.10.211：jstorm supervisor
172.16.10.212：jstorm supervisor
172.16.10.220:2181 ： zookeeper

集群部署

准备安装包

wget https://github.com/alibaba/jstorm/releases/download/2.2.1/jstorm-2.2.1.zip
unzip jstorm-2.2.1.zip
mv jstorm-2.2.1 /opt/jstorm

环境变量

在~/.bash_profile 文件中或/etc/profile 文件中添加如下配置

vi ~/.bash_profile
vi /etc/profile

···
export JSTORM_HOME=/opt/jstorm
export PATH=$PATH:$JSTORM_HOME/bin
···

配置

编辑$JSTORM_HOME/conf/storm.yaml 配置文件

vim $JSTORM_HOME/conf/storm.yaml

配置storm.zookeeper.servers为zookeeper地址

# 端口默认2181
 storm.zookeeper.servers:
     - "172.16.10.220"

配置storm.zookeeper.root为jstorm在zookeeper的节点名称

 storm.zookeeper.root: "/jstorm"

去掉nimbus.host的注释，配置nimbus节点

 nimbus.host: "172.16.10.210"

配置storm.local.dir，表示jstorm的临时数据存放目录

 storm.local.dir: "%JSTORM_HOME%/data"

去掉supervisor.slots.ports前面的注释，设置supervisor节点执行worker使用的端口列表

 supervisor.slots.ports:
    - 6800
    - 6801
    - 6802
    - 6803

完整配置

########### These MUST be filled in for a storm configuration
 storm.zookeeper.servers:
     - "172.16.10.220"

 storm.zookeeper.root: "/jstorm"

# cluster.name: "default"

 #nimbus.host/nimbus.host.start.supervisor is being used by $JSTORM_HOME/bin/start.sh
 #it only support IP, please don't set hostname
 # For example
 nimbus.host: "172.16.10.210"
 #nimbus.host: "localhost"
 #nimbus.host.start.supervisor: false

# %JSTORM_HOME% is the jstorm home directory
 storm.local.dir: "%JSTORM_HOME%/data"
 # please set absolute path, default path is JSTORM_HOME/logs
# jstorm.log.dir: "absolute path"

# java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib"



# if supervisor.slots.ports is null,
# the port list will be generated by cpu cores and system memory size
# for example,
# there are cpu_num = system_physical_cpu_num/supervisor.slots.port.cpu.weight
# there are mem_num = system_physical_memory_size/(worker.memory.size * supervisor.slots.port.mem.weight)
# The final port number is min(cpu_num, mem_num)
# supervisor.slots.ports.base: 6800
# supervisor.slots.port.cpu.weight: 1.2
# supervisor.slots.port.mem.weight: 0.7
# supervisor.slots.ports: null
 supervisor.slots.ports:
    - 6800
    - 6801
    - 6802
    - 6803
# Default disable user-define classloader
# If there are jar conflict between jstorm and application,
# please enable it
# topology.enable.classloader: false
# ...

部署nimbus

运行

切换路径

cd $JSTORM_HOME/bin

直接运行nimbus

jstorm nimbus

后台运行

nohup jstorm nimbus >> nimbus.txt 2>&1 &

HA

jstorm支持多个nimbus做HA，在另一台机器上执行同样的命令即可，根据zookeeper的原理，后执行的就自动standby了，充当slave，部署了jstorm-ui后，可以在jstorm-ui上可以看到。

部署supervisor

切换路径

cd $JSTORM_HOME/bin

直接运行nimbus

jstorm supervisor

后台运行

nohup jstorm supervisor >> supervisor.txt 2>&1 &

部署storm-ui

下载的jstorm包解压后在根目录下有jstorm-ui-2.2.1.war，这是一个标准的war包，放在tomcat中执行即可

配置

因为storm-ui也需要storm.yaml的配置，默认使用~/.jstorm/storm.yaml

mkdir ~/.jstorm
cp -f $JSTORM_HOME/conf/storm.yaml ~/.jstorm

启动tomcat，就可以看到熟悉的jstorm-ui了，在_Nimbus Summary_可以很清楚的看到主备信息
如果有多个jstorm集群需要管理的话，自JStorm 0.9.6.1 后，无需一个jstorm集群对应一个jstorm-ui了，使用一个web UI 可以管理多个集群， 修改storm.yaml里最末尾的配置即可

开机启动

nimbus

编写脚本 nimbus-start.sh

#!/bin/bash
#chkconfig: 112 63 37

nohup $JSTORM_HOME/bin/jstorm nimbus >> $JSTORM_HOME/bin/nimbus.txt 2>&1 &

将脚本移动到/etc/rc.d/init.d目录下

mv nimbus-start.sh /etc/rc.d/init.d

赋予脚本可执行权限

chmod +x /etc/rc.d/init.d/nimbus-start.sh

添加脚本到开机自动启动项目中

cd /etc/rc.d/init.d
chkconfig --add nimbus-start.sh
chkconfig nimbus-start.sh on

supervisor

编写脚本 supervisor-start.sh

#!/bin/bash
#chkconfig: 112 63 37

nohup $JSTORM_HOME/bin/jstorm supervisor >> $JSTORM_HOME/bin/supervisor 2>&1 &

将脚本移动到/etc/rc.d/init.d目录下

mv supervisor-start.sh /etc/rc.d/init.d

赋予脚本可执行权限

chmod +x /etc/rc.d/init.d/supervisor-start.sh

添加脚本到开机自动启动项目中

cd /etc/rc.d/init.d
chkconfig --add supervisor-start.sh
chkconfig supervisor-start.sh on

jstorm-ui

编写脚本 jstorm-ui-start.sh

#!/bin/bash
#chkconfig: 112 63 37

${TOMCAT_HOME}/bin/startup.sh

将脚本移动到/etc/rc.d/init.d目录下

mv jstorm-ui-start.sh /etc/rc.d/init.d

赋予脚本可执行权限

chmod +x /etc/rc.d/init.d/jstorm-ui-start.sh

添加脚本到开机自动启动项目中

cd /etc/rc.d/init.d
chkconfig --add jstorm-ui-start.sh
chkconfig jstorm-ui-start.sh on

提交topology

提交topology的命令格式

jstorm jar topology-jar-path class …
例如:

jstorm jar springboot-jstorm-cal.jar com.***.***.CalBootstrap

杀死topology

jstorm kill topologyname

例如:

jstorm kill topologyname