安装版本如下:
jstorm-2.2.1
zookeeper-3.4.10
zeromq-4.1.5
先行启动zookeeper的集群服务
修改配置文件conf/storm.yaml内容
########### These MUST be filled in for a storm configuration
storm.zookeeper.servers:
- "192.168.0.11"
- "192.168.0.12"
- "192.168.0.13"
storm.zookeeper.root: "/jstorm"
cluster.name: "jstorm-cluster"
#nimbus.host/nimbus.host.start.supervisor is being used by $JSTORM_HOME/bin/start.sh
#it only support IP, please don't set hostname
# For example
nimbus.host: "192.168.0.11"
#nimbus.host: "localhost"
nimbus.host.start.supervisor: true
# %JSTORM_HOME% is the jstorm home directory
storm.local.dir: "/home/data/jstorm-2.2.1/data"
# please set absolute path, default path is JSTORM_HOME/logs
jstorm.log.dir: "/home/data/jstorm-2.2.1/logs"
java.library.path: "/usr/local/jdk1.8.0_102/lib:/usr/local/lib:/home/data/zeromq-4.1.5:/home/data/jstorm-2.2.1/extlib"
# if supervisor.slots.ports is null,
# the port list will be generated by cpu cores and system memory size
# for example,
# there are cpu_num = system_physical_cpu_num/supervisor.slots.port.cpu.weight
# there are mem_num = system_physical_memory_size/(worker.memory.size * supervisor.slots.port.mem.weight)
# The final port number is min(cpu_num, mem_num)
# supervisor.slots.ports.base: 6800
# supervisor.slots.port.cpu.weight: 1.2
# supervisor.slots.port.mem.weight: 0.7
# supervisor.slots.ports: null
supervisor.slots.ports:
- 6800
- 6801
- 6802
- 6803
# Default disable user-define classloader
# If there are jar conflict between jstorm and application,
# please enable it
topology.enable.classloader: false
# enable supervisor use cgroup to make resource isolation
# Before enable it, you should make sure:
# 1. Linux version (>= 2.6.18)
# 2. Have installed cgroup (check the file's existence:/proc/cgroups)
# 3. You should start your supervisor on root
# You can get more about cgroup:
# http://t.cn/8s7nexU
supervisor.enable.cgroup: false
## send message with sync or async mode
## if this setting is true, netty will use sync mode which means client can send one batch message only after receive one server's response
## Async mode means client can send message without server's response
storm.messaging.netty.sync.mode: false
### Netty will send multiple messages in one batch
### Setting true will improve throughput, but more latency
storm.messaging.netty.transfer.async.batch: true
### default worker memory size, unit is byte
worker.memory.size: 2147483648
# Metrics Monitor
# topology.performance.metrics: it is the switch flag for performance
# purpose. When it is disabled, the data of timer and histogram metrics
# will not be collected.
# topology.alimonitor.metrics.post: If it is disable, metrics data
# will only be printed to log. If it is enabled, the metrics data will be
# posted to alimonitor besides printing to log.
topology.performance.metrics: true
topology.alimonitor.metrics.post: false
# when supervisor is shutdown, automatically shutdown worker
# if run jstorm under other container such as hadoop-yarn,
# this must be set as true
worker.stop.without.supervisor: false
#set how many tuple can spout send in one time.
# For example, if this is setting 100,
# spout can't send the No. 101th tuple until spout receive one tuple's ack message
# topology.max.spout.pending: null
# UI MultiCluster
# Following is an example of multicluster UI configuration
# ui.clusters:
# - {
# name: "jstorm",
# zkRoot: "/jstorm",
# zkServers:
# [ "localhost"],
# zkPort: 2181,
# }
mkdir ~/.jstorm
cp -f /home/data/jstorm-2.1.1/conf/storm.yaml ~/.jstorm
拷贝目录到集群各个节点,在NimbusServer节点上将jstorm-ui-2.2.1.war拷贝到tomcat指定目录下
编辑脚本 startup.sh、shutdown.sh
#!/bin/bash
/home/data/jstorm-2.1.1/bin/jstorm nimbus &
/home/data/jstorm-2.1.1/bin/jstorm supervisor &
/home/data/jstorm-2.1.1/webui/bin/startup.sh
#!/bin/bash
jps|grep NimbusServer|awk '{print "kill -9 " $1}'|sh
jps|grep Supervisor|awk '{print "kill -9 " $1}'|sh
/home/data/jstorm-2.1.1/webui/bin/shutdown.sh
启动成功后可以通过http://192.168.0.11:16666/jstorm-ui-2.2.1
查看集群和拓扑的信息
用例可以在本地模式环境下直接运行测试,也可以打成jar包在集群环境中通过运行
bin/jstorm jar example.jar main类 参数1 参数2 参数3 进行测试
jstorm命令
jstorm command [--config client_storm.yaml] [--exclude-jars exclude1.jar,exclude2.jar] [-c key1=value1,key2=value2][command parameter]
Commands:
activate
blacklist
blobstore
classpath
deactivate
drpc
help
jar
kill
list
localconfvalue
metricsMonitor
nimbus
rebalance
remoteconfvalue
restart
supervisor
update_topology
zktool
[--config client_storm.yaml] optional, setting client's storm.yaml
[--exclude-jars exclude1.jar,exclude2.jar] optional, exclude jars, avoid jar conflict
[-c key1=value1,key2=value2] optional, add key=value pair to configuration
支持添加第三方依赖jar包,在JStorm目录下新建目录extlib,编辑bin/jstorm,修改内容
def get_classpath(extrajars):
ret = []
ret.extend(extrajars)
ret.extend(get_jars_full(JSTORM_DIR))
ret.extend(get_jars_full(JSTORM_DIR + "/lib"))
ret.extend(get_jars_full(JSTORM_DIR + "/extlib"))
ret.extend(INCLUDE_JARS)
return normclasspath(":".join(ret))
第三方依赖jar包放入extlib目录下重启服务即可。