Storm集群，zookeeper集群搭建

环境：

1. CentOS 6.5
2. storm 1.0.2
3. jdk 1.8

4. zookeeper 3.4.6

   准备工作：
   1、storm 3台机器 例：192.168.168.110，192.168.168.111，192.168.168.112
   2、zookeeper 3台 192.168.168.1，192.168.168.2，192.168.168.3

1、安装JDK

安装jdk1.8 ，配置环境变量，这里不多说了，不会的自行百度

2、配置zookeeper环境

上传zookeeper安装包到一台机器（192.168.1.1） /usr/local 下

解压安装包 tar zxvf  zookeeper-3.4.6.tar.gz
建立软连接 ln -sv zookeeper-3.4.6 zookeeper

配置环境变量
export ZOOKEEPER_HOME=/usr/local/zookeeper
export PATH=$ZOOKEEPER_HOME/bin:$JAVA_HOME/bin:$PATH

进入/usr/local/zookeeper/conf文件夹下修改一下配置文件名字

mv zoo_simple.cfg zoo.cfg
vi zoo.cfg

首先 修改 dataDir，顾名思义就是【数据目录】了，修改成我们自定义的即可。

server.X=A:B:C

X-代表服务器编号

A-代表ip

B和C-代表端口，这个端口用来系统之间通信






每台服务器上新建dataDir目录，在data文件夹新建文件myid

vi myid

 server.0=192.168.127.129:2888:3888【192.168.127.129服务器上面的myid填写0】
 server.1=192.168.127.130:2888:3888【192.168.127.130服务器上面的myid填写1】
 server.2=192.168.127.131:2888:3888【192.168.127.131服务器上面的myid填写2】

然后

zkServer.sh start

3、配置storm集群

安装storm：

下载storm：http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.0.0/apache-storm-1.0.0.tar.gz

解压tar -zxvf apache-storm-1.0.0.tar.gz

cp到/usr/local下，并建立软连接

运行授权

修改storm.yaml文件

运行服务

# 
# 
# ##### These may optionally be filled in:
#    
## List of custom serializations
# topology.kryo.register:
#     - org.mycompany.MyType
#     - org.mycompany.MyType2: org.mycompany.MyType2Serializer
#
## List of custom kryo decorators
# topology.kryo.decorators:
#     - org.mycompany.MyDecorator
#
## Locations of the drpc servers
# drpc.servers:
#     - "server1"
#     - "server2"

drpc.servers:
      - "storm018164"


## Metrics Consumers
# topology.metrics.consumer.register:
#   - class: "org.apache.storm.metric.LoggingMetricsConsumer"
#     parallelism.hint: 1
#   - class: "org.mycompany.MyMetricsConsumer"
#     parallelism.hint: 1
#     argument:
#       - endpoint: "metrics-collector.mycompany.org"
storm.local.dir: "/usr/local/storm"
storm.zookeeper.servers:
      - "zookeeper018172"
      - "zookeeper018173"
      - "zookeeper018174"
storm.zookeeper.port: 2181
nimbus.seeds: ["storm018164"]
storm.zookeeper.root: "/storm-1.0.2"
storm.cluster.mode: "distributed"
storm.log.dir: "/var/log/storm/logs"
logviewer.port: 60099
logviewer.appender.name: "A1"
ui.port: 9990
topology.debug: false
topology.executor.receive.buffer.size: 16384
topology.executor.send.buffer.size: 16384
topology.worker.receiver.thread.count: 4
storm.messaging.netty.server_worker_threads: 4
storm.messaging.netty.client_worker_threads: 4
storm.messaging.transport: "org.apache.storm.messaging.netty.Context"
storm.messaging.netty.buffer_size: 52428800
storm.messaging.netty.max_retries: 10
storm.messaging.netty.min_wait_ms: 5000
storm.messaging.netty.max_wait_ms: 10000
nimbus.childopts: "-Xmx2048m -Xms2048m -Xmn500m -XX:PermSize=256M -XX:MaxPermSize=256M -XX:MaxDirectMemorySize=128M -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSConcurrentMTEnabled -Djava.net.preferIPv4Stack=true"
supervisor.childopts: "-Xmx2048m -Xms2048m -Xmn500m -XX:PermSize=256M -XX:MaxPermSize=256M -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSConcurrentMTEnabled -Djava.net.preferIPv4Stack=true"
ui.childopts: "-Xmx2048m -Xms2048m -Xmn500m -XX:PermSize=256M -XX:MaxPermSize=256M -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSConcurrentMTEnabled -Djava.net.preferIPv4Stack=true"
worker.childopts: "-Xmx2048m -Xms2048m -Xmn500m -XX:PermSize=256M -XX:MaxPermSize=256M -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSConcurrentMTEnabled -Djava.net.preferIPv4Stack=true"
supervisor.slots.ports:
      - 6700
      - 6701
      - 6702
      - 6703
      - 6704
      - 6705

其它机器把配置好的storm复制过去，同样环境变量，host不能少

启动storm

脚本如下：

启动 nimbus ui 等

nohup $STORM_HOME/bin/storm nimbus > /dev/null 2>&1 &
nohup $STORM_HOME/bin/storm ui > /dev/null 2>&1 &
nohup $STORM_HOME/bin/storm logviewer > /dev/null 2>&1 &
nohup $STORM_HOME/bin/storm drpc > /dev/null 2>&1 &

启动 supervisor

source /etc/profile

nohup $STORM_HOME/bin/storm supervisor > /dev/null 2>&1 &
nohup $STORM_HOME/bin/storm logviewer > /dev/null 2>&1 &

然后jps 检查