Install Zookeeper+Storm cluster

Install Zookeeper+Storm cluster

1.Set up a Zookeeper cluster

2.Install dependencies on Nimbus and worker machines

3.Download and extract a Storm release to Nimbus and worker machines

4.Fill in mandatory configurations into storm.yaml

5.Launch daemons under supervision using "storm" script and a supervisor of your choice

server information

ubuntu-s-storm-1       137.184.63.172       Zookeeper+Storm

ubuntu-s-storm-2       167.172.138.180     Zookeeper+Storm

ubuntu-s-storm-3       143.244.165.32       Zookeeper+Storm

一、Set up a Zookeeper cluster

https://zookeeper.apache.org/releases.html

1.download and install apache-zookeeper

cd /usr/local/src

wget https://dlcdn.apache.org/zookeeper/zookeeper-3.7.1/apache-zookeeper-3.7.1-bin.tar.gz

2.Decompress the package

tar -zxvf apache-zookeeper-3.7.1-bin.tar.gz

mv apache-zookeeper-3.7.1-bin ../zookeeper

3. install JDK9

tar -zxvf jdk-8u333-linux-x64.tar.gz && mv jdk1.8.0_333 ../jdk_1.8

cat >> /etc/profile  << EOF

export JAVA_HOME=/usr/local/jdk_1.8/

export JRE_HOME=$JAVA_HOME/jre

export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:./

EOF

source /etc/profile

4.createa data and logs

cd /usr/local/zookeeper && mkdir data logs

5.change hosts

cat >> /etc/hosts << EOF

137.184.63.172 ubuntu-s-storm-1

167.172.138.180 ubuntu-s-storm-2

143.244.165.32 ubuntu-s-storm-3

EOF

6.cp conf bak

cd /usr/local/zookeeper/conf && cp zoo_sample.cfg zoo.cfg

7.configure the zoo.cfg  file

sed -i 's/dataDir=.*/dataDir=\/usr\/local\/zookeeper\/data/g' /usr/local/zookeeper/conf/zoo.cfg

cat >> /usr/local/zookeeper/conf/zoo.cfg << EOF

dataLogDir=/usr/local/zookeeper/logs

server.1=137.184.63.172:2881:3881

server.2=167.172.138.180:2881:3881

server.3=143.244.165.32:2881:3881

EOF

Explanation: server. Server ID=server IP address: communication port between servers: voting port between servers

Default port for communication between client and server: 2181

Default port for communication between servers: 2881

Default port for voting between servers: 3881

8.Change zookeeper defalut port ,to avoid conflcts whit the storm

echo "admin.serverPort=8800" >> /usr/local/zookeeper/conf/zoo.cfg

9.config Record the ID of each server,Execute separately

ubuntu-s-storm-1

echo 1 >/usr/local/zookeeper/data/myid

ubuntu-s-storm-2

echo 2 >/usr/local/zookeeper/data/myid

ubuntu-s-storm-13

echo 3 >/usr/local/zookeeper/data/myid

10.start zookeeper

/usr/local/zookeeper/bin/zkServer.sh start/status/restart/stop

二.Install dependencies on Nimbus and worker machines

1.Java 7+ (Apache Storm 1.x is tested through travis ci against both java 7 and java 8 JDKs)

2.Python 2.6.6 (Python 3.x should work too, but is not tested as part of our CI enviornment)

apt install  python-is-python3

三.Download and extract a Storm release to Nimbus and worker machines

1.install Storm

wget https://dlcdn.apache.org/storm/apache-storm-2.4.0/apache-storm-2.4.0.tar.gz

tar -zxvf apache-storm-2.4.0.tar.gz

mv apache-storm-2.4.0 ../storm

四.Fill in mandatory configurations into storm.yaml

1.config storm.yaml

cat >> storm.yaml << EOF

storm.zookeeper.servers:

    - "ubuntu-s-storm-1"

    - "ubuntu-s-storm-2"

    - "ubuntu-s-storm-3"

storm.local.dir: "/usr/local/storm/data"

nimbus.seeds: ["ubuntu-s-storm-1", "ubuntu-s-storm-2", "ubuntu-s-storm-2"]

supervisor.slots.ports:

    - 6700

    - 6701

    - 6702

    - 6703

EOF

2.start Storm

nohup /usr/local/storm/bin/storm nimbus &

nohup /usr/local/storm/bin/storm supervisor &

nohup /usr/local/storm/bin/storm ui &