centos 下storm集群的搭建

1.安装jdk
我安装的是
 
[master@node0 bin]$ java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
2.安装python
我安装的是
[master@node0 bin]$ python
Python 2.6.6 (r266:84292, Aug 18 2016, 15:13:37) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
3.storm下载
下载命令是
wget http://mirrors.hust.edu.cn/apache/storm/apache-storm-1.0.5/apache-storm-1.0.5.tar.gz 
4.安装与搭建
[master@node0 soft]$ tar -zxvf apache-storm-1.0.5.tar.gz 
修改配置文件
[master@node0 storm1.0.5]$ vi conf/storm.yaml 
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

########### These MUST be filled in for a storm configuration
storm.zookeeper.servers:
     - "server-01"
     - "server-02"
     - "server-03"
# 
nimbus.seeds: ["node0"]
# 
# 
# ##### These may optionally be filled in:
#    
## List of custom serializations
# topology.kryo.register:
#     - org.mycompany.MyType
#     - org.mycompany.MyType2: org.mycompany.MyType2Serializer
#
## List of custom kryo decorators
# topology.kryo.decorators:
#     - org.mycompany.MyDecorator
#
## Locations of the drpc servers
# drpc.servers:
#     - "server1"
#     - "server2"

## Metrics Consumers
# topology.metrics.consumer.register:
#   - class: "org.apache.storm.metric.LoggingMetricsConsumer"
#     parallelism.hint: 1
#   - class: "org.mycompany.MyMetricsConsumer"
#     parallelism.hint: 1
#     argument:
#       - endpoint: "metrics-collector.mycompany.org"
supervisor.slots.ports:  
   - 6700  
   - 6701  
   - 6702  
   - 6703 

5.单机启动
启动nimbus
[master@node0 bin]$ ./storm nimbus&
[master@node0 bin]$ Running: /opt/java/bin/java -server -Ddaemon.name=nimbus -Dstorm.options= -Dstorm.home=/home/master/soft/storm1.0.5 -Dstorm.log.dir=/home/master/soft/storm1.0.5/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/master/soft/storm1.0.5/lib/log4j-api-2.8.jar:/home/master/soft/storm1.0.5/lib/log4j-core-2.8.jar:/home/master/soft/storm1.0.5/lib/kryo-3.0.3.jar:/home/master/soft/storm1.0.5/lib/log4j-over-slf4j-1.6.6.jar:/home/master/soft/storm1.0.5/lib/storm-rename-hack-1.0.5.jar:/home/master/soft/storm1.0.5/lib/slf4j-api-1.7.21.jar:/home/master/soft/storm1.0.5/lib/asm-5.0.3.jar:/home/master/soft/storm1.0.5/lib/log4j-slf4j-impl-2.8.jar:/home/master/soft/storm1.0.5/lib/minlog-1.3.0.jar:/home/master/soft/storm1.0.5/lib/servlet-api-2.5.jar:/home/master/soft/storm1.0.5/lib/clojure-1.7.0.jar:/home/master/soft/storm1.0.5/lib/storm-core-1.0.5.jar:/home/master/soft/storm1.0.5/lib/reflectasm-1.10.1.jar:/home/master/soft/storm1.0.5/lib/objenesis-2.1.jar:/home/master/soft/storm1.0.5/lib/disruptor-3.3.2.jar:/home/master/soft/storm1.0.5/conf -Xmx1024m -Dlogfile.name=nimbus.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/home/master/soft/storm1.0.5/log4j2/cluster.xml org.apache.storm.daemon.nimbus
查看是否成功:
[master@node0 bin]$ jps
14066 Main
15987 nimbus
16119 Jps
启动UI:
[master@node0 bin]$ ./storm ui&
[master@node0 bin]$ jpRunning: /opt/java/bin/java -server -Ddaemon.name=ui -Dstorm.options= -Dstorm.home=/home/master/soft/storm1.0.5 -Dstorm.log.dir=/home/master/soft/storm1.0.5/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/master/soft/storm1.0.5/lib/log4j-api-2.8.jar:/home/master/soft/storm1.0.5/lib/log4j-core-2.8.jar:/home/master/soft/storm1.0.5/lib/kryo-3.0.3.jar:/home/master/soft/storm1.0.5/lib/log4j-over-slf4j-1.6.6.jar:/home/master/soft/storm1.0.5/lib/storm-rename-hack-1.0.5.jar:/home/master/soft/storm1.0.5/lib/slf4j-api-1.7.21.jar:/home/master/soft/storm1.0.5/lib/asm-5.0.3.jar:/home/master/soft/storm1.0.5/lib/log4j-slf4j-impl-2.8.jar:/home/master/soft/storm1.0.5/lib/minlog-1.3.0.jar:/home/master/soft/storm1.0.5/lib/servlet-api-2.5.jar:/home/master/soft/storm1.0.5/lib/clojure-1.7.0.jar:/home/master/soft/storm1.0.5/lib/storm-core-1.0.5.jar:/home/master/soft/storm1.0.5/lib/reflectasm-1.10.1.jar:/home/master/soft/storm1.0.5/lib/objenesis-2.1.jar:/home/master/soft/storm1.0.5/lib/disruptor-3.3.2.jar:/home/master/soft/storm1.0.5:/home/master/soft/storm1.0.5/conf -Xmx768m -Dlogfile.name=ui.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/home/master/soft/storm1.0.5/log4j2/cluster.xml org.apache.storm.ui.core
查看是否成功:
[master@node0 bin]$ jps
14066 Main
15987 nimbus
16277 Jps
16139 core
core就是ui,端口是8080

启动supervisor
[master@node0 bin]$ ./storm supervisor&
[master@node0 bin]$ Running: /opt/java/bin/java -server -Ddaemon.name=supervisor -Dstorm.options= -Dstorm.home=/home/master/soft/storm1.0.5 -Dstorm.log.dir=/home/master/soft/storm1.0.5/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/master/soft/storm1.0.5/lib/log4j-api-2.8.jar:/home/master/soft/storm1.0.5/lib/log4j-core-2.8.jar:/home/master/soft/storm1.0.5/lib/kryo-3.0.3.jar:/home/master/soft/storm1.0.5/lib/log4j-over-slf4j-1.6.6.jar:/home/master/soft/storm1.0.5/lib/storm-rename-hack-1.0.5.jar:/home/master/soft/storm1.0.5/lib/slf4j-api-1.7.21.jar:/home/master/soft/storm1.0.5/lib/asm-5.0.3.jar:/home/master/soft/storm1.0.5/lib/log4j-slf4j-impl-2.8.jar:/home/master/soft/storm1.0.5/lib/minlog-1.3.0.jar:/home/master/soft/storm1.0.5/lib/servlet-api-2.5.jar:/home/master/soft/storm1.0.5/lib/clojure-1.7.0.jar:/home/master/soft/storm1.0.5/lib/storm-core-1.0.5.jar:/home/master/soft/storm1.0.5/lib/reflectasm-1.10.1.jar:/home/master/soft/storm1.0.5/lib/objenesis-2.1.jar:/home/master/soft/storm1.0.5/lib/disruptor-3.3.2.jar:/home/master/soft/storm1.0.5/conf -Xmx256m -Dlogfile.name=supervisor.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/home/master/soft/storm1.0.5/log4j2/cluster.xml org.apache.storm.daemon.supervisor.Supervisor
查看

[master@node0 bin]$ jps
16401 Jps
14066 Main
15987 nimbus
16297 Supervisor
16139 core
最后查看UI:
图片在这
单机版完毕:

6.集群版搭建
10.0.2.10 node0  // 主机
10.0.2.15 server-01 //zk
10.0.2.6 server-02 //zk
10.0.2.5 server-03 //zk
10.0.2.7 kafka-01 //kafka
10.0.2.8 kafka-02 //kafka
10.0.2.9 kafka-03 //kafka
10.0.2.11 storm-1 //节点1
10.0.2.12 storm-2//节点2
10.0.2.13 storm-3//节点3
上面是我的机器的hosts
几个节点的配置文件统一为:
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

########### These MUST be filled in for a storm configuration
storm.zookeeper.servers:
     - "server-01"
     - "server-02"
     - "server-03"
# 
nimbus.seeds: ["node0","storm-1","storm-2","storm-3"]
# 
# 
# ##### These may optionally be filled in:
#    
## List of custom serializations
# topology.kryo.register:
#     - org.mycompany.MyType
#     - org.mycompany.MyType2: org.mycompany.MyType2Serializer
#
## List of custom kryo decorators
# topology.kryo.decorators:
#     - org.mycompany.MyDecorator
#
## Locations of the drpc servers
# drpc.servers:
#     - "server1"
#     - "server2"

## Metrics Consumers
# topology.metrics.consumer.register:
#   - class: "org.apache.storm.metric.LoggingMetricsConsumer"
#     parallelism.hint: 1
#   - class: "org.mycompany.MyMetricsConsumer"
#     parallelism.hint: 1
#     argument:
#       - endpoint: "metrics-collector.mycompany.org"
supervisor.slots.ports:  
   - 6700  
   - 6701  
   - 6702  
   - 6703 
然后node0之启动nimbus和ui,其他节点之启动supervisor

分别看看三个节点的启动情况:
 storm-1:
[root@localhost bin]# ./storm supervisor &
[root@localhost bin]# Running: /opt/java/bin/java -server -Ddaemon.name=supervisor -Dstorm.options= -Dstorm.home=/opt/storm1.0.5 -Dstorm.log.dir=/opt/storm1.0.5/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /opt/storm1.0.5/lib/asm-5.0.3.jar:/opt/storm1.0.5/lib/log4j-slf4j-impl-2.8.jar:/opt/storm1.0.5/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm1.0.5/lib/kryo-3.0.3.jar:/opt/storm1.0.5/lib/objenesis-2.1.jar:/opt/storm1.0.5/lib/minlog-1.3.0.jar:/opt/storm1.0.5/lib/slf4j-api-1.7.21.jar:/opt/storm1.0.5/lib/storm-rename-hack-1.0.5.jar:/opt/storm1.0.5/lib/servlet-api-2.5.jar:/opt/storm1.0.5/lib/reflectasm-1.10.1.jar:/opt/storm1.0.5/lib/storm-core-1.0.5.jar:/opt/storm1.0.5/lib/log4j-api-2.8.jar:/opt/storm1.0.5/lib/disruptor-3.3.2.jar:/opt/storm1.0.5/lib/clojure-1.7.0.jar:/opt/storm1.0.5/lib/log4j-core-2.8.jar:/opt/storm1.0.5/conf -Xmx256m -Dlogfile.name=supervisor.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/opt/storm1.0.5/log4j2/cluster.xml org.apache.storm.daemon.supervisor.Supervisor

[root@localhost bin]# 
[root@localhost bin]# 
[root@localhost bin]# jps
2420 Jps
2325 Supervisor
[root@localhost bin]# 
 storm-2:
[root@localhost bin]#  ./storm supervisor &
[1] 2253
[root@localhost bin]# jps
2307 config_value
2318 Jps
[root@localhost bin]# Running: /opt/java/bin/java -server -Ddaemon.name=supervisor -Dstorm.options= -Dstorm.home=/opt/storm1.0.5 -Dstorm.log.dir=/opt/storm1.0.5/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /opt/storm1.0.5/lib/log4j-slf4j-impl-2.8.jar:/opt/storm1.0.5/lib/objenesis-2.1.jar:/opt/storm1.0.5/lib/storm-rename-hack-1.0.5.jar:/opt/storm1.0.5/lib/minlog-1.3.0.jar:/opt/storm1.0.5/lib/reflectasm-1.10.1.jar:/opt/storm1.0.5/lib/asm-5.0.3.jar:/opt/storm1.0.5/lib/disruptor-3.3.2.jar:/opt/storm1.0.5/lib/log4j-api-2.8.jar:/opt/storm1.0.5/lib/slf4j-api-1.7.21.jar:/opt/storm1.0.5/lib/kryo-3.0.3.jar:/opt/storm1.0.5/lib/log4j-core-2.8.jar:/opt/storm1.0.5/lib/servlet-api-2.5.jar:/opt/storm1.0.5/lib/storm-core-1.0.5.jar:/opt/storm1.0.5/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm1.0.5/lib/clojure-1.7.0.jar:/opt/storm1.0.5/conf -Xmx256m -Dlogfile.name=supervisor.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/opt/storm1.0.5/log4j2/cluster.xml org.apache.storm.daemon.supervisor.Supervisor

[root@localhost bin]# 
[root@localhost bin]# jps
2358 Jps
2253 Supervisor
[root@localhost bin]# 
storm-3:
[root@localhost bin]#  ./storm supervisor &
[1] 2256
[root@localhost bin]# jps
2308 Jps
2297 config_value
[root@localhost bin]# Running: /opt/java/bin/java -server -Ddaemon.name=supervisor -Dstorm.options= -Dstorm.home=/opt/storm1.0.5 -Dstorm.log.dir=/opt/storm1.0.5/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /opt/storm1.0.5/lib/storm-core-1.0.5.jar:/opt/storm1.0.5/lib/disruptor-3.3.2.jar:/opt/storm1.0.5/lib/log4j-core-2.8.jar:/opt/storm1.0.5/lib/storm-rename-hack-1.0.5.jar:/opt/storm1.0.5/lib/kryo-3.0.3.jar:/opt/storm1.0.5/lib/log4j-slf4j-impl-2.8.jar:/opt/storm1.0.5/lib/servlet-api-2.5.jar:/opt/storm1.0.5/lib/reflectasm-1.10.1.jar:/opt/storm1.0.5/lib/asm-5.0.3.jar:/opt/storm1.0.5/lib/slf4j-api-1.7.21.jar:/opt/storm1.0.5/lib/clojure-1.7.0.jar:/opt/storm1.0.5/lib/objenesis-2.1.jar:/opt/storm1.0.5/lib/minlog-1.3.0.jar:/opt/storm1.0.5/lib/log4j-api-2.8.jar:/opt/storm1.0.5/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm1.0.5/conf -Xmx256m -Dlogfile.name=supervisor.log -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dlog4j.configurationFile=/opt/storm1.0.5/log4j2/cluster.xml org.apache.storm.daemon.supervisor.Supervisor

[root@localhost bin]# 
[root@localhost bin]# jps
2256 Supervisor
2344 Jps
[root@localhost bin]# 
三个节点都启动起来了
下来看看ui:


由于我没改三个storm的节点的hostname 所以分不出来具体是哪个






















  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值