部署storm统计服务开启zookeeper(ACL认证)

部署storm统计服务开启zookeeper(ACL认证)

  • 当前部署环境服务版本
服务IP模块版本信息
10.130.41.42zookeeperzookeeper-3.6.3
10.130.41.43kafkakafka_2.11-2.3.1
10.130.41.44stormapache-storm-0.9.6
zookeeper部署
  • 部署mongodb_1服务器的zookeeper
  • 传安装部署包和配置文件
[root@mongodb_1 ~]# su - storm
[storm@mongodb_1 ~]$ rz -be  ====>  mongodb_1 ~]$ rz -be  storm_node1.tar.gz
[storm@mongodb_1 ~]$ tar xvf storm_node1.tar.gz 
[storm@mongodb_1 ~]$ cd  storm_node1
[storm@mongodb_1 storm_node1]$ mv * .bash_profile  ../
[storm@mongodb_1 ~]$ source .bash_profile ;java -version 
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
  • 修改zookeeper配置文件,然后在启动
[storm@mongodb_1 ~]$ cd zookeeper-3.6.3/conf/
[storm@mongodb_1 conf]$ vim zoo.cfg 
dataDir=/home/storm/zookeeper-3.6.3/data
dataLogDir=/home/storm/zookeeper-3.6.3/log
server.1:mongodb_1:2182:3181
server.2:mongodb_2:2182:3181
server.3:mongodb_3:2182:3181
#peerType=observer
autopurge.purgeInterval=1
autopurge.snapRetainCount=3
4lw.commands.whitelist=*
[storm@mongodb_1 ~]$ mkdir -p /home/storm/zookeeper-3.6.3/log /home/storm/zookeeper-3.6.3/data
[storm@mongodb_1 ~]$ echo "1" >  /home/storm/zookeeper-3.6.3/data/myid 
[storm@mongodb_1 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_1 bin]$ ./zkServer.sh  start 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

#需要登录三台zookepper启动完成之后,在查看状态
[storm@mongodb_1 bin]$ ./zkServer.sh  status 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
  • 部署mongodb_2服务器的zookeeper
  • 上传安装部署包和配置文件
[root@mongodb_2 ~]# su - storm
[storm@mongodb_2 ~]$ rz -be  ====>  mongodb_1 ~]$ rz -be storm_node2.tar.gz
[storm@mongodb_2 ~]$ tar xvf storm_node2.tar.gz 
[storm@mongodb_2 ~]$ cd  storm_node2
[storm@mongodb_2 storm_node1]$ mv * .bash_profile  ../
[storm@mongodb_2 ~]$ source .bash_profile ;java -version 
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
  • 修改zookeeper配置文件,然后在启动
[storm@mongodb_2 ~]$ cd zookeeper-3.6.3/conf/
[storm@mongodb_2 conf]$ vim zoo.cfg 
dataDir=/home/storm/zookeeper-3.6.3/data
dataLogDir=/home/storm/zookeeper-3.6.3/log
server.1:mongodb_1:2182:3181
server.2:mongodb_2:2182:3181
server.3:mongodb_3:2182:3181
#peerType=observer
autopurge.purgeInterval=1
autopurge.snapRetainCount=3
4lw.commands.whitelist=*
[storm@mongodb_2 ~]$ mkdir -p /home/storm/zookeeper-3.6.3/log /home/storm/zookeeper-3.6.3/data
[storm@mongodb_2 ~]$ echo "2" >  /home/storm/zookeeper-3.6.3/data/myid 
[storm@mongodb_2 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_2 bin]$ ./zkServer.sh  start 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

#需要登录三台zookepper启动完成之后,在查看状态
[storm@mongodb_2 bin]$ ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader
  • 部署mongodb_3服务器的zookeeper
  • 上传安装部署包和配置文件
[root@mongodb_3 ~]# su - storm
[storm@mongodb_3 ~]$ rz -be  ====>  mongodb_1 ~]$ rz -be storm_node3.tar.gz
[storm@mongodb_3 ~]$ tar xvf storm_node3.tar.gz 
[storm@mongodb_3 ~]$ cd  storm_node3
[storm@mongodb_3 storm_node1]$ mv * .bash_profile  ../
[storm@mongodb_3 ~]$ source .bash_profile ;java -version 
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
  • 修改zookeeper配置文件,然后在启动
[storm@mongodb_3 ~]$ cd zookeeper-3.6.3/conf/
[storm@mongodb_3 conf]$ vim zoo.cfg 
dataDir=/home/storm/zookeeper-3.6.3/data
dataLogDir=/home/storm/zookeeper-3.6.3/log
server.1:mongodb_1:2182:3181
server.2:mongodb_2:2182:3181
server.3:mongodb_3:2182:3181
#peerType=observer
autopurge.purgeInterval=1
autopurge.snapRetainCount=3
4lw.commands.whitelist=*
[storm@mongodb_3 ~]$ mkdir -p /home/storm/zookeeper-3.6.3/log /home/storm/zookeeper-3.6.3/data
[storm@mongodb_3 ~]$ echo "3" >  /home/storm/zookeeper-3.6.3/data/myid 
[storm@mongodb_3 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_3 bin]$ ./zkServer.sh  start 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

#需要登录三台zookepper启动完成之后,在查看状态
[storm@mongodb_3 bin]$ ./zkServer.sh  status 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
  • zookeeper设置用户密码,并设置连接zookeeper白名单。
[storm@mongodb_2 bin]$ ./zkCli.sh 
[zk: localhost:2181(CONNECTED) 0] addauth digest admin:123456  #admin为用户名, 123456为密码。
[zk: localhost:2181(CONNECTED) 1] setAcl / auth:admin:123456:cdrwa  
[zk: localhost:2181(CONNECTED) 2] getAcl /
'digest,'admin:0uek/hZ/V9fgiM35b0Z2226acMQ=
: cdrwa

[storm@mongodb_2 bin]$ ./zkCli.sh  #登录连接成功后每次连接需要输入   addauth digest admin:123456
[zk: localhost:2181(CONNECTED) 0] ls /
Insufficient permission : /
[zk: localhost:2181(CONNECTED) 1]  addauth digest admin:123456
[zk: localhost:2181(CONNECTED) 2] ls /
[test, zookeeper]

[zk: localhost:2181(CONNECTED) 8] setAcl / ip:10.130.41.42:cdrwa,ip:10.130.41.43:cdrwa,ip:10.130.41.44:cdrwa,ip:10.130.41.86:cdrwa,ip:10.130.41.119:cdrwa,ip:127.0.0.1:cdrwa,auth:kafka:cdrwa  #设置连接zookeeper白名单
初始化storm配置
  • 需要三个随便找一个zookeeper进行登录
[storm@mongodb_1 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_1 bin]$ ./zkCli.sh 
[zk: localhost:2181(CONNECTED) 2]   addauth digest admin:123456
create  /storm config
create  /storm/config 1
create /umg_cloud cluster
create /umg_cloud/cluster 1
create /umg_cloud/cfg  1 
create /umg_cloud/cfg/notify  1
create  /storm/config/ENT_CALL_RECORD_LIST “”
create  /storm/config/KAFKA_BROKER_LIST mongodb_3:9092
create  /storm/config/KAFKA_ZOOKEEPER_CONNECT cmongodb_1:2181
create  /storm/config/MAX_SPOUT_PENDING 100
create  /storm/config/MONGO_ADMIN admin
create  /storm/config/MONGO_BLOCKSIZE 100
create  /storm/config/MONGO_HOST mongodb_1:30000
create  /storm/config/MONGO_NAME ""
create  /storm/config/MONGO_PASSWORD ""
create  /storm/config/MONGO_POOLSIZE 100
create  /storm/config/MONGO_SWITCH 0
create  /storm/config/OFF_ON_DTS 0
create  /storm/config/REDIS_HOST mongodb_3:27381
create  /storm/config/REDIS_PASSWORD  ""
create  /storm/config/SEND_MESSAGE 0
create  /storm/config/TOPO_COMPUBOLT_NUM 4
create  /storm/config/TOPO_MONGOBOLT_NUM 16
create  /storm/config/TOPO_SPOUT_NUM 3
create  /storm/config/TOPO_WORK_NUM 2
create  /storm/config/ENTID_MESSAGE  1
create /storm/config/MESSAGE_TOPIC  1

kafka部署
  • 部署mongodb_1服务器的kafka
[storm@mongodb_1 ~]$ cd ~/kafka_2.11-0.8.2.2/config/
broker.id=0
host.name=mongodb_1
log.dirs=/home/storm/kafka_2.11-0.8.2.2/log
zookeeper.connect=mongodb_1:2181,mongodb_2:2181,mongodb_3:2181
zookeeper.connection.timeout.ms=30000
default.replication.factor=2
delete.topic.enable=true
log.cleaner.dedupe.buffer.size = 250000000
# 启用ACL用户密码认证
zookeeper.set.acl=true #开启zookeeperacl认证
zookeeper.defaultAcl=root;123456:rwcd #连接zookeeper用户和密码
[storm@mongodb_1 ~]$mkdir /home/storm/kafka_2.11-0.8.2.2/log -p 
[storm@mongodb_1 ~]$cd  /home/storm/kafka_2.11-0.8.2.2/
[storm@mongodb_1 ~]$ bin/kafka-server-start.sh -daemon config/server.properties
  • 部署mongodb_2服务器的kafka
[storm@mongodb_2 ~]$ cd ~/kafka_2.11-0.8.2.2/config/
broker.id=1
host.name=mongodb_2
log.dirs=/home/storm/kafka_2.11-0.8.2.2/log
zookeeper.connect=mongodb_1:2181,mongodb_2:2181,mongodb_3:2181
zookeeper.connection.timeout.ms=30000
default.replication.factor=2
delete.topic.enable=true
log.cleaner.dedupe.buffer.size = 250000000
# 启用ACL用户密码认证
zookeeper.set.acl=true #开启zookeeperacl认证
zookeeper.defaultAcl=root;123456:rwcd #连接zookeeper用户和密码
[storm@mongodb_2 ~]$ -p /home/storm/kafka_2.11-0.8.2.2/log
[storm@mongodb_2 ~]$cd  /home/storm/kafka_2.11-0.8.2.2/
[storm@mongodb_2 ~]$ bin/kafka-server-start.sh -daemon config/server.properties
  • 部署mongodb_3服务器的kafka
[storm@mongodb_3 ~]$ cd ~/kafka_2.11-0.8.2.2/config/
broker.id=2
host.name=mongodb_3
log.dirs=/home/storm/kafka_2.11-0.8.2.2/log
zookeeper.connect=mongodb_1:2181,mongodb_2:2181,mongodb_3:2181
zookeeper.connection.timeout.ms=30000
default.replication.factor=2
delete.topic.enable=true
log.cleaner.dedupe.buffer.size = 250000000
# 启用ACL用户密码认证
zookeeper.set.acl=true #开启zookeeperacl认证
zookeeper.defaultAcl=root;123456:rwcd #连接zookeeper用户和密码
[storm@mongodb_3 ~]$ -p /home/storm/kafka_2.11-0.8.2.2/log
[storm@mongodb_3 ~]$cd  /home/storm/kafka_2.11-0.8.2.2/
[storm@mongodb_3 ~]$ bin/kafka-server-start.sh -daemon config/server.properties
  • 初始化创建topics,在其中一台kafka上面操作即可
[storm@mongodb_1 kafka_2.11-0.8.2.2]$  cd kafka_2.11-0.8.2.2/
[storm@mongodb_1 kafka_2.11-0.8.2.2]$  bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic agentStateDetail

[storm@mongodb_1 kafka_2.11-0.8.2.2]$  bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic call_detail

[storm@mongodb_1 kafka_2.11-0.8.2.2]$  bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic chatLog

[storm@mongodb_1 kafka_2.11-0.8.2.2]$  bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic ivr_message

[storm@mongodb_1 kafka_2.11-0.8.2.2]$  bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic new_r_ags_e

[storm@mongodb_1 kafka_2.11-0.8.2.2]$  bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic t_srvappraise

[storm@mongodb_1 kafka_2.11-0.8.2.2]$  bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic ent_record_bx_table

[storm@mongodb_1 kafka_2.11-0.8.2.2]$  bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic ent_record_fastdfs_url
storm部署
  • 部署mongodb_1服务器的storm
[storm@mongodb_1 ~]$ cd ~/apache-storm-0.9.6/conf/
[storm@mongodb_1 conf]$ vim storm.yaml 
storm.zookeeper.servers:
    - "mongodb_1"
    - "mongodb_2"
    - "mongodb_3"
nimbus.host: "mongodb_1"
ui.port: 8888
storm.local.dir: "/home/storm/apache-storm-0.9.6/data"
supervisor.slots.ports:
   - 6700
   - 6701
   - 6702
   - 6703
   - 6704
   - 6705
   - 6706
storm.zookeeper.authProvider.1: org.apache.zookeeper.server.auth.DigestAuthenticationProvider
storm.zookeeper.auth.digest.1: admin=123456

[storm@mongodb_1 ~]$  mkdir -p /home/storm/apache-storm-0.9.6/data
[storm@mongodb_1 ~]$  cd  /home/storm/apache-storm-0.9.6/bin
[storm@mongodb_1 bin]$  nohup ./storm nimbus &
[storm@mongodb_1 bin]$  nohup ./storm ui &
[storm@mongodb_1 bin]$ jps  #查看是否正常运行
12770 Jps
12519 core
12473 nimbus
8602 QuorumPeerMain
11007 Kafka
  • 部署mongodb_2服务器的storm
[storm@mongodb_2 ~]$ cd ~/apache-storm-0.9.6/conf/
[storm@mongodb_2 conf]$ vim storm.yaml 
 storm.zookeeper.servers:
     - "mongodb_1"
     - "mongodb_2"
     - "mongodb_3"

 nimbus.host: "mongodb_1"
 storm.local.dir: "/home/storm/apache-storm-0.9.6/data"
 supervisor.slots.ports:
     - 6700
     - 6701
     - 6702
     - 6703
     - 6704
     - 6705
     - 6706
storm.zookeeper.authProvider.1: org.apache.zookeeper.server.auth.DigestAuthenticationProvider
storm.zookeeper.auth.digest.1: admin=123456
[storm@mongodb_2 ~]$  mkdir -p /home/storm/apache-storm-0.9.6/data
[storm@mongodb_2 ~]$  cd  /home/storm/apache-storm-0.9.6/bin
[storm@mongodb_2 bin]$ nohup ./storm supervisor &
[storm@mongodb_2 bin]$ jps  #查看是否正常运行
21075 supervisor
20968 Kafka
20794 QuorumPeerMain
21150 Jps
  • 部署mongodb_3服务器的storm
[storm@mongodb_3 ~]$ cd ~/apache-storm-0.9.6/conf/
[storm@mongodb_3 conf]$ vim storm.yaml 
 storm.zookeeper.servers:
     - "mongodb_1"
     - "mongodb_2"
     - "mongodb_3"

 nimbus.host: "mongodb_1"
 storm.local.dir: "/home/storm/apache-storm-0.9.6/data"
 supervisor.slots.ports:
     - 6700
     - 6701
     - 6702
     - 6703
     - 6704
     - 6705
     - 6706
storm.zookeeper.authProvider.1: org.apache.zookeeper.server.auth.DigestAuthenticationProvider
storm.zookeeper.auth.digest.1: admin=123456

[storm@mongodb_3 ~]$  mkdir -p /home/storm/apache-storm-0.9.6/data
[storm@mongodb_3 ~]$  cd  /home/storm/apache-storm-0.9.6/bin
[storm@mongodb_3 bin]$ nohup ./storm supervisor &
[storm@mongodb_3 bin]$ jps #查看是否正常运行
20962 Kafka
21074 supervisor
20791 QuorumPeerMain
21147 Jps
  • storm的主节点nimbus在第一台上启动拓扑图
  • 启动拓扑图时需要指定stormStatics.jar 包,如果使用新storm安装包没有,请自行找测试部要stormStatics.jar包
[storm@mongodb_1 bin]$ cd /home/storm/apache-storm-0.9.6/bin

[storm@mongodb_1 bin]$ ./storm jar ../lib/stormStatics.jar com.channelsoft.stormStatics.topology.BxRecordTopology BxRecordTopology mongodb_1,mongodb_2,mongodb_3 2181 ent_record_bx_table /brokers 

[storm@mongodb_1 bin]$ ./storm jar ../lib/stormStatics.jar com.channelsoft.stormStatics.topology.FastdfsUrlTopology FastdfsUrlTopology mongodb_1,mongodb_2,mongodb_3 2181 ent_record_fastdfs_url /brokers

[storm@mongodb_1 bin]$ ./storm jar ../lib/stormStatics.jar com.channelsoft.stormStatics.topology.TSrvappraiseTopology TSrvappraiseTopology mongodb_1,mongodb_2,mongodb_3 2181 t_srvappraise /brokers

[storm@mongodb_1 bin]$ ./storm jar ../lib/stormStatics.jar com.channelsoft.stormStatics.topology.RAGSETopology RAGSETopology mongodb_1,mongodb_2,mongodb_3 2181 new_r_ags_e /brokers 

[storm@mongodb_1 bin]$ ./storm jar ../lib/stormStatics.jar com.channelsoft.stormStatics.topology.IVRMESSAGETopology IVRMESSAGETopology mongodb_1,mongodb_2,mongodb_3 2181 ivr_message /brokers

[storm@mongodb_1 bin]$ ./storm jar ../lib/stormStatics.jar com.channelsoft.stormStatics.topology.CallDetailTopology CallDetailTopology mongodb_1,mongodb_2,mongodb_3 2181 call_detail /brokers

[storm@mongodb_1 bin]$ ./storm jar ../lib/stormStatics.jar com.channelsoft.stormStatics.topology.AgentStateDetailTopology  AgentStateDetailTopology mongodb_1,mongodb_2,mongodb_3 2181 agentStateDetail  /brokers
  • 使用storm-ui查看刚启动拓扑是否正常
  • 访问第一台的服务器ip端口是8888
  • 10
    点赞
  • 7
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

平凡的运维之路

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值