Ansible 安装 Zookeeper/Kafka集群

运维需求见表：

主机IP	部署服务	部署目录	说明
192.168.21.66	Ansible	inventory:/etc/ansible/hosts roles:/opt/ansible	见我的Ansible系列的其他章节，此处不赘述安装Ansible和Ansible安装JDK
192.168.21.67	Zookeeper,Kafka	/opt/zookeeper-1/{data,logs,inuse} /opt/kafka-1/{inuse,log}	其中/opt/zookeeper-{1,2,3}/inuse为zookeeper-3.4.10目录的软链, /opt/kakfa-{1,2,3}/inuse为kafka_2.11-1.0.0目录的软链。 zookeeper的端口为2181,313326,313327。 kafka的端口为9092。 运行用户为：jiangwei。
192.168.21.68	Zookeeper,Kafka	/opt/zookeeper-2/{data,logs,inuse} /opt/kafka-1/{inuse,log}
192.168.21.69	Zookeeper,Kafka	/opt/zookeeper-3/{data,logs,inuse} /opt/kafka-1/{inuse,log}

 

 

 

 

 

 

 

 

1.关联文档：

• 见我的Ansible学习系列之安装 Ansible
• 见我的Ansible学习系列之Ansible 安装 JDK

2.前提是：部署服务器的jiangwei用户安装了JDK软件

Ansible 安装 Zookeeper 集群

3.vi  /etc/ansible/hosts，内容如下：

[zookeeper1]
192.168.21.67

[zookeeper2]
192.168.21.68

[zookeeper3]
192.168.21.69

[zookeeper:children]
zookeeper1
zookeeper2
zookeeper3
[zookeeper:vars]
ansible_python_interpreter=/usr/bin/python3
ansible_ssh_user=root

[kafka]
192.168.21.67
192.168.21.68
192.168.21.69
[kafka:vars]
ansible_python_interpreter=/usr/bin/python3
ansible_ssh_user=root

4./opt/ansible/zookeeper目录结构：

zookeeper
├── files
│ ├── myid1
│ ├── myid2
│ ├── myid3
│ ├── zoo1.cfg
│ ├── zoo2.cfg
│ ├── zoo3.cfg
│ └── zookeeper-3.4.10.tar.gz
└── tasks
├── main.yml
├── zk1.yml
├── zk2.yml
└── zk3.yml

其中zookeeper-3.4.10.tar.gz为安装包。

4.1）myid1、myid2和myid3的文件内容分别为1、2、3。

4.2）zoo1.cfg内容如下：

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/zookeeper-1/data
dataLogDir=/opt/zookeeper-1/logs
clientPort=2181
server.1=192.168.21.67:31326:31327
server.2=192.168.21.68:31326:31327
server.3=192.168.21.69:31326:31327

备注：zoo2.cfg、zoo3.cfg同理，只需改变dataDir和dataLogDir即可，参考对应的运维需求表的部署目录。

4.3）zk1.yml内容如下：            

- name: Create directory /opt/zookeeper-1
file: path=/opt/zookeeper-1 state=directory owner=jiangwei group=jiangwei
when: ansible_default_ipv4.address == "192.168.21.67"
- name: Unarchive Zookeeper Software
unarchive: src=zookeeper-3.4.10.tar.gz dest=/opt/zookeeper-1 owner=jiangwei group=jiangwei
when: ansible_default_ipv4.address == "192.168.21.67"
- name: copy zoo.cfg
copy: src=zoo1.cfg dest=/opt/zookeeper-1/zookeeper-3.4.10/conf/zoo.cfg owner=jiangwei group=jiangwei
when: ansible_default_ipv4.address == "192.168.21.67"
- name: create directory /opt/zookeeper-1/data
file: path=/opt/zookeeper-1/data state=directory owner=jiangwei group=jiangwei
when: ansible_default_ipv4.address == "192.168.21.67"
- name: create directory /opt/zookeeper-1/logs
file: path=/opt/zookeeper-1/logs state=directory owner=jiangwei group=jiangwei
when: ansible_default_ipv4.address == "192.168.21.67"
- name: set myid
copy: src=myid1 dest=/opt/zookeeper-1/data/myid owner=jiangwei group=jiangwei
when: ansible_default_ipv4.address == "192.168.21.67"
- name: make link
file: path=/opt/zookeeper-1/inuse src=/opt/zookeeper-1/zookeeper-3.4.10 state=link owner=jiangwei group=jiangwei
when: ansible_default_ipv4.address == "192.168.21.67"
- name: start zookeeper
command: sudo su - jiangwei -c "/opt/zookeeper-1/inuse/bin/zkServer.sh start" /bin/bash
when: ansible_default_ipv4.address == "192.168.21.67"

备注：zk2.yml和zk3.yml同理，目录按运维需求而定。main.yml包含zk1.yml、zk2.yml、zk3.yml

5.zkinstall.yml内容如下：

- hosts: zookeeper
roles:
     - zookeeper

6.执行：ansible-playbook  zkinstall.yml即可完成zookeeper集群安装

 

Ansible 安装 kafka 集群

 7.vi  /etc/ansible/hosts，内容如下：

[kafka]
192.168.21.67
192.168.21.68
192.168.21.69
[kafka:vars]
ansible_python_interpreter=/usr/bin/python3
ansible_ssh_user=root

8./opt/ansible/kafka目录结构如下：

kafka
├── files
│ ├── kafka_2.11-1.0.0.tgz
│ ├── server1.properties
│ ├── server2.properties
│ └── server3.properties
└── tasks
├── kf1.yml
├── kf2.yml
├── kf3.yml
└── main.yml

其中kafka_2.11-1.0.0.tgz为安装包。

8.1）server1.properties内容如下：

broker.id=67
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0

replica.fetch.max.bytes=5048576
message.max.byte=5048576
default.replication.factor=1
host.name=192.168.21.67
port=9092
zookeeper.connect=192.168.21.67:2181,192.168.21.68:2181,192.168.21.69:2181
log.dirs=/opt/kafka-1/log
num.partitions=2

备注：server2.properties和server3.properties同理，broker.id、log.dirs按照运维需求而来。

8.2）kf1.yml内容如下：

- name: Create directory /opt/kafka-1
file: path=/opt/kafka-1 state=directory owner=jiangwei group=jiangwei
when: ansible_default_ipv4.address == "192.168.21.67"
- name: Unarchive Kafka Software
unarchive: src=kafka_2.11-1.0.0.tgz dest=/opt/kafka-1 owner=jiangwei group=jiangwei
when: ansible_default_ipv4.address == "192.168.21.67"
- name: make link
file: path=/opt/kafka-1/inuse src=/opt/kafka-1/kafka_2.11-1.0.0 state=link owner=jiangwei group=jiangwei
when: ansible_default_ipv4.address == "192.168.21.67"
- name: copy kafka
copy: src=server1.properties dest=/opt/kafka-1/inuse/config/server.properties owner=jiangwei group=jiangwei
when: ansible_default_ipv4.address == "192.168.21.67"
- name: start kafka-1
command: sudo su - jiangwei -c "/opt/kafka-1/inuse/bin/kafka-server-start.sh -daemon /opt/kafka-1/inuse/config/server.properties"
when: ansible_default_ipv4.address == "192.168.21.67"

备注：kf2.yml和kf3.yml同理，注意目录和配置文件的对应关系。main.yml包含kf1.yml、kf2.yml、kf3.yml。

9.kafkainstall.yml内容如下：

- hosts: kafka
roles:
       - kafka

10.执行：ansible-playbook  kakfkainstall.yml 完成kafka集群部署

验证

 11.在192.168.21.67上发送消息，执行命令：

/opt/kafka-1/inuse/bin/kafka-console-producer.sh --broker-list 192.168.21.67:9092, 192.168.21.68:9092, 192.168.21.69:9092 --topic test

12.在192.168.21.68上接受消息，执行命令：

/opt/kafka-1/inuse/bin/kafka-console-consumer.sh --zookeeper 192.168.21.67:2181, 192.168.21.68:2181, 192.168.21.69:2181 --topic test –from-beginning

转载于:https://www.cnblogs.com/WSQL/p/9965057.html