kafka和rabbitmq 的区别,rabbitmq 是针对业务的,kafka是针对大数据和日志
kafka简介:
topic:kafka在逻辑上对生产的数据进行分组保存
分区:为了实现kafka的高性能和高可用,我们可以指定一个topic的分区数量,通常分区数不会高于kafka的节点数
每个分区都会有有一个broker(kafka节点作为leader)
我们至少要为一个分区提供一个备份节点(followor),followor用来备份record的数据
kafka 集群搭建需要依赖zookeeper 集群,请先看上篇zookeeper 搭建
kafka注册到zookeeper,通过zoo 来管理,kafka没有集群管理的功能,通过zoo来数据同步
[root@ansible-11 kafka]# tree
.
├── conf
│ └── server.properties
├── hosts
├── install.kakfa.sh
├── ka1.yaml
├── ka2.yaml
├── ka3.yaml
├── kafka_class.yaml
├── kafka.yaml
├── pkg
│ └── kafka_2.13-2.4.1.tgz
└── test.yaml
[root@ansible-11 kafka]# cat install.kakfa.sh
#!/bin/bash
ansible-playbook -i hosts kafka.yaml
ansible-playbook -i hosts ka1.yaml
ansible-playbook -i hosts ka2.yaml
ansible-playbook -i hosts ka3.yaml
ansible-playbook -i hosts kafka_class.yaml
echo '等待10s再看状态,java程序启动较慢'
sleep 10
ansible-playbook -i hosts test.yaml
测试写入数据
[root@node1 ~]# /usr/local/kafka/bin/kafka-topics.sh -create --zookeeper 10.1.234.110:2181 --partitions 3 --replication-factor 3 --topic magedu
Created topic magedu.
验证topic
[root@node1 ~]# /usr/local/kafka/bin/kafka-topics.sh --describe --zookeeper 10.1.234.110:2181 -topic magedu
Topic: magedu PartitionCount: 3 ReplicationFactor: 3 Configs:
Topic: magedu Partition: 0 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
Topic: magedu Partition: 1 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
Topic: magedu Partition: 2 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
获取所有的topic
[root@node1 ~]# /usr/local/kafka/bin/kafka-topics.sh --list --zookeeper 10.1.234.110:2181
magedu
测试发消息,ctrl+c 退出
[root@node1 ~]# /usr/local/kafka/bin/kafka-console-producer.sh --broker-list 10.1.234.110:9092 --topic magedu
>msg1
>msg2
>msg3
>exit
>^C[root@node1 ~]#
测试收消息,ctrl+c 退出
[root@node1 ~]# /usr/local/kafka/bin/kafka-console-consumer.sh --topic magedu --bootstrap-server 10.1.234.111:9092 --from-beginning
msg1
msg3
msg2
exit
^CProcessed a total of 4 messages
删除topic
[root@node1 ~]# /usr/local/kafka/bin/kafka-topics.sh --delete --zookeeper 10.1.234.110:2181 --topic magedu
Topic magedu is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.