Kafka架构
producer:生产者,生产馒头(厨师)
consumer:消费者,就是吃馒头的(顾客)
broker:篮子
topic:主题,给馒头带一个标签,topic的馒头是给一号顾客吃的,topicb的馒头是给二号顾客吃的
- Kafka is run as a cluster on one or more servers that can span multiple datacenters.
- The Kafka cluster stores streams of records in categories called topics.
- Each record consists of a key, a value, and a timestamp.
Kafka部署及使用
- zookeeper安装
- 单节点单Broker部署及使用
- 单节点多Broker部署及使用
zookper安装
>tar -zxvf zookeeper-3.4.5-cdh5.7.0.tar.gz -C ~/app
>cd cpp/
>cd zookeeper-3.4.5-cdh5.7.0/
>pwd
/home/hadoop/app/zookeeper-3.4.5-cdh5.7.0
>vim ~/.bash_profile
>source ~/.bash_profile
>cd conf/
>cp zoo_sample.cfg zoo.cfg
>vi zoo.cfg
>cd /app
>mkdir tmp/zk
>cd %ZK_HOME
>cd bin
>./zkServer.sh start
>jps
4361 OuorumPeerMain
4383 jps
>./zkCli.sh
bash_profile文件配置
zoo.cfg文件配置
Quickstart Kafka单节点Broker的部署及使用
Step1:Download the Code
>tar -zxvf kafka_2.12-2.3.0.tgz -C ~/app
>cd kafka_2.12-2.3.0
>vi ~/.bash_profile
>source ~/.bash_profile
>cd config/
>vim server.properties
>cd /app/tmp
>mkidr kafka-logs
>pwd
Step2:Start the server
声明:kafka使用Zookeeper,如果没有,就需要先启动Zookeeper服务器。
>bin/zookeeper-server-start.sh config/zookeeper.properties
[2013-04-22 15:01:37,495] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
...
现在启动Kafka服务器:
> bin/kafka-server-start.sh config/server.properties
[2013-04-22 15:01:47,028] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2013-04-22 15:01:47,051] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
...
Step3:Create a topic
首先用一个分区和一个副本来创建一个名为“test”的主题
> bin/kafka-topics.sh --create --zookeeper 192.168.0.230:2181 --replication-factor 1 --partitions 1 --topic hello_topic
运行list topic命令,便可以看到该主题
> bin/kafka-topics.sh --list --zookeeper 192.168.0.230:2181
hello_topic
kafka_streaming_topic
my-replicated-topic
streamingtopic
Step4:Send some messags
>bin/kafka-console-producer.sh --broker-list 192.168.0.230:9092 --topic hello_topic
Step5:Start a consumer
> bin/kafka-console-consumer.sh --zookeeper 192.168.0.230:2181 --topic hello_topic --from-beginning
This is a message
This is another message
查看所有topic详细信息
kafka-topics.sh describe --zookeeper 192.168.0.230:2181 --topic hello_topic
QuickStart Kafka单节点多broker部署及使用
First we make a config file for each of the brokers
>cp config/server.properties config/server-1.properties
config/server-1.properties:
broker.id=1
listeners=PLAINTTEXT://9093
log.dir=/tmp/kafka-logs-1
Second start the new nodes:
>kafka-server-start.sh -daemon $KAFKA_HOME/config/server-1.properties &
>kafka-server-start.sh -daemon $KAFKA_HOME/config/server-2.properties &
>kafka-server-start.sh -daemon $KAFKA_HOME/config/server-3.properties &
成功开启Kafka后界面
then create a new topic with a replication factor of three
>kafka-topics.sh --create --zookeeper 192.168.0.230:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
查看topic详细信息
kafka-topics.sh --describe --zookeeper 192.168.0.230:2181
Let’s publish a few message to our new topic:
>kafka-console-producer.sh --broker-list 192.168.0.230:9093,192.168.0.230:9094,192.168.0.230:9095 --topic my-replicated-topic
kafka-console-consumer.sh --zookeeper 192.168.0.230:2181 --topic my-replicated-topic --from-beginning