kafka学习

Kafka架构

producer：生产者，生产馒头（厨师）
consumer：消费者，就是吃馒头的（顾客）
broker：篮子
topic：主题,给馒头带一个标签，topic的馒头是给一号顾客吃的，topicb的馒头是给二号顾客吃的

• Kafka is run as a cluster on one or more servers that can span multiple datacenters.
• The Kafka cluster stores streams of records in categories called topics.
• Each record consists of a key, a value, and a timestamp.

Kafka部署及使用

• zookeeper安装
• 单节点单Broker部署及使用
• 单节点多Broker部署及使用

zookper安装

>tar -zxvf zookeeper-3.4.5-cdh5.7.0.tar.gz -C ~/app
>cd cpp/
>cd zookeeper-3.4.5-cdh5.7.0/
>pwd
/home/hadoop/app/zookeeper-3.4.5-cdh5.7.0
>vim ~/.bash_profile
>source ~/.bash_profile
>cd conf/
>cp zoo_sample.cfg zoo.cfg
>vi zoo.cfg
>cd /app
>mkdir tmp/zk
>cd %ZK_HOME
>cd bin
>./zkServer.sh start 
>jps
4361 OuorumPeerMain
4383 jps
>./zkCli.sh

bash_profile文件配置

zoo.cfg文件配置

Quickstart Kafka单节点Broker的部署及使用

Step1:Download the Code

>tar -zxvf kafka_2.12-2.3.0.tgz -C ~/app
>cd kafka_2.12-2.3.0
>vi ~/.bash_profile
>source ~/.bash_profile
>cd config/
>vim server.properties
>cd /app/tmp
>mkidr kafka-logs
>pwd

Step2:Start the server

声明：kafka使用Zookeeper，如果没有，就需要先启动Zookeeper服务器。

>bin/zookeeper-server-start.sh config/zookeeper.properties
[2013-04-22 15:01:37,495] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
...

现在启动Kafka服务器：

> bin/kafka-server-start.sh config/server.properties
[2013-04-22 15:01:47,028] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2013-04-22 15:01:47,051] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
...

Step3：Create a topic

首先用一个分区和一个副本来创建一个名为“test”的主题

> bin/kafka-topics.sh --create --zookeeper 192.168.0.230:2181 --replication-factor 1 --partitions 1 --topic hello_topic

运行list topic命令，便可以看到该主题

> bin/kafka-topics.sh --list --zookeeper 192.168.0.230:2181
hello_topic
kafka_streaming_topic
my-replicated-topic
streamingtopic

Step4:Send some messags

>bin/kafka-console-producer.sh --broker-list 192.168.0.230:9092 --topic hello_topic

Step5:Start a consumer

> bin/kafka-console-consumer.sh --zookeeper 192.168.0.230:2181 --topic hello_topic --from-beginning
This is a message
This is another message

查看所有topic详细信息

kafka-topics.sh describe --zookeeper 192.168.0.230:2181 --topic hello_topic

QuickStart Kafka单节点多broker部署及使用

First we make a config file for each of the brokers

>cp config/server.properties config/server-1.properties

config/server-1.properties:
	broker.id=1
	listeners=PLAINTTEXT://9093
	log.dir=/tmp/kafka-logs-1

Second start the new nodes:

>kafka-server-start.sh -daemon $KAFKA_HOME/config/server-1.properties &
>kafka-server-start.sh -daemon $KAFKA_HOME/config/server-2.properties &
>kafka-server-start.sh -daemon $KAFKA_HOME/config/server-3.properties &

成功开启Kafka后界面

then create a new topic with a replication factor of three

>kafka-topics.sh --create --zookeeper 192.168.0.230:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic

查看topic详细信息

kafka-topics.sh --describe --zookeeper 192.168.0.230:2181

Let’s publish a few message to our new topic:

>kafka-console-producer.sh --broker-list 192.168.0.230:9093,192.168.0.230:9094,192.168.0.230:9095 --topic my-replicated-topic

kafka-console-consumer.sh --zookeeper 192.168.0.230:2181 --topic my-replicated-topic --from-beginning