Spark Streaming实时流处理笔记（4）—— 分布式消息队列Kafka

1 Kafka概述

和消息系统类似

1.1 消息中间件

生产者和消费者

1.2 Kafka 架构和概念

• producer:生产者（生产馒头）
• consumer：消费者（吃馒头）
• broker:篮子
• topic : 主题，给馒头带一个标签，（topic-A 给A吃，topic-B 给B吃）
  

2 Kafka 部署及使用（单节点单broker）

• 单节点单 Broker
• 单节点多 Broker
• 多节点多 Broker

2.1 需要先安装 Zookeeper

http://archive-primary.cloudera.com/cdh5/cdh/5/


配置到环境变量，然后刷新

export ZK_HOME=/home/hadoop/apps/zookeeper-3.4.5-cdh5.7.0
export PATH=$PATH:$ZK_HOME/bin

修改 zoo.cfg

启动测试

2.2 安装 Kafka

下载https://kafka.apache.org/downloads

解压，配置到环境变量

export KAFKA_HOME=/home/hadoop/apps/kafka_2.11-0.9.0.0
export PATH=$PATH:$KAFKA_HOME/bin

2.3 配置Kafka

/home/hadoop/apps/kafka_2.11-0.9.0.0/config
server.properties

host.name=node1
log.dirs=/home/hadoop/appsData/kafkaData/kafka-logs
zookeeper.connect=node1:2181

2.4 测试 Kafka

前提首先启动 zookeeper,zkServer.sh start

kafka-server-start.sh $KAFKA_HOME/config/server.properties

创建 topic：指定zk

[hadoop@node1 ~]$ kafka-topics.sh --create --zookeeper node1:2181 --replication-factor 1 --partitions 1 --topic hello_tpoic
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic "hello_tpoic".
[hadoop@node1 ~]$ kafka-topics.sh --list --zookeeper node1:2181
hello_tpoic
[hadoop@node1 ~]$ 

发送消息:指定broker

kafka-console-producer.sh --broker-list node1:9092 --topic hello_topic

启动消费者:指定zk

kafka-console-consumer.sh --zookeeper node1:2181 --topic hello_topic --from-beginning

查看信息

[hadoop@node1 ~]$ kafka-topics.sh --describe --zookeeper node1:2181
Topic:hello_topic	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: hello_topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
Topic:hello_tpoic	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: hello_tpoic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

[hadoop@node1 ~]$ kafka-topics.sh --describe --zookeeper node1:2181 --topic hello_topic
Topic:hello_topic	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: hello_topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
[hadoop@node1 ~]$ 

3 单节点多 broker 的部署

[hadoop@node1 config]$ cp server.properties server-1.properties
[hadoop@node1 config]$ cp server.properties server-2.properties
[hadoop@node1 config]$ cp server.properties server-3.properties

3.1 修改配置文件

server-1.properties

broker.id=1
listeners=PLAINTEXT://:9093
log.dirs=/home/hadoop/appsData/kafkaData/kafka-logs-1

server-2.properties

broker.id=2
listeners=PLAINTEXT://:9094
log.dirs=/home/hadoop/appsData/kafkaData/kafka-logs-2

server-3.properties

broker.id=3
listeners=PLAINTEXT://:9095
log.dirs=/home/hadoop/appsData/kafkaData/kafka-logs-3

3.2 启动

kafka-server-start.sh -daemon $KAFKA_HOME/config/server-1.properties &
kafka-server-start.sh -daemon $KAFKA_HOME/config/server-2.properties &
kafka-server-start.sh -daemon $KAFKA_HOME/config/server-3.properties &

创建 topic

kafka-topics.sh --create --zookeeper node1:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic

[hadoop@node1 config]$ kafka-topics.sh --list --zookeeper node1:2181
hello_topic
hello_tpoic
my-replicated-topic

[hadoop@node1 config]$ kafka-topics.sh --describe --zookeeper node1:2181
Topic:hello_topic	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: hello_topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
Topic:hello_tpoic	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: hello_tpoic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
Topic:my-replicated-topic	PartitionCount:1	ReplicationFactor:3	Configs:
	Topic: my-replicated-topic	Partition: 0	Leader: 3	Replicas: 3,2,1	Isr: 3,2,1
[hadoop@node1 config]$ 

创建生产者

kafka-console-producer.sh --broker-list node1:9093,node1:9094,node1:9095 --topic my-replicated-topic

创建消费者

kafka-console-consumer.sh --zookeeper node1:2181 --topic my-replicated-topic