Python操作Kafka新手教程

一、本地安装与启动
下载zookeeper镜像与kafka镜像：

docker pull wurstmeister/zookeeper
docker pull wurstmeister/kafka

.本地启动zookeeper

docker run -d --name zookeeper -p 2181:2181 -t wurstmeister/zookeeper  

本地启动kafka

docker run -d --name kafka --publish 9092:9092 --link zookeeper \
--env KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
--env KAFKA_ADVERTISED_HOST_NAME=localhost \
--env KAFKA_ADVERTISED_PORT=9092 \
wurstmeister/kafka:latest 

注意：上述代码，将kafka启动在9092端口

进入kafka bash

docker exec -it kafka bash
cd /opt/kafka/bin

创建Topic，分区为2，Topic name为’kafka_demo’

kafka-topics.sh --create --zookeeper zookeeper:2181 \
--replication-factor 1 --partitions 2 --topic kafka_demo

查看当前所有topic

kafka-topics.sh --zookeeper zookeeper:2181 --list

安装kafka-python

pip install kafka-python

生产者和消费者的简易Demo：

from kafka import KafkaProducer, KafkaConsumer
from kafka.errors import kafka_errors
import traceback
import json


def producer_demo():
    # 假设生产的消息为键值对（不是一定要键值对），且序列化方式为json
    producer = KafkaProducer(
        bootstrap_servers=['localhost:9092'], 
        key_serializer=lambda k: json.dumps(k).encode(),
        value_serializer=lambda v: json.dumps(v).encode())
    # 发送三条消息
    for i in range(0, 3):
        future = producer.send(
            'kafka_demo',
            key='count_num',  # 同一个key值，会被送至同一个分区
            value=str(i),
            partition=1)  # 向分区1发送消息
        print("send {}".format(str(i)))
        try:
            future.get(timeout=10) # 监控是否发送成功           
        except kafka_errors:  # 发送失败抛出kafka_errors
            traceback.format_exc()


def consumer_demo():
    consumer = KafkaConsumer(
        'kafka_demo', 
        bootstrap_servers=':9092',
        group_id='test'
    )
    for message in consumer:
        print("receive, key: {}, value: {}".format(
            json.loads(message.key.decode()),
            json.loads(message.value.decode())
            )
        )

这里建议起两个terminal，或者两个jupyter notebook页面来验证。

先执行消费者：
consumer_demo()
再执行生产者：
producer_demo()
会看到如下输出：

>>> producer_demo()
send 0
send 1
send 2

>>> consumer_demo()
receive, key: count_num, value: 0
receive, key: count_num, value: 1
receive, key: count_num, value: 2

参考：https://zhuanlan.zhihu.com/p/279784873