python连接kafka简单案例

docker部署kafka

1.拉取镜像
docker pull wurstmeister/zookeeper

docker pull wurstmeister/kafka

2.启动，先启动zookeeper
docker run -d --name zookeeper -p 2181:2181 -t wurstmeister/zookeeper

再启动kafka
docker run -d --name kafka -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=172.17.137.55（虚拟机地址）:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.17.137.55（虚拟机地址）:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -v /etc/localtime:/etc/localtime wurstmeister/kafka

本地部署kafka

https://blog.csdn.net/Eternal_Blue/article/details/95598942

案例代码如下

python-product

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import datetime
import json
import time
import uuid

from kafka import KafkaProducer
from kafka.errors import KafkaError

producer = KafkaProducer(bootstrap_servers='172.17.137.55:9092')
topic = 'test_20181105'


def test():
    print('begin')
    try:
        n = 0
        while True:
            dic = {}
            dic['id'] = n
            n = n + 1
            dic['myuuid'] = str(uuid.uuid4().hex)
            dic['time'] = datetime.datetime.now().strftime("%Y%m%d %H:%M:%S")
            producer.send(topic, json.dumps(dic).encode())
            print("send:" + json.dumps(dic))
            time.sleep(2)
    except KafkaError as e:
        print(e)
    finally:
        producer.close()
        print('done')


if __name__ == '__main__':
    test()

python-consumer

#!/bin/env python
from kafka import KafkaConsumer

# connect to Kafka server and pass the topic we want to consume
consumer = KafkaConsumer('test_20181105', group_id='test_group2', bootstrap_servers='192.168.29.128:9092')
try:
    for msg in consumer:
        print("consumer" + str(msg))
        # print("%s:%d:%d: key=%s value=%s" % (msg.topic, msg.partition, msg.offset, msg.key, msg.value))
except KeyboardInterrupt as e:
    print(e)

执行consumer控制台打印

执行product控制台打印