mac下kafka环境搭建 测试

kafka介绍：https://blog.csdn.net/see_you_see_me/article/details/78468108

1、安装工具brew install kafka 会自动安装依赖zookeeper

2、安装配置文件位置 /usr/local/etc/kafka|zookeeper

3、启动 zookeeper

cd /usr/local/Cellar/kafka/0.10.0.1

./bin/zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties &

4、启动kafka服务

./bin/kafka-server-start /usr/local/etc/kafka/server.properties &

5、创建topic 

./bin/kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test1

6、查看创建的topic

./bin/kafka-topics --list --zookeeper localhost:2181

7、生产数据

./bin/kafka-console-producer --broker-list localhost:9092 --topic test1

8、消费数据

./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test1--from-beginning

下面给出一个python代码演示demo，首先保证你已经安装python-kafak。

创建生产者：procucer.py

from kafka import KafkaProducer
import time
import logging
import sys

logging.basicConfig(
    stream=sys.stdout,
    level=logging.INFO,
    format='%(asctime)s.%(msecs)d %(filename)s[line:%(lineno)d] %(levelname)s: %(message)s',
    datefmt='%Y/%m/%d %H:%M:%S'
)

log = logging.getLogger('kafka demo')


def list_file():
    while True:
        log.info("===>>>running")
        time.sleep(2)
        producer.send('world',key=b'foo', value=b'bar')
        producer.flush()


if __name__ == '__main__':
    producer = KafkaProducer(bootstrap_servers='localhost:9092')
    list_file()
    producer.close()

创建消费者：consumer.py

# -*- encoding=utf-8 -*-
from kafka import KafkaConsumer
import logging
import sys

logging.basicConfig(
    stream=sys.stdout,
    level=logging.INFO,
    format='%(asctime)s.%(msecs)d %(filename)s[line:%(lineno)d] %(levelname)s: %(message)s',
    datefmt='%Y/%m/%d %H:%M:%S'
)

log = logging.getLogger('kafka demo')


if __name__ == '__main__':
    consumer = KafkaConsumer('world', group_id='consumer-20171017', bootstrap_servers=['localhost:9092'])
    for msg in consumer:
        recv = "%s:%d:%d: key=%s value=%s" % (msg.topic, msg.partition, msg.offset, msg.key, msg.value)
        log.info(recv)

然后，分别运行这2个文件就可以模拟了。

完整源代码：https://github.com/sujeek/python_data_stract/tree/master/src/pds/python_kafka/py_kafka_demo1

参考：

1. https://kafka-python.readthedocs.io/en/master/index.html

2.https://github.com/dpkp/kafka-python/blob/master/docs/index.rst