Kafka 单机部署

Kafka可以当作队列使用，这里记录一下单机环境下，简单的部署过程。

准备环境

1. 下载

wget http://mirrors.hust.edu.cn/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz

2. 解压到

/opt/apache/home/kafka_2.11-1.0.0

3. 设置环境变量

# Kafka

export KAFKA_HOME=/opt/apache/home/kafka_2.11-1.0.0

export KAFKA_LOG_DIR=/opt/apache/logs/kafka

4. 编辑配置文件

$KAFKA_HOME/config/server.properties

# The id of the broker. This must be set to a unique integer for each broker. broker.id=0 # 每个节点不同数字 # A comma seperated list of directories under which to store log files log.dirs=/opt/apache/logs/kafka # The address the socket server listens on. It will get the value returned from # java.net.InetAddress.getCanonicalHostName() if not configured. #   FORMAT: #     listeners = security_protocol://host_name:port #   EXAMPLE: #     listeners = PLAINTEXT://your.host.name:9092 listeners = PLAINTEXT://:9092 # The default number of log partitions per topic. More partitions allow greater # parallelism for consumption, but this will also result in more files across # the brokers. num.partitions=2 # Zookeeper connection string (see zookeeper docs for details). # This is a comma separated host:port pairs, each corresponding to a zk # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002". # You can also append an optional chroot string to the urls to specify the # root directory for all kafka znodes. zookeeper.connect=localhost:2181

---

启动

[root@izwz94kk4bpmj71fa7immvz kafka_2.11-1.0.0]# bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

[root@izwz94kk4bpmj71fa7immvz kafka_2.11-1.0.0]# bin/kafka-server-start.sh -daemon config/server.properties

[root@izwz94kk4bpmj71fa7immvz kafka_2.11-1.0.0]#

[root@izwz94kk4bpmj71fa7immvz kafka_2.11-1.0.0]#

查看端口9092是否监听

[root@izwz94kk4bpmj71fa7immvz kafka_2.11-1.0.0]# netstat -an | grep 9092

tcp        0      0 0.0.0.0:9092            0.0.0.0:*               LISTEN     

tcp        0      0 192.168.145.240:9092    192.168.145.240:45762   ESTABLISHED

tcp        0      0 192.168.145.240:45762   192.168.145.240:9092    ESTABLISHED

查看端口topic、并创建

[root@izwz94kk4bpmj71fa7immvz kafka_2.11-1.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181 --list

[root@izwz94kk4bpmj71fa7immvz kafka_2.11-1.0.0]#

[root@izwz94kk4bpmj71fa7immvz kafka_2.11-1.0.0]#

[root@izwz94kk4bpmj71fa7immvz kafka_2.11-1.0.0]#

[root@izwz94kk4bpmj71fa7immvz kafka_2.11-1.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181 --topic test-topic1 --replication-factor 1 --partitions 1 --create

Created topic "test-topic1".

[root@izwz94kk4bpmj71fa7immvz kafka_2.11-1.0.0]#

[root@izwz94kk4bpmj71fa7immvz kafka_2.11-1.0.0]#

[root@izwz94kk4bpmj71fa7immvz kafka_2.11-1.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181 --list

test-topic1

[root@izwz94kk4bpmj71fa7immvz kafka_2.11-1.0.0]# bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic test-topic1

Topic:test-topic1    PartitionCount:1    ReplicationFactor:1    Configs:

    Topic: test-topic1    Partition: 0    Leader: 1    Replicas: 1    Isr: 1

---

测试

生产消息：

[root@izwz94kk4bpmj71fa7immvz ~]# /opt/apache/home/kafka_2.11-1.0.0/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test-demo

>No.1 abaaaaaaaaaaaaaaaaaaaaaaaaaaaa

>No.2 ddddddddddddddddddddddddddddddd

>No.3 sssssssssssssssssssssssssssssss

>^C[root@izwz94kk4bpmj71fa7immvz ~]#

[root@izwz94kk4bpmj71fa7immvz ~]# /opt/apache/home/kafka_2.11-1.0.0/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test-demo

>aaaaaaaaaaaaaaaaaaaaaaaa

>^C[root@izwz94kk4bpmj71fa7immvz ~]#

消费消息：

[root@izwz94kk4bpmj71fa7immvz kafka_2.11-1.0.0]# /opt/apache/home/kafka_2.11-1.0.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-demo --from-beginning

[2018-01-16 14:43:46,914] WARN [Consumer clientId=consumer-1, groupId=console-consumer-15592] Error while fetching metadata with correlation id 2 : {test-demo=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2018-01-16 14:43:47,039] WARN [Consumer clientId=consumer-1, groupId=console-consumer-15592] Error while fetching metadata with correlation id 4 : {test-demo=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

No.1 abaaaaaaaaaaaaaaaaaaaaaaaaaaaa

No.2 ddddddddddddddddddddddddddddddd

No.3 sssssssssssssssssssssssssssssss

aaaaaaaaaaaaaaaaaaaaaaaa

^CProcessed a total of 4 messages

[root@izwz94kk4bpmj71fa7immvz kafka_2.11-1.0.0]#

搞定。