Kafka 入门教程之一: 安装

Kafka 入门教程之一: 安装

1. 官网下载软件

http://kafka.apache.org/

2. 安装单节点kafka

a) 创建用户kafka

[root@tjtestrac1 ~]# useradd kafka 
[root@tjtestrac1 ~]# passwd kafka
Changing password for user kafka.
New password: 
BAD PASSWORD: The password is shorter than 8 characters
Retype new password: 
passwd: all authentication tokens updated successfully.

b) 解压软件包

[kafka@tjtestrac1 ~]$ ls
kafka_2.12-2.1.0.tgz
[kafka@tjtestrac1 ~]$ tar -xvf kafka_2.12-2.1.0.tgz

c) 配置环境变量 kafka_home 和 java_home

[kafka@tjtestrac1 ~]$ vi .bash_profile
export KAFKA_HOME=/home/kafka/kafka_2.12-2.1.0
export JAVA_HOME=/u02/cassandra/java/jdk1.8.0_162
PATH=$PATH:$HOME/.local/bin:$HOME/bin:$JAVA_HOME/bin:$KAFKA_HOME/bin [kafka@tjtestrac1 ~]$ source .bash_profile 

3. 启动zookeeper, kafka 组件

a) 为zookeeper 创建文件夹

[kafka@tjtestrac1 config]$ mkdir -p /home/kafka/zk 

b) 修改kafka 内置的zk 配置文件

[kafka@tjtestrac1 config]$ vi zookeeper.properties 
dataDir=/home/kafka/zk
 
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0

c) 启动zk

kafka@tjtestrac1 config]$ zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties & 

d) 查看后台进程

[kafka@tjtestrac1 ~]$ jps
30323 QuorumPeerMain
30660 Jps

4. 启动kafka

a) 为kafka 创建文件夹

[kafka@tjtestrac1 ~]$ mkdir -p /home/kafka/kf

b) 修改kafka 的配置文件

[kafka@tjtestrac1 config]$ vi server.properties 
log.dirs=/home/kafka/kf 

c) 启动kafka 进程

[kafka@tjtestrac1 config]$ kafka-server-start.sh $KAFKA_HOME/config/server.properties & 

d) 查看后台进程

[kafka@tjtestrac1 config]$ jps
30323 QuorumPeerMain
4739 Kafka
5414 Jps

5. 创建一个主题topic

[kafka@tjtestrac1 config]$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

[2018-11-22 17:27:47,517] INFO Accepted socket connection from /127.0.0.1:58228 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2018-11-22 17:27:47,519] INFO Client attempting to establish new session at /127.0.0.1:58228 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-11-22 17:27:47,521] INFO Established session 0x10052ce304e0001 with negotiated timeout 30000 for client /127.0.0.1:58228 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-11-22 17:27:47,802] INFO Got user-level KeeperException when processing sessionid:0x10052ce304e0001 type:setData cxid:0x4 zxid:0x1f txntype:-1 reqpath:n/a Error Path:/config/topics/test Error:KeeperErrorCode = NoNode for /config/topics/test (org.apache.zookeeper.server.PrepRequestProcessor)
Created topic "test".

列出主题:

[kafka@tjtestrac1 config]$ kafka-topics.sh --list --zookeeper localhost:2181
[2018-11-22 17:29:31,251] INFO Accepted socket connection from /0:0:0:0:0:0:0:1:64976 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2018-11-22 17:29:31,253] INFO Client attempting to establish new session at /0:0:0:0:0:0:0:1:64976 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-11-22 17:29:31,255] INFO Established session 0x10052ce304e0002 with negotiated timeout 30000 for client /0:0:0:0:0:0:0:1:64976 (org.apache.zookeeper.server.ZooKeeperServer)
test
[2018-11-22 17:29:31,302] INFO Processed session termination for sessionid: 0x10052ce304e0002 (org.apache.zookeeper.server.PrepRequestProcessor)
[2018-11-22 17:29:31,303] INFO Closed socket connection for client /0:0:0:0:0:0:0:1:64976 which had sessionid 0x10052ce304e0002 (org.apache.zookeeper.server.NIOServerCnxn)

6. 启动生产者来发送数据

[kafka@tjtestrac1 ~]$ kafka-console-producer.sh --broker-list localhost:9092 --topic test
>jason 
>comm on
>test message

7. 启动消费者来接收数据

 [kafka@tjtestrac1 ~]$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

[kafka@tjtestrac1 ~]$ kafka-console-producer.sh --broker-list localhost:9092 --topic test
>jason 
>comm on
>test message 
>next message