Ubuntu系统伪分布式kafka集群搭建

伪分布式kafka集群搭建

1、Kafka简介

Kafka是一个分布式、支持分区的（partition）、多副本的（replica），基于zookeeper协调的分布式消息系统，它的最大的特性就是可以实时的处理大量数据以满足各种需求场景：比如基于hadoop的批处理系统、低延迟的实时系统、storm/Spark流式处理引擎，web/nginx日志、访问日志，消息服务等等，用scala语言编写，Linkedin于2010年贡献给了Apache基金会并成为顶级开源 项目。

2、zookeeper集群搭建

2.1、定义目录结构及伪分布式结点

• mkdir -p /usr/local/src/zookeeper #创建zookeeper路径
• mkdir -p /usr/local/src/zookeeper/server1 #结点1
• mkdir -p /usr/local/src/zookeeper/server2 #结点2
• mkdir -p /usr/local/src/zookeeper/server3 #结点3

2.2、下载并解压zookeeper

cd /usr/local/src/zookeeper进入zookeeper目录
输入以下命令进行zookeeper-3.5.6的下载

wget https://archive.apache.org/dist/zookeeper/zookeeper-3.5.6/apache-zookeeper-3.5.6-bin.tar.gz

下载完成后解压到指定结点目录

tar zxvf  apache-zookeeper-3.5.6-bin.tar.gz -C server1
tar zxvf  apache-zookeeper-3.5.6-bin.tar.gz -C server2
tar zxvf  apache-zookeeper-3.5.6-bin.tar.gz -C server3

2.3、创建相关目录及分配zookeeper结点id

切换到结点1目录cd /usr/local/src/zookeeper/server1
mkdir -p data #保存数据的目录
mkdir -p logs #写数据的日志文件保存在这个目录里
进入data目录并为该结点指定一个id，如1

cd data
vim myid

另外俩个进行相同操作

2.4、修改配置文件

cd /usr/local/src/zookeeper/server1/apache-zookeeper-3.5.6-bin/conf #进入配置目录
cp zoo_sample.cfg zoo_sample.cfg.bak #备份默认配置文件
mv zoo_sample.cfg zoo.cfg #zookeeper的启动配置文件
vim zoo.cfg #修改配置文件

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/usr/local/src/zookeeper/server1/apache-zookeeper-3.5.6-bin/data
dataLogDir=/usr/local/src/zookeeper/server1/apache-zookeeper-3.5.6-bin/logs
# the port at which the clients will connect
clientPort=2181
server.1=127.0.0.0:2888:3888
server.2=127.0.0.1:2889:3889
server.3=127.0.0.1:2890:3890
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

其他俩结点的配置修改基本一致注意路径以及端口即可

2.5、启动zookeeper集群

/usr/local/src/zookeeper/server1/apache-zookeeper-3.5.6-bin/bin/zkServer.sh start  
/usr/local/src/zookeeper/server2/apache-zookeeper-3.5.6-bin/bin/zkServer.sh start
/usr/local/src/zookeeper/server3/apache-zookeeper-3.5.6-bin/bin/zkServer.sh start

3、kafka集群搭建

3.1、kafka的下载与解压

• mkdir -p /usr/local/src/kafka 创建kafka目录路径
• 进入所创建的kafka目录中
• 输入以下命令进行kafka_2.13-2.6.0.tgz的下载

wget https://archive.apache.org/dist/kafka/2.6.0/kafka_2.13-2.6.0.tgz

• tar zxvf kafka_2.13-2.6.0.tgz #解压kafka

3.2、kafka集群配置

3.2.1、配置前准备

• cd /usr/local/src/kafka/kafka_2.13-2.6.0/config #进入kafka配置目录
• cp server.properties server.properties.bak #备份kafka默认配置文件
• cp server.properties server-1.properties #创建kafka-1的broker
• cp server.properties server-2.properties #创建kafka-2的broker

3.2.2、修改配置

vim server.properties 	# 修改kafka的server配置文件
vim server-1.properties	# 修改kafka的server1配置文件
vim server-2.properties	# 修改kafka的server2配置文件

配置内容基本一致，端口id以及日志存放路径根据不同的broker分别配置

broker.id=0       #broker的id

host.name=192.168.193.128   #本机名或者ip

port=9092               #每个id内Kakfa服务器监听的端口

log.dirs=/usr/local/src/kafka/kafka_2.13-2.6.0/logs/log-0   #日志存放路径

zookeeper.connect=192.168.193.128:2181                #zookeeper连接端口

3.3、启动kafka集群

cd /usr/local/src/kafka/kafka_2.13-2.6.0 #进入kafka目录
输入以下命令后台启动kafka

bin/kafka-server-start.sh -daemon config/server.properties
bin/kafka-server-start.sh -daemon config/server-1.properties
bin/kafka-server-start.sh -daemon config/server-2.properties

3.4、测试

• 创建一个topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

• 查看以创建的topic

bin/kafka-topics.sh --list --zookeeper localhost:2181

• 启动生产者，发送消息

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test 

• 启动消费者，接受消息

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

• 删除主题

bin/kafka-topics.sh --delete --zookeeper localhost -topic test