Linux部署kafka（单节点与集群）

_凌云木_

已于 2024-08-23 21:09:55 修改

阅读量1.9k

点赞数

分类专栏：服务器部署文章标签： kafka linux

于 2021-09-15 19:06:36 首次发布

本文链接：https://blog.csdn.net/jhz13075/article/details/120310251

版权

服务器部署专栏收录该内容

13 篇文章 2 订阅

订阅专栏

一、单节点无ZK部署（假设机器名为 node）

1.1、下载并解压 jdk、kafka，备份配置文件，配置并加载JAVA环境变量

rm -rf ~/jdk-8u181-linux-x64.tar.gz && \
wget -P ~ https://repo.huaweicloud.com/java/jdk/8u181-b13/jdk-8u181-linux-x64.tar.gz && \
 
tar -zxf ~/jdk-8u181-linux-x64.tar.gz -C /opt && \
 
 
echo "
export JAVA_HOME=/opt/jdk1.8.0_181
export JRE_PATH=\${JAVA_HOME}/jre
export CLASSPATH=.:\${JAVA_HOME}/lib:\${JRE_PATH}/lib
export PATH=\${PATH}:\${JAVA_HOME}/bin
" >> /etc/profile && source /etc/profile

rm -rf ~/kafka_2.13-2.8.0.tgz && \
wget -P ~ https://repo.huaweicloud.com/apache/kafka/2.8.0/kafka_2.13-2.8.0.tgz && \

tar -zxf ~/kafka_2.13-2.8.0.tgz -C /opt && \

echo "

export KAFKA_HOME=/opt/kafka_2.13-2.8.0
export PATH=\${PATH}:\${KAFKA_HOME}/bin
" >> /etc/profile && source /etc/profile && \


rm -rf ${KAFKA_HOME}/config/kraft/server_sample.properties && \
cp ${KAFKA_HOME}/config/kraft/server.properties ${KAFKA_HOME}/config/kraft/server_sample.properties

1.2、配置单节点kafka

cp ${KAFKA_HOME}/config/kraft/server_sample.properties ${KAFKA_HOME}/config/kraft/server.properties && \
sed -i "s/#advertised.listeners=PLAINTEXT:\/\/your.host.name:9092/advertised.listeners=PLAINTEXT:\/\/${HOSTNAME}:9092/g" ${KAFKA_HOME}/config/kraft/server.properties

1.3、配置开机启动并启动服务

创建启动脚本，配置kafka开机启动，启动服务

vi /etc/systemd/system/kafka.service

[Unit]
Description=Apache Kafka server (broker)
After=network.target
[Service]
Type=simple
WorkingDirectory=/opt/kafka_2.13-2.8.0
Environment="JAVA_HOME=/opt/jdk1.8.0_181/"
User=root
Group=root
ExecStart=/bin/sh -c "echo `/opt/kafka_2.13-2.8.0/bin/kafka-storage.sh random-uuid` | xargs /opt/kafka_2.13-2.8.0/bin/kafka-storage.sh format --ignore-formatted -c /opt/kafka_2.13-2.8.0/config/kraft/server.properties -t && /opt/kafka_2.13-2.8.0/bin/kafka-server-start.sh /opt/kafka_2.13-2.8.0/config/kraft/server.properties"
ExecStop=/bin/sh -c "/opt/kafka_2.13-2.8.0/bin/kafka-server-stop.sh"
Restart=on-failure
[Install]
WantedBy=multi-user.target



systemctl daemon-reload && \
systemctl enable kafka && \
echo `systemctl start kafka` && \
systemctl status kafka

1.4、测试

创建一个topic,创建失败可能要重启kafka,重启后要重建topic
kafka-topics.sh --create --topic warnRuleTopic --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic VehicleGPSInput --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic VehicleAlarmInput --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic VehicleGPSData --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
kafka-topics.sh --create --topic VehicleAlarmData --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
发送
kafka-console-producer.sh --bootstrap-server localhost:9092 --topic warnRuleTopic
接收
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic warnRuleTopic --from-beginning
查看topic信息
kafka-topics.sh --describe --topic warnRuleTopic --bootstrap-server localhost:9092
删除topic
kafka-topics.sh --delete --topic warnRuleTopic --bootstrap-server localhost:9092

二、集群部署

准备至少三台机器：node1、node2、node3，且机器个数为奇数

以下操作须在每台机器上执行。

2.1、下载并解压 jdk、kafka、zookeeper，备份配置文件，配置并加载JAVA环境变量

rm -rf ~/jdk-8u181-linux-x64.tar.gz ~/kafka_2.13-2.8.0.tgz ~/apache-zookeeper-3.6.3-bin.tar.gz && \
wget -P ~ https://repo.huaweicloud.com/java/jdk/8u181-b13/jdk-8u181-linux-x64.tar.gz && \
wget -P ~ https://repo.huaweicloud.com/apache/kafka/2.8.0/kafka_2.13-2.8.0.tgz && \
wget -P ~ https://repo.huaweicloud.com/apache/zookeeper/zookeeper-3.6.3/apache-zookeeper-3.6.3-bin.tar.gz && \

tar -zxf ~/jdk-8u181-linux-x64.tar.gz -C /opt && \
tar -zxf ~/kafka_2.13-2.8.0.tgz -C /opt && \
tar -zxf ~/apache-zookeeper-3.6.3-bin.tar.gz -C /opt && \


echo "
export JAVA_HOME=/opt/jdk1.8.0_181
export JRE_PATH=\${JAVA_HOME}/jre
export CLASSPATH=.:\${JAVA_HOME}/lib:\${JRE_PATH}/lib
export PATH=\${PATH}:\${JAVA_HOME}/bin

export KAFKA_HOME=/opt/kafka_2.13-2.8.0
export PATH=\${PATH}:\${KAFKA_HOME}/bin

export ZK_HOME=/opt/apache-zookeeper-3.6.3-bin
export PATH=\${PATH}:\${ZK_HOME}/bin
" >> /etc/profile && source /etc/profile && \


rm -rf ${KAFKA_HOME}/config/server_sample.properties && \
cp ${KAFKA_HOME}/config/server.properties ${KAFKA_HOME}/config/server_sample.properties

2.2、配置kafka、zookeeper

不同机器下需修改的配置项说明：

broker.id：kafka集群中的每个broker都有一个唯一的id值用来区分彼此

myid文件和server.myid：zk集群用来发现彼此的一个重要标识

hostname1=node1 && \
hostname2=node2 && \
hostname3=node3 && \
myid=1 && \


cp ${KAFKA_HOME}/config/server_sample.properties ${KAFKA_HOME}/config/server.properties && \
sed -i "s/broker.id=0/broker.id=${myid}/g" ${KAFKA_HOME}/config/server.properties && \
sed -i "s/#listeners=PLAINTEXT:\/\/:9092/listeners=PLAINTEXT:\/\/${HOSTNAME}:9092/g" ${KAFKA_HOME}/config/server.properties && \
sed -i "s/#advertised.listeners=PLAINTEXT:\/\/your.host.name:9092/advertised.listeners=PLAINTEXT:\/\/${HOSTNAME}:9092/g" ${KAFKA_HOME}/config/server.properties && \
sed -i "s/zookeeper.connect=localhost:2181/zookeeper.connect=${hostname1}:2181,${hostname2}:2181,${hostname3}:2181/g" ${KAFKA_HOME}/config/server.properties


rm -rf /opt/zk_data_log && mkdir -p /opt/zk_data_log/data /opt/zk_data_log/log && \

touch /opt/zk_data_log/data/myid && echo "${myid}" > /opt/zk_data_log/data/myid && \

cp ${ZK_HOME}/conf/zoo_sample.cfg ${ZK_HOME}/conf/zoo.cfg && \
sed -i "s/dataDir=\/tmp\/zookeeper/dataDir=\/opt\/zk_data_log\/data/g" ${ZK_HOME}/conf/zoo.cfg && \
sed -i "s/#maxClientCnxns=60/maxClientCnxns=2000/g" ${ZK_HOME}/conf/zoo.cfg && \

echo "dataLogDir=/opt/zk_data_log/log
server.1=${hostname1}:2888:3888
server.2=${hostname2}:2888:3888
server.3=${hostname3}:2888:3888
forceSync=no
4lw.commands.whitelist=*" >> ${ZK_HOME}/conf/zoo.cfg

2.3、配置开机启动并启动服务

zookeeper开机启动

installDir=/opt && \

echo "
[Unit]
Description=Zookeeper Service
After=network.target

[Service]
Type=forking
Environment=\"ZOO_LOG_DIR=${installDir}/zk_data_log/log\"
Environment=\"PATH=${PATH}\"
ExecStart=/bin/sh -c 'zkServer.sh start'
ExecStop=/bin/sh -c 'zkServer.sh stop'
ExecReload=/bin/sh -c 'zkServer.sh restart'

[Install]
WantedBy=multi-user.target
" > /etc/systemd/system/zookeeper.service && systemctl daemon-reload && systemctl enable zookeeper

kafka开机启动

installDir=/opt && \

echo "
[Unit]
Description=Apache Kafka server (broker)
After=network.target zookeeper.service

[Service]
Type=simple
Environment=\"KAFKA_HOME=${KAFKA_HOME}\"
Environment=\"PATH=${PATH}\"

User=root
Group=root
ExecStart=/bin/sh -c 'kafka-server-start.sh \${KAFKA_HOME}/config/server.properties'
ExecStop=/bin/sh -c 'kafka-server-stop.sh'
Restart=on-failure

[Install]
WantedBy=multi-user.target
" > /etc/systemd/system/kafka.service && systemctl daemon-reload && systemctl enable kafka

启动服务

systemctl start zookeeper && \
systemctl start kafka

2.4、运维

#查看zk启动状态 installDir/apache-zookeeper-3.6.3-bin/bin
zkServer.sh status

#连接
zkCli.sh -server 127.0.0.1:2181

删除
/opt/apache-zookeeper-3.6.3-bin/bin/zkCli.sh -server 127.0.0.1:2181
[zk: 127.0.0.1:2181(CONNECTED) 1] ls /
[zk: 127.0.0.1:2181(CONNECTED) 1] deleteall /clickhouse

2. 5、错误解决

ERROR [main:QuorumPeer@1148] - Unable to load database on disk

删除 /tmp/zookeeper , 重新创建data,log,myid 然后重启

2.6、kafka-manager

================= 安装kafka-manager =======================
https://blog.wolfogre.com/posts/kafka-manager-download/
https://github.com/wolfogre/kafka-manager-docker/releases/download/3.0.0.4/cmak-3.0.0.4.zip
https://github.com/wolfogre/kafka-manager-docker/releases/download/2.0.0.2/kafka-manager-2.0.0.2.zip
解压
unzip kafka-manager-2.0.0.2.zip && sudo mv ./kafka-manager-2.0.0.2 /opt
修改配置
cd /opt/kafka-manager-2.0.0.2
vi ./conf/application.conf
kafka-manager.zkhosts="node1:2181,node2:2181,node3:2181"
启动停止脚本
vi km.sh

#! /bin/bash
export JAVA_HOME=/opt/jdk1.8.0_181
export PATH=$JAVA_HOME/bin:$PATH
case $1 in
	"start"){
		echo " -------- Start KafkaManager -------"
		#export ZK_HOSTS="elk-01:2181,elk-02:2181,elk-03:2181"
		echo `rm -rf /opt/kafka-manager-2.0.0.2/RUNNING_PID`
		nohup /opt/kafka-manager-2.0.0.2/bin/kafka-manager -Dconfig.file=/opt/kafka-manager-2.0.0.2/conf/application.conf -Dhttp.port=9000 >/opt/kafka-manager-2.0.0.2/kafka-manager.log 2>&1 &
		sleep 1 && jps
		};;
	"stop"){
		echo " -------- Stop KafkaManager -------"
		#ps -ef | grep ProdServerStart | grep -v grep | awk '{print $2}' | xargs kill
		cat /opt/kafka-manager-2.0.0.2/RUNNING_PID | xargs kill -9
		rm -rf /opt/kafka-manager-2.0.0.2/RUNNING_PID
		};;
	* )
		echo "usage:./km.sh start|stop"
	;;
esac

sudo chmod +x km.sh

启动
./km.sh start
===========================================================================================