linux 虚拟机部署kafka+zookeeper单机方案

背景:

1.这里介绍了如何在Linux系统上安装和配置Apache ZooKeeper3.6.4,以及Kafka3.4.1,
2.包括创建目录、下载安装包、修改配置、添加环境变量、启动服务、编写启动和停止脚本。


1、安装zookeeper服务

注:这里使用zookeeper3.6.4版本

前提:zookeeper运行需要jdk环境,所以需要提前安装好jdk环境。


1.安装jdk环境:

注:这里采用了yum安装jdk的方式:

先查询jdk版本,通过yum命令

[root@xionghj /]# yum -y list *openjdk*
.......

注:这里选择的是

[root@xionghj /]# yum install -y java-1.8.0-openjdk-devel.x86_64
.....

查询jdk环境是否生效:

[root@xionghj /]# java -version
openjdk version "1.8.0_412"
OpenJDK Runtime Environment (build 1.8.0_412-b08)
OpenJDK 64-Bit Server VM (build 25.412-b08, mixed mode)


2.安装zookeeper服务:

创建zookeeper运行目录:

[root@xionghj /]# mkdir -p /usr/local/zookeeper
[root@xionghj /]# chmod 777 /usr/local/zookeeper/

下载zookeeper安装包:

[root@xionghj /]#  wget https://archive.apache.org/dist/zookeeper/zookeeper-3.6.4/apache-zookeeper-3.6.4-bin.tar.gz

解压安装包

[root@xionghj /]# mv apache-zookeeper-3.6.4-bin.tar.gz /usr/local/zookeeper/
[root@xionghj /]# cd /usr/local/zookeeper/

[root@xionghj zookeeper]# tar -zxvf apache-zookeeper-3.6.4-bin.tar.gz
.......


3.修改zookeeper的配置:

[root@xionghj zookeeper]# cd zookeeper-3.6.4/conf/

[root@xionghj conf]# cp zoo_sample.cfg zoo.cfg

[root@xionghj conf]# vim zoo.cfg 

修改的内容如下:

dataDir=/usr/local/zookeeper/zookeeper-3.6.4/data

dateLogDir=/usr/local/zookeeper/zookeeper-3.6.4/logs

clientPort=2181

4.添加全局环境变量配置:

编辑环境变量文件

[root@xionghj conf] vim /etc/profile
#尾部添加如下内容
export ZOOKEEPER_HOME=/usr/local/zookeeper/zookeeper-3.6.4/
export PATH=$ZOOKEEPER_HOME/bin:$PATH
export PATH

#退出保存
:wq

重新执行一下

[root@xionghj conf] source /etc/profile


5.启动并测试zookeeper服务:

进入bin目录

[root@xionghj zookeeper-3.6.4]# cd bin/

启动服务

[root@xionghj bin]# zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/zookeeper-3.6.4/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

查看状态

[root@xionghj bin]# zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/zookeeper-3.6.4/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: standalone

查看进程

[root@xionghj bin]# ps -ef | grep zookeeper
root      1070     1  2 14:16 pts/1    00:00:01 java -Dzookeeper.log.dir=/usr/local/zookeeper/zookeeper-3.6.4/bin/../logs -Dzookeeper.log.file=zookeeper-root-server-xionghj.log -Dzookeeper.root.logger=INFO,CONSOLE -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError=kill -9 %p -cp /usr/local/zookeeper/zookeeper-3.6.4/bin/../zookeeper-server/target/classes:/usr/local/zookeeper/zookeeper-3.6.4/bin/../build/classes:/usr/local/zookeeper/zookeeper-3.6.4/bin/../zookeeper-server/target/lib/*.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../build/lib/*.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/zookeeper-prometheus-metrics-3.6.4.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/zookeeper-jute-3.6.4.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/zookeeper-3.6.4.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/snappy-java-1.1.8.4.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/slf4j-reload4j-1.7.36.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/slf4j-api-1.7.36.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/simpleclient_servlet-0.6.0.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/simpleclient_hotspot-0.6.0.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/simpleclient_common-0.6.0.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/simpleclient-0.6.0.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/reload4j-1.2.24.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-transport-native-unix-common-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-transport-native-epoll-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-transport-classes-epoll-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-transport-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-resolver-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-handler-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-common-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-codec-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/netty-buffer-4.1.86.Final.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/metrics-core-3.2.5.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/json-simple-1.1.1.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jline-2.14.6.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jetty-util-ajax-9.4.49.v20220914.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jetty-util-9.4.49.v20220914.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jetty-servlet-9.4.49.v20220914.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jetty-server-9.4.49.v20220914.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jetty-security-9.4.49.v20220914.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jetty-io-9.4.49.v20220914.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jetty-http-9.4.49.v20220914.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/javax.servlet-api-3.1.0.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jackson-databind-2.13.4.2.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jackson-core-2.13.4.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/jackson-annotations-2.13.4.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/commons-cli-1.4.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../lib/audience-annotations-0.13.0.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../zookeeper-*.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../zookeeper-server/src/main/resources/lib/*.jar:/usr/local/zookeeper/zookeeper-3.6.4/bin/../conf: -Xmx1000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/local/zookeeper/zookeeper-3.6.4/bin/../conf/zoo.cfg

关闭服务

[root@xionghj bin]#  zkServer.sh stop

重启服务

[root@xionghj bin]#  zkServer.sh restart

2、安装kafka服务:

1.获取安装包:

需要从官方上获取,https://kafka.apache.org/downloads

选择的版本是kafka-3.4.1。

下载地址:https://archive.apache.org/dist/kafka/3.4.1/kafka_2.12-3.4.1.tgz

[root@xionghj kafka]# wget https://archive.apache.org/dist/kafka/3.4.1/kafka_2.12-3.4.1.tgz

2.创建kafka安装目录:

#用于存放kafka的解压包
[root@xionghj /]# mkdir -p /opt/kafka

#用于存放kafka的数据
[root@xionghj /]# mkdir -p /opt/kafka/kafka_data

#用于存放kafka的日志
[root@xionghj /]# mkdir -p /opt/kafka/kafka_log


3.解压并安装kafka服务:

[root@xionghj kafka]# tar -zxvf kafka_2.12-3.4.1.tgz 


4.修改kafka服务配置:

[root@xionghj kafka]# cd /opt/kafka/kafka_2.12-3.4.1/config

[root@xionghj config ]# vim server.properties

#修改一下参数
broker.id=0

#端口号
port=9092   

#服务器IP地址,修改为自己的服务器IP
host.name=localhost

#日志存放路径,上面创建的目录
log.dirs=/opt/kafka/kafka_log

#zookeeper地址和端口,单机配置部署,localhost:2181
zookeeper.connect=localhost:2181   

5.启动并测试kafka服务:

启动kafka服务

[root@xionghj config ]# /opt/kafka/kafka_2.12-3.4.1/bin/kafka-server-start.sh /opt/kafka/kafka_2.12-3.4.1/config/server.properties
........
[2024-09-05 15:28:13,286] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT://iZwz9ippftvoxv000lpl0tZ:9092, czxid (broker epoch): 25 (kafka.zk.KafkaZkClient)
[2024-09-05 15:28:13,403] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2024-09-05 15:28:13,424] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2024-09-05 15:28:13,462] INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2024-09-05 15:28:13,478] INFO Successfully created /controller_epoch with initial epoch 0 (kafka.zk.KafkaZkClient)
[2024-09-05 15:28:13,485] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.group.GroupCoordinator)
[2024-09-05 15:28:13,494] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
[2024-09-05 15:28:13,519] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2024-09-05 15:28:13,529] INFO Feature ZK node created at path: /feature (kafka.server.FinalizedFeatureChangeListener)
[2024-09-05 15:28:13,533] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2024-09-05 15:28:13,542] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2024-09-05 15:28:13,583] INFO [MetadataCache brokerId=0] Updated cache from existing <empty> to latest FinalizedFeaturesAndEpoch(features=Map(), epoch=0). (kafka.server.metadata.ZkMetadataCache)
[2024-09-05 15:28:13,618] INFO [ExpirationReaper-0-AlterAcls]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2024-09-05 15:28:13,760] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2024-09-05 15:28:13,777] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Enabling request processing. (kafka.network.SocketServer)
[2024-09-05 15:28:13,818] INFO Kafka version: 3.4.1 (org.apache.kafka.common.utils.AppInfoParser)
[2024-09-05 15:28:13,818] INFO Kafka commitId: 8a516edc2755df89 (org.apache.kafka.common.utils.AppInfoParser)
[2024-09-05 15:28:13,818] INFO Kafka startTimeMs: 1725521293814 (org.apache.kafka.common.utils.AppInfoParser)
[2024-09-05 15:28:13,821] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
[2024-09-05 15:28:13,934] INFO [BrokerToControllerChannelManager broker=0 name=alterPartition]: Recorded new controller, from now on will use node iZwz9ippftvoxv000lpl0tZ:9092 (id: 0 rack: null) (kafka.server.BrokerToControllerRequestThread)
[2024-09-05 15:28:13,935] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Recorded new controller, from now on will use node iZwz9ippftvoxv000lpl0tZ:9092 (id: 0 rack: null) (kafka.server.BrokerToControllerRequestThread)

测试kafka服务:

前提开启两个终端,一个生产者,一个消费者

生产者

[root@xionghj ~]# cd /opt/kafka/kafka_2.12-3.4.1/bin/
[root@xionghj bin]# ./kafka-console-producer.sh --broker-list localhost:9092 --topic test
>

这里输入消息

[root@xionghj bin]# ./kafka-console-producer.sh --broker-list localhost:9092 --topic test
>xionghj
>kafak zookeeper
>

消费者

[root@xionghj ~]# cd /opt/kafka/kafka_2.12-3.4.1/bin
[root@xionghj bin]# ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
[2024-09-05 15:31:36,908] WARN [Consumer clientId=console-consumer, groupId=console-consumer-74237] Error while fetching metadata with correlation id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2024-09-05 15:31:37,031] WARN [Consumer clientId=console-consumer, groupId=console-consumer-74237] Error while fetching metadata with correlation id 4 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2024-09-05 15:31:37,191] WARN [Consumer clientId=console-consumer, groupId=console-consumer-74237] Error while fetching metadata with correlation id 6 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

消费者就可以收到消息了

[root@xionghj bin]# ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
[2024-09-05 15:31:36,908] WARN [Consumer clientId=console-consumer, groupId=console-consumer-74237] Error while fetching metadata with correlation id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2024-09-05 15:31:37,031] WARN [Consumer clientId=console-consumer, groupId=console-consumer-74237] Error while fetching metadata with correlation id 4 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2024-09-05 15:31:37,191] WARN [Consumer clientId=console-consumer, groupId=console-consumer-74237] Error while fetching metadata with correlation id 6 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
xionghj
kafak zookeeper


3、开发zookeeper+kafka服务启动停止脚本:

启动脚本:

[root@xionghj / ]# cd /opt/kafka/kafka_2.12-3.4.1/bin

[root@xionghj bin]# vim kafka_start.sh

#内容如下:

#!/bin/bash
#
echo "启动zookeeper服务..."
sh /usr/local/zookeeper/zookeeper-3.6.4/bin/zkServer.sh start
sleep 5
echo "启动kafka服务..."
/opt/kafka/kafka_2.12-3.4.1/bin/kafka-server-start.sh /opt/kafka/kafka_2.12-3.4.1/config/server.properties &

停止脚本:

[root@xionghj / ]# cd /opt/kafka/kafka_2.12-3.4.1/bin

[root@xionghj bin]# vim kafka_start.sh

#!/bin/bash
echo "停止zookeeper服务..."
sh /usr/local/zookeeper/zookeeper-3.6.4/bin/zkServer.sh stop
sleep 5
echo "停止kafka服务..."
/opt/kafka/kafka_2.12-3.4.1/bin/kafka-server-stop.sh /opt/kafka/kafka_2.12-3.4.1/config/server.properties 

设置可执行权限:

[root@xionghj bin]# chmod +x kafka_start.sh 
[root@xionghj bin]# chmod +x kafka_stop.sh 

设置开机自起

[root@xionghj / ]#  vim /etc/rc.d/rc.local 
#设置开机自动在后台运行脚本
bash /opt/kafka/kafka_2.12-3.4.1/kafka_start.sh & 

  • 2
    点赞
  • 4
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

jiang0615csdn

你对鼓励是我最大的动力来源

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值