kafka操作4

kafka-eagle安装

Mysql安装

安装mysql

kafka环境准备

1. 关闭kafka集群
2. 修改/opt/kafka/bin/kafka-server-start.sh命令中的内存

if [ "x$KAFKA HEAP OPTS" = "x"]; then
	export KAFKA_HEAP_OPTS="-server -Xms2G -Xmx2G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=5 -XX:InitiatingHeap0ccupancyPercent=70"
	export JMX_PORT="9999"
	#export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fi

3. 分发到其他节点，启动kafka。

kafka-eagle安装

0. 官网
1. 下载efak并解压
2. 修改conf文件夹下的文件vi system-config.properties

######################################
# multi zookeeper & kafka cluster list
# Settings prefixed with 'kafka.eagle.' will be deprecated, use 'efak.' instead
######################################
efak.zk.cluster.alias=cluster1
cluster1.zk.list=hadoop103:2181,hadoop102:2181,hadoop103:2181/kafka

######################################
# kafka sqlite jdbc driver address
######################################
#efak.driver=com.mysql.jdbc.Driver
#efak.url=jdbc:mysql://hadoop101:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBe
havior=convertToNull
#efak.username=root
#efak.password=root

# 环境变量设置
#EFAK_HOME
export KE_HOME=/opt/efak
export PATH=$PATH:$KE_HOME/bin

启动efak的bin目录下的ke.sh即可
然后访问提示的网址

http://192.168.106.101:8048
账号admin
密码123456

kafka-kraft模式

1. 重新解压一份kafka，进入/config/kraft目录，修改server.properties文件
   

# The node id associated with this instance's roles
node.id=1

# The connect string for the controller quorum
controller.quorum.voters=1@hadoop101:9093,2@hadoop102:9093,3@hadoop103:9093

# Listener name, hostname and port the broker will advertise to clients.
# If not set, it uses the value for "listeners".
advertised.listeners=PLAINTEXT://hadoop101:9092


############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/opt/logs/kafka2

2. 其他两台机子亦如此。
3. 初始化集群数据目录

- 首先生成存储目录唯一ID
 bin/kafka-storage.sh random-uuid
- 用该ID格式化kafka存储目录（三台节点）
bin/kafka-storage.sh format -t m6Xa9I9gQ2ybqg7SksaLHQ -c /opt/kafka2/config/kraft/server.properties

4. 启动kafka/opt/kafka2/bin/kafka-server-start.sh -daemon /opt/kafka2/config/kraft/server.properties
5. 生产者bin/kafka-console-producer.sh --bootstrap-server hadoop101:9092 --topic first ，消费者bin/kafka-console-consumer.sh --bootstrap-server hadoop101:9092 --topic first
6. 群起脚本

#!/bin/bash

case $1 in
"start")
        for i in hadoop101 hadoop102 hadoop103
        do
                echo "--- 启动 $i kafka ---"
                ssh $i "source /etc/profile;/opt/kafka2/bin/kafka-server-start.sh -daemon /opt/kafka2/config/kraft/server.properties"
        done
;;
"stop")
        for i in hadoop101 hadoop102 hadoop103
        do
                echo "--- 停止 $i kafka ---"
                ssh $i "/opt/kafka2/bin/kafka-server-stop.sh"
        done
;;
esac