1、环境信息
1) 关闭防火前、禁用selinux
systemctl stop firewalld.service
systemctl disable firewalld.service
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
2)打开文件数连接
----------------------------------------------------------------------------------------------------------
echo "* soft noproc 20480" >> /etc/security/limits.conf
echo "* hard noproc 20480" >> /etc/security/limits.conf
echo "root soft nofile 655360" >> /etc/security/limits.conf
echo "root hard nofile 655360" >> /etc/security/limits.conf
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* - memlock unlimited" >>/etc/security/limits.conf
----------------------------------------------------------------------------------------------------------
对于/etc/security/limits.d/20-nproc.conf文件,要么清空,要么添加配置
echo "* soft noproc 20480" > /etc/security/limits.d/20-nproc.conf
echo "* soft nproc unlimited" >> /etc/security/limits.d/20-nproc.conf
或者
sed -i "s#^*##g" /etc/security/limits.d/20-nproc.conf
sed -i "s#^root##g" /etc/security/limits.d/20-nproc.conf
3)优化内核参数(/etc/sysctl.conf)
Kernel sysctl configuration file for Red Hat Linux
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.ip_local_port_range = 10000 65000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 5000
vm.overcommit_memory = 1
vm.max_map_count = 6553600
vm.file-max = 655350
vm.swappiness = 1
保存退出执行下边命令使配置生效
sysctl -p
4)主机信息
4)jdk安装
参考java环境(jdk1.8)安装-linux&windows
2、zookeeper集群搭建(非root用户)
1)下载zookeeper3.4.10
cd /u01/isi/application/
wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/stable/zookeeper-3.4.13.tar.gz
2)解压改配置
tar -xf zookeeper-3.4.10.tar.gz
cd zookeeper-3.4.10/conf
cp zoo_sample.cfg zoo.cfg
cat zoo.cfg #主要修改日志路径和集群ip地址
3)创建zoo.cfg配置文件中的data目录并创建myid文件
mkdir /u01/isi/application/zookeeper-3.4.10/zk_data
在dataDir目录下创建一个myid文件,然后分别在myid文件中按照zoo.cfg文件的server.num中num的数值,在不同机器上的该文件中填写相应的值
如:
cd /u01/isi/application/zookeeper-3.4.10/zk_data
echo '1' > myid #另外两台分别是2和3即可
4)分发安装包到其他节点
for i in 100.0.0.6 100.0.0.7 100.0.0.8
do
scp -r /u01/isi/application/zookeeper-3.4.10 $i:/u01/isi/application/
done
5)在另外2台服务器创建相应的data目录并修改myid文件中的值
6) 启动zookeeper服务
cd /u01/isi/application/zookeeper-3.4.10/bin
./zkServer.sh start #启动zookeeper 服务
./zkServer.sh status #查询zookeeper 服务
./zkServer.sh stop #停止zookeeper 服务
./zkServer.sh restart #重启zookeeper 服务
tailf zookeeper.out #查看zookeeper服务后台日志
3、kafka集群搭建(非root用户)
1)下载kafka
cd /u01/isi/application/
wget https://archive.apache.org/dist/kafka/1.1.0/kafka_2.11-1.1.0.tgz
2)解压改配置
tar -xf kafka_2.11-1.1.0.tgz
cd kafka_2.11-1.1.0/config
cat server.properties
3)配置说明:
broker.id:当前机器在集群中的唯一标识,和zookeeper的myid性质一样
num.network.threads:borker进行网络处理的线程数
num.io.threads:borker进行I/O处理的线程数
socket.send.buffer.bytes:发送缓冲区buffer大小,数据不是一下子就发送的,先会存储到缓冲区到达一定的大小后在发送,能提高性能
socket.receive.buffer.bytes:kafka接收缓冲区大小,当数据到达一定大小后在序列化到磁盘
socket.request.max.bytes:这个参数是向kafka请求消息或者向kafka发送消息的请请求的最大数,这个值不能超过java的堆栈大小
log.dirs: 消息日志存放的路径
num.partitions:默认的分区数,一个topic默认1个分区数
num.recovery.threads.per.data.dir: 每个数据目录用来日志恢复的线程数目
log.retention.hours: 默认消息的最大持久化时间
log.segment.bytes: 这个参数是:因为kafka的消息是以追加的形式落地到文件,当超过这个值的时候,kafka会新起一个文件
log.retention.check.interval.ms: 每隔300000毫秒去检查上面配置的log失效时间
log.cleaner.enable: 是否启用log压缩,一般不用启用,启用的话可以提高性能
zookeeper.connect:设置zookeeper的连接端口
zookeeper.connection.timeout.ms: 设置zookeeper的连接超时时间
4)分发安装包到其他节点,修改另外两个节点的broker.id值
for i in 100.0.0.6 100.0.0.7 100.0.0.8
do
scp -r /u01/isi/application/kafka_2.11-1.1.0 $i:/u01/isi/application/
done
5)kafka集群启动
先启动zookeeper集群,待zookeeper集群启动之后,再启kafka集群
cd /u01/isi/application/kafka_2.11-1.1.0
nohup bin/kafka-server-start.sh config/server.properties &
6)kafka命令行操作
cd /u01/isi/application/kafka_2.11-1.1.0
1> 创建新topic
bin/kafka-topics.sh --create --zookeeper 100.0.0.6:2181,100.0.0.7:2181,100.0.0.8:2181 --replication-factor 3 --partitions 3 --topic TestTopic
选项说明:
–topic 定义topic名
–replication-factor 定义副本数
–partitions 定义分区数
2> 查看topic副本信息
bin/kafka-topics.sh --describe --zookeeper 100.0.0.6:2181,100.0.0.7:2181,100.0.0.8:2181 --topic TestTopic
3> 查看已经创建的topic信息
bin/kafka-topics.sh --list --zookeeper 100.0.0.6:2181,100.0.0.7:2181,100.0.0.8:2181
4> 测试生产者发送消息
bin/kafka-console-producer.sh --broker-list 100.0.0.6:2181,100.0.0.7:2181,100.0.0.8:2181 --topic TestTopic
7)kafka-offset-console的安装(由于远程无法访问,用本地演示)
1> 从本地直接获取kafka-offset-console.tar.gz,并放置于/tmp目录下
获取地址连接:https://github.com/BangpengGao/kafka-offset-monitor
2> 解压
cd /tmp
tar -xf kafka-offset-console -C /u01/isi/application/
cd /u01/isi/application/kafka-offset-console
3> 修改启动脚本并启动服务
cat start.sh
配置说明:
–zk:表示为需要连接的zk集群
–port:表示kafka-offset-console的监听端口
–refresh:10秒刷新一次
–retain:日志保存时长
./start.sh #启动服务
netstat -tanlp | grep 8086
4> 浏览器验证
http://100.0.6.5:8086