接上一篇: 点击打开链接
2.4 kafka搭建
2.4.1搭建zookeeper
这里搭建的zookeeper是伪集群的,在同一台机器上根据不同的端口搭出来的;
dataDir=/opt/server/tomcat/zookeeper/zookeeper01/data
dataLogDir=/opt/server/tomcat/zookeeper/zookeeper01/logs
# the port at which the clients will connect
clientPort=2181
server.02=10.101.111.222:2283:2284
server.03=10.101.111.222:2285:2286
端口分别是2181,2182,2183,
2.4.2搭建kafka
同样解压至软件目录,修改配置文件,在config目录:server.properties
broker.id=1
修改
listeners=PLAINTEXT://:9092
zookeeper.connect=localhost:2183
broker.id是唯一标识;添加zookeeper的集群信息,还有日志存放目录:
log.dirs=/opt/ser****************;
启动:
./kafka-server-start.sh -daemon ../config/server.properties
创建一个
./kafka-topics.sh --create --zookeeper 10.101.218.220:2181,10.101.218.220:2182,10.101.218.220:2183 --replication-factor 2 --partitions 1 --topic shuaige
参考博客:
https://www.cnblogs.com/kevingrace/p/9021508.html
2.3 logstash搭建
在3号机器安装:
同样下载解压至软件目录,给他单独设置一下java_home:
vi logstash-5.6.3/bin/logstash.lib.sh
添加配置:exportJAVA_HOME=/home/qr/jdk1.8.0_151
这里还需要配置要收集的日志和elasticsearch的地址:
input{
file{
path =>"/opt/server/tomcat/logstash/logstash-6.3.0/logs/logstash-plain.log"
start_position => "beginning"
}
}
output{
kafka{
# workers => 2
topic_id => "logstashkafka"
bootstrap_servers => "*:9092,*:9093"
}
}
参考这边博文配置:
https://blog.csdn.net/jiangpeng59/article/details/53750408
./kafka-console-consumer.sh --zookeeper 10.101.218.220:2181 --topic logstashkafka --from-beginning