Logstash kafka伪集群高可用部署方案

logstash本身不带高可用部署方案 ，但是可以借助KAFKA的消费者 GROUP，构建Logstash伪集群从而实现高可用
这里对这种方式进行测试

环境准备

这里使用单节点的kafka，并部署两个Logstash 模拟 集群实现 kafka消息消费的高可用。准备两台虚拟机并安装jdk1.8
192.168.195.11 kafka logstash jdk1.8
192.168.196.12 logtash jdk1.8
jdk自行百度安装

Kafka部署

下载安装包并上传
下载kafka 安装包并上传 到虚拟机 192.168.195.11

解压安装包

[root@192 ~]# tar -zvxf kafka_2.12-2.8.1.tgz 
kafka_2.12-2.8.1/
kafka_2.12-2.8.1/LICENSE
kafka_2.12-2.8.1/NOTICE
kafka_2.12-2.8.1/bin/
kafka_2.12-2.8.1/bin/kafka-delete-records.sh
kafka_2.12-2.8.1/bin/trogdor.sh

进入解压目录

[root@192 ~]# ln -s kafka_2.12-2.8.1 kafka
[root@192 ~]# cd kafla
-bash: cd: kafla: No such file or directory
[root@192 ~]# cd kafka
[root@192 kafka]# ll
total 40
drwxr-xr-x. 3 root root  4096 Sep 14  2021 bin
drwxr-xr-x. 3 root root  4096 Sep 14  2021 config
drwxr-xr-x. 2 root root  8192 Apr 11 06:24 libs
-rw-r--r--. 1 root root 14520 Sep 14  2021 LICENSE
drwxr-xr-x. 2 root root   262 Sep 14  2021 licenses
-rw-r--r--. 1 root root   953 Sep 14  2021 NOTICE
drwxr-xr-x. 2 root root    44 Sep 14  2021 site-docs

启动kafka自带zk

[root@192 kafka]# bin/zookeeper-server-start.sh   config/zookeeper.properties 
[2022-04-11 06:29:06,215] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-04-11 06:29:06,216] WARN config/zookeeper.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-04-11 06:29:06,226] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-04-11 06:29:06,226] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-04-11 06:29:06,228] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2022-04-11 06:29:06,228] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2022-04-11 06:29:06,228] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2022-04-11 06:29:06,228] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2022-04-11 06:29:06,233] INFO Log4j 1.2 jmx support found and enabled. (org.apache.zookeeper.jmx.ManagedUtil)
[2022-04-11 06:29:06,247] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-04-11 06:29:06,247] WARN config/zookeeper.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-04-11 06:29:06,248] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-04-11 06:29:06,248] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-04-11 06:29:06,248] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2022-04-11 06:29:06,251] INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2022-04-11 06:29:06,358] INFO Server environment:zookeeper.version=3.5.9-83df9301aa5c2a5d284a9940177808c01bc35cef, built on 01/06/2021 20:03 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-11 06:29:06,358] INFO Server environment:host.name=192.168.0.102 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-11 06:29:06,358] INFO Server environment:java.version=1.8.0_191 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-11 06:29:06,358] INFO Server environment:java.vendor=Oracle Corporation 

启动kafka
使用程序包自带的配置文件启动kafka，配置文件中配置的zk地址就是本机的zk地址，因此不用个修改，

bin/kafka-server-start.sh  -daemon   config/server.properties

也可以使用daemon参数 后台运行服务

[root@192 kafka]# bin/kafka-server-start.sh -daemon  config/server.properties 
[root@192 kafka]# jps
5024 Kafka
4641 QuorumPeerMain
5105 Jps

服务验证
通过监控工具可以连接到zk 和kafka 证明服务启动正常

创建topic并发消息
通过监控工具创建一个 logstashHA 的topic 然后通过kafka消息发送工具 发送几条消息

[root@192 kafka]#  bin/kafka-console-producer.sh --broker-list localhost:9092 --topic  logstashHA
>qweqwe
>1
>2
>3
>4
>4
>5
>5
>6

监控工具可以看到刚发送的消息

Logstash部署

上传程序包并解压
下载程序安装包,上传到服务器，解压，如下

[root@192 ~]# tar -zvxf logstash-6.5.4.tar.gz 
......
[root@192 ~]# mv logstash-6.5.4 logstash
[root@192 ~]# cd logstash
[root@192 logstash]# ll
total 844
drwxr-xr-x.  2 root root   4096 Apr 11 06:55 bin
drwxr-xr-x.  2 root root    142 Apr 11 06:55 config
-rw-r--r--.  1 root root   2276 Dec 18  2018 CONTRIBUTORS
drwxr-xr-x.  2 root root      6 Dec 18  2018 data
-rw-r--r--.  1 root root   4056 Dec 18  2018 Gemfile
-rw-r--r--.  1 root root  21862 Dec 18  2018 Gemfile.lock
drwxr-xr-x.  6 root root     84 Apr 11 06:55 lib
-rw-r--r--.  1 root root  13675 Dec 18  2018 LICENSE.txt
drwxr-xr-x.  4 root root     90 Apr 11 06:55 logstash-core
drwxr-xr-x.  3 root root     57 Apr 11 06:55 logstash-core-plugin-api
drwxr-xr-x.  4 root root     55 Apr 11 06:55 modules
-rw-r--r--.  1 root root 808305 Dec 18  2018 NOTICE.TXT
drwxr-xr-x.  3 root root     30 Apr 11 06:55 tools
drwxr-xr-x.  4 root root     33 Apr 11 06:55 vendor
drwxr-xr-x. 10 root root    205 Apr 11 06:55 x-pack

创建配置文件
配置文件配置 从kafka节点读取消息并在终端打印

[root@192 logstash]# vi config/testHa.conf 
input{
    kafka {
        bootstrap_servers => "192.168.195.11:9092"
        topics => "logstashHA"
        consumer_threads => 1
        decorate_events => true
        group_id => "logstashGroup"
        client_id => "logstashCli"
    }

}
output{
    stdout{}
}

启动logstash

[root@192 logstash]# bin/logstash -f config/testHa.conf 
Sending Logstash logs to /root/logstash/logs which is now configured via log4j2.properties
[2022-04-11T08:10:19,195][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-04-11T08:10:19,213][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.5.4"}

消费消息
再次想kafka发送消息后可以看到logstash消费消息并打印了出来，证明服务正常

高可用测试

部署另一台logstash
按照上述相同步骤在195.12 虚拟机也部署一个logstash,但是 kafka消费者组id不同

[root@192 logstash]# vi config/testHA.conf 
input{
    kafka {
        bootstrap_servers => "192.168.195.11:9092"
        topics => "logstashHA"
        consumer_threads => 1
        decorate_events => true
        group_id => "logstashGroup2"
        client_id => "logstashCli2"
    }

}
output{
    stdout{}
}

启动logstash

[root@192 logstash]# bin/logstash -f config/testHA.conf 

......
Consumer clientId=logstashCli2-0, groupId=logstashGroup2] (Re-)joining group
[2022-05-01T20:24:07,766][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstashCli2-0, groupId=logstashGroup2] Successfully joined group with generation 1
[2022-05-01T20:24:07,776][INFO ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] [Consumer clientId=logstashCli2-0, groupId=logstashGroup2] Setting newly assigned partitions [logstashHA-0]
[2022-05-01T20:24:07,803][INFO ][org.apache.kafka.clients.consumer.internals.Fetcher] [Consumer clientId=logstashCli2-0, groupId=logstashGroup2] Resetting offset for partition logstashHA-0 to offset 25.
[2022-05-01T20:24:07,878][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

两个logstash不同组时消息会同时被二者消费
这个时候 再向 kafka中发送一条消息 发现另个logstash 都有打印


两个logstash 属于统一组时，只有一个节点可以消费到消息
这是因为目前他们属于多个消费组。现在我们修改 192.12虚拟机的配置文件，group ip 和 client_id 参数修改为何195.11相同。 "

[root@192 logstash]# vi config/testHA.conf 
input{
    kafka {
        bootstrap_servers => "192.168.195.11:9092"
        topics => "logstashHA"
        consumer_threads => 1
        decorate_events => true
        group_id => "logstashGroup"
        client_id => "logstashCli"
    }

}
output{
    stdout{}
}

再次启动服务，向kafka发送消息，观察两个logstash 的输出
发送了四条消息 都被11上的logstash 消费

而 12 上的logstash 未做任何输出，没有消费任何消息.

模拟logstash宕机
我们直接kill 掉11 上logstash的进程，

[root@192 ~]# kill -9 5770
[root@192 ~]# 

可以看到11 上的logstash被kill 掉了

再次进行发送消息测试，可以看到kafka的消息被另外一个logstash节点正常消费

如此便实现了logstash的高可用。