Centos7安装部署logstash8.6.2启动CPU异常99%

1，安装部署过程

下载rpm包后使用命令安装

rpm -ivh logstash-8.6.2-x86_64.rpm

会自动新增守护进程，查看守护进程状态，从显示内容可以看到守护进程配置文件为【/usr/lib/systemd/system/logstash.service】

systemctl status logstash

查看文件内容

[root@jenkins-harbor001 ]# cat /usr/lib/systemd/system/logstash.service
[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

# When stopping, how long to wait before giving up and sending SIGKILL?
# Keep in mind that SIGKILL on a process can cause data loss.
TimeoutStopSec=infinity

[Install]
WantedBy=multi-user.target

可以看到logstash的配置目录为【/etc/logstash】，根据业务需求，我上传了logstash.conf到这个目录下。

接着启动logstash服务。正常情况下，logstash刚启动会导致服务器CPU到达99%，一两分钟后就会下降至正常水平。

systemctl start logstash

2，发现异常
logstash一开始能够实现了它的功能，比如把上游kafka的数据写入下游的mongodb数据库，但很快就不再写了。观察发现服务器CPU持续在99%。

3，定位问题原因
查看logstash日志，在默认目录下的日志文件为【/var/log/logstash/logstash-plain.log】，日志不断重复一段报错

[2024-08-12T15:59:15,459][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2024-08-12T15:59:15,464][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.6.2", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.6+10 on 17.0.6+10 +indy +jit [x86_64-linux]"}
[2024-08-12T15:59:15,467][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2024-08-12T15:59:15,487][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:91) ~[?:?]
[2024-08-12T15:59:28,018][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties

百度搜了关键词，找不到正确答案。
于是尝试使用非systemctl的方式启动，而是手动执行命令来启动logstash

/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"

发现是能够正常运行的，CPU一开始升至99%后一两分钟就下降到正常水平，并且不再上升。logstash也能够持续写入下游的mongodb数据库。
于是分析问题原因出在守护配置文件【/usr/lib/systemd/system/logstash.service】，首先执行kill命令停掉刚才手动启动的logstash进程。再将它修改为：

[Unit]
Description=logstash
 
[Service]
Type=simple
User=root
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
 
[Install]
WantedBy=multi-user.target

之后，执行命令

systemctl daemon-reload
systemctl start logstash

问题解决。