ELK 通过 Logtsash 收集 TCP/UDP 日志

帮助文档：https://www.elastic.co/guide/en/logstash/5.6/input-plugins.html

一、修改 logstash 配置文件

root@web1:~# vim /etc/logstash/conf.d/tcplog-to-es.conf
input {
  tcp {
    host => "10.0.0.37"
    port => "8899"
    type => "tcplog"
  }
}

output {
  stdout {}
}

二、加载指定配置文件，并验证端口是否起来

# 停止服务
root@web1:~# systemctl stop logstash

# 加载指定配置文件
root@web1:~# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tcplog-to-es.conf

# 验证8899端口是否起来
root@web1:~# ss -ntl
State                    Recv-Q                    Send-Q                                             Local Address:Port                                         Peer Address:Port
LISTEN                   0                         128                                                127.0.0.53%lo:53                                                0.0.0.0:*
LISTEN                   0                         128                                                      0.0.0.0:22                                                0.0.0.0:*
LISTEN                   0                         128                                                    127.0.0.1:6010                                              0.0.0.0:*
LISTEN                   0                         128                                                    127.0.0.1:6012                                              0.0.0.0:*
LISTEN                   0                         128                                                    127.0.0.1:6013                                              0.0.0.0:*
LISTEN                   0                         128                                                      0.0.0.0:80                                                0.0.0.0:*
LISTEN                   0                         128                                                         [::]:22                                                   [::]:*
LISTEN                   0                         128                                                        [::1]:6010                                                 [::]:*
LISTEN                   0                         128                                                        [::1]:6012                                                 [::]:*
LISTEN                   0                         128                                                        [::1]:6013                                                 [::]:*
LISTEN                   0                         50                                            [::ffff:127.0.0.1]:9600                                                    *:*
LISTEN                   0                         128                                           [::ffff:10.0.0.37]:8899                                                    *:*
LISTEN                   0                         1                                             [::ffff:127.0.0.1]:8005                                                    *:*
LISTEN                   0                         100                                                            *:8080                                                    *:*

三、在其他服务器使用 nc 命令发送一行文字

NetCat 简称 nc，在网络工具中有“瑞士军刀”美誉，其功能实用，是一个简单、可靠的网络工具，可通过 TCP 或 UDP 协议传输读写数据，另外还具有很多其他功能。

[root@centos7 ~]# yum -y install nc
echo "nc test 测试" | nc 10.0.0.37 8899

四、验证 Logstash 是否接收到数据

root@logstash1:~# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tcplog-to-es.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-08-28 16:01:26.107 [main] runner - Starting Logstash {"logstash.version"=>"7.12.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on11.0.10+9 +indy +jit [linux-x86_64]"}
[WARN ] 2021-08-28 16:01:26.474 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-08-28 16:01:27.531 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2021-08-28 16:01:28.301 [Converge PipelineAction::Create<main>] Reflections - Reflections took 31 ms to scan 1 urls, producing 23 keys and 47 values
[INFO ] 2021-08-28 16:01:28.810 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/etc/logstash/conf.d/tcplog-to-es.conf"], :thread=>"#<Thread:0x34a1f739 run>"}
[INFO ] 2021-08-28 16:01:29.397 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.59}
[INFO ] 2021-08-28 16:01:29.516 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2021-08-28 16:01:29.559 [[main]<tcp] tcp - Starting tcp input listener {:address=>"10.0.0.37:8899", :ssl_enable=>false}
[INFO ] 2021-08-28 16:01:29.573 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
      "@version" => "1",
       "message" => "nc test 测试",
          "type" => "tcplog",
          "host" => "10.0.0.20",
    "@timestamp" => 2021-08-28T16:05:09.148Z,
          "port" => 46324
}

五、在其他服务器使用 nc 命令发送一个文件

root@ubuntu1804:~# nc 10.0.0.37 8899 < /var/log/syslog

六、修改配置文件将输出改为 elasticsearch

root@web1:~# vim /etc/logstash/conf.d/tcplog-to-es.conf
input {
  tcp {
    host => "10.0.0.37"
    port => "8899"
    type => "tcplog"
  }
}

output {
if [type] == "tcplog" {
    elasticsearch {
      hosts => ["10.0.0.31:9200"]
      index => "logstash-lck-tcplog-%{+YYYY.ww}"
    }
  }
}

七、检测配置文件语法是否正确

root@web1:~# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tcplog-to-es.conf -t

八、启动服务并验证

# 启动服务
systemctl restart logstash.service

# 测试数据传入899端口
root@ubuntu1804:~# nc 10.0.0.37 8899 < /var/log/syslog

九、创建索引方便查询日志