02-Logstash 输入

Logstash 02

Logstash 01

插件列表

查看所有的输入插件

logstash-plugin list --group input

直接 logstash-plugin list 会列举所有的 logstash 插件

准备工作

自动加载 pipeline 配置

初期测试会频繁修改 pipeline 的配置文件, 可以设置 Logstash 为自动加载

修改 /usr/share/logstash-7.14.0/config/logstash.yml, 具体路径根据 Logstash 安装目录修改

也可以在启动 Logstash 时指定 --config.reload.automatic

logstash -f pipeline.conf --config.reload.automatic

检查 pipeline 配置

logstash -f pipeline.conf --config.test_and_exit

# 也可简写为
logstash -t -f pipeline.conf

每次修改配置后, 可以检查 pipeline.conf 文件是否有误

输入插件

Heartbeat input plugin

新建配置文件 pipeline.conf, 内容如下

input {
	heartbeat {
		interval => 10
	}
}

output {
	stdout { }
}

每 10 秒发送一个 “ok”

运行 Logstash

logstash -f pipeline.conf

Logstash 输出

{
      "@version" => "1",
    "@timestamp" => 2021-08-16T07:54:33.391Z,
          "host" => "DESKTOP-JHX",
       "message" => "ok"
}
{
      "@version" => "1",
    "@timestamp" => 2021-08-16T07:54:43.377Z,
          "host" => "DESKTOP-JHX",
       "message" => "ok"
}

TCP/UDP input plugin

开启一个 TCP/UDP 侦听接口, 接收 TCP/UDP 客户端的输入

pipeline.conf

input {
	tcp {
		port => 7000
		type => "tcp"
	}
	udp {
		port => 9000
		type => "udp"
	}
}

output {
	stdout { }
}

同时包含两种输入, 通过 type 区分

通过 netcat 工具发送 TCP/UDP 报文, 发送完后按 Ctrl + C 退出交互

# TCP
echo -n "Hi" | nc 127.0.0.1 7000
# UDP
echo -n "Hi" | nc -u 127.0.0.1 9000

Logstash 输出

{
    "@timestamp" => 2021-08-16T07:57:44.493Z,
          "host" => "localhost",
          "type" => "tcp",
          "port" => 39484,
      "@version" => "1",
       "message" => "Hi"
}
{
      "@version" => "1",
    "@timestamp" => 2021-08-16T07:57:46.322Z,
          "type" => "udp",
          "host" => "127.0.0.1",
       "message" => "Hi"
}

两条输出的 message 相同, 但 type 不同

File input plugin

读取本地文件的内容作为输入

pipeline.conf

input {
	file {
		path => "/mnt/g/Test.txt"
		start_position => "beginning"
		sincedb_path => "/dev/null"
		codec => json
	}
}

output {
	stdout { }
}

1. start_position

默认为 “end”, 表示从末尾开始读(日志一般追加到末尾), 仅读取以后新增的内容, 设为 “beginning” 可以从头读起

2. sincedb_path

Logstash 使用 sincedb 文件记录上一次读取的位置, 为了方便测试, 将 sincedb 文件的路径设为空, 这样每次启动 Logstash 均会读取整个文件的内容

默认情况下, 文件的每一行对应一个输入事件(delimiter 默认为 “\n”)

当 /mnt/g/ 目录下有一个 Test.txt 待读取文件, 内容如下

{"name": "Hoss", "age": 10}
{"name": "JHX", "age": 100}

注意末尾新增一空行

Logstash 输出

{
          "path" => "/mnt/g/Test.txt",
           "age" => 10,
    "@timestamp" => 2021-08-16T08:06:28.475Z,
          "host" => "DESKTOP-JHX",
          "name" => "Hoss",
      "@version" => "1"
}
{
          "path" => "/mnt/g/Test.txt",
           "age" => 100,
    "@timestamp" => 2021-08-16T08:06:28.495Z,
          "host" => "DESKTOP-JHX",
          "name" => "JHX",
      "@version" => "1"
}

由于 input 部分使用了 codec, 输出中包含 “name” 和 “age” 字段

读取多行

一次读取多行内容需要在输入中使用 multiline codec

1. pattern

“^\[” 表示匹配以 “[” 开头的行

2. negate

为 true 时, 表示不匹配的内容需要整合在一起

3. what

取值为 “next” 或 “previous”, “next” 表示待整合的内容归到下一行, “previous” 与之相反

测试数据 Test.txt

[1]
content 1
[2]
content 2
content 2
[3]
content 3
content 3
content 3

Logstash 配置 pipeline.conf

input {
	file {
		path => "/mnt/g/Test.txt"
		start_position => "beginning"
		sincedb_path => "/dev/null"
		codec => multiline {
			pattern => "^\["
			negate => true
			what => "previous"
		}
	}
}

output {
	stdout { }
}

输出

{
      "@version" => "1",
          "host" => "DESKTOP-JHX",
          "tags" => [
        [0] "multiline"
    ],
          "path" => "/mnt/g/Test.txt",
    "@timestamp" => 2021-08-16T08:34:05.241Z,
       "message" => "[1]\r\ncontent 1\r"
}
{
      "@version" => "1",
          "host" => "DESKTOP-JHX",
          "tags" => [
        [0] "multiline"
    ],
          "path" => "/mnt/g/Test.txt",
    "@timestamp" => 2021-08-16T08:34:05.268Z,
       "message" => "[2]\r\ncontent 2\r\ncontent 2\r"
}

由此可见 “content” 部分的内容被归为上一行, 但是末尾一行没有输出

如果改为 “next”, 输出为

{
      "@version" => "1",
          "host" => "DESKTOP-JHX",
    "@timestamp" => 2021-08-16T08:39:22.149Z,
          "path" => "/mnt/g/Test.txt",
       "message" => "[1]\r"
}
{
      "@version" => "1",
          "host" => "DESKTOP-JHX",
          "tags" => [
        [0] "multiline"
    ],
          "path" => "/mnt/g/Test.txt",
    "@timestamp" => 2021-08-16T08:39:22.151Z,
       "message" => "content 1\r\n[2]\r"
}
{
      "@version" => "1",
          "host" => "DESKTOP-JHX",
          "tags" => [
        [0] "multiline"
    ],
          "path" => "/mnt/g/Test.txt",
    "@timestamp" => 2021-08-16T08:39:22.152Z,
       "message" => "content 2\r\ncontent 2\r\n[3]\r"
}

Exec input plugin

在本地定期执行一个 shell 命令, 将命令的输出作为 Logstash 的输入

pipeline.conf

input {
	exec {
		command => "du -sh /usr/share/logstash-7.14.0"
		interval => 60
	}

}

output {
	stdout { }
}

每分钟获取一次 Logstash 安装目录的大小

Logstash 输出

{
          "host" => "DESKTOP-JHX",
    "@timestamp" => 2021-08-16T08:53:14.900Z,
       "command" => "du -sh /usr/share/logstash-7.14.0",
      "@version" => "1",
       "message" => "646M\t/usr/share/logstash-7.14.0\n"
}