logstash入门学习

1、入门示例

1.1、安装

Redhat 平台

rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch
cat > /etc/yum.repos.d/logstash.repo <<EOF
[logstash-5.0]
name=logstash repository for 5.0.x packages
baseurl=http://packages.elasticsearch.org/logstash/5.0/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1
EOF
yum clean all
yum install logstash

yum install -y java-1.8.0-openjdk

1.2Hello World

运行

/usr/share/logstash/bin/logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'

结果

1.3、其他

[root@db01-84-31 conf.d]# /usr/share/logstash/bin/logstash -f logstash.conf
--- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.1 - omit version 2.7.0
--- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.1 - omit version 2.7.1-1
Logstash has a new settings file which defines start up time settings. This file is typically located in $LS_HOME/config or /etc/logstash. If you installed Logstash through a package and are starting it manually please specify the location to this settings file by passing in "--path.settings=/path/.." in the command line options {:level=>:warn}
Failed to load settings file from "path.settings". Aborting... {"path.settings"=>"/usr/share/logstash/config", "exception"=>Errno::ENOENT, "message"=>"No such file or directory - /usr/share/logstash/config/logstash.yml", :level=>:fatal}
[root@db01-84-31 conf.d]# touch /usr/share/logstash/config/logstash.yml
touch: 无法创建"/usr/share/logstash/config/logstash.yml": 没有那个文件或目录
[root@db01-84-31 conf.d]# mkdir /usr/share/logstash/config/
[root@db01-84-31 conf.d]# touch /usr/share/logstash/config/logstash.yml

[root@db01-84-31 conf.d]# cat logstash.conf
input{
  stdin{}
}

filter{}

output{
  stdout{codec=>rubydebug}
}
[root@db01-84-31 conf.d]#

[root@db01-84-31 conf.d]# /usr/share/logstash/bin/logstash -f logstash.conf
--- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.1 - omit version 2.7.0
--- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.1 - omit version 2.7.1-1
Pipeline main started
12345678910
{
    "@timestamp" => 2024-08-21T04:05:46.003Z,
      "@version" => "1",
          "host" => "db01-84-31",
       "message" => "12345678910"
}

2、插件配置

[root@db01-84-31 conf.d]# /usr/share/logstash/bin/logstash-plugin --help
Usage:
    bin/logstash-plugin [OPTIONS] SUBCOMMAND [ARG] ...

Parameters:
    SUBCOMMAND                    subcommand
    [ARG] ...                     subcommand arguments

Subcommands:
    install                       Install a plugin
    uninstall                     Uninstall a plugin
    update                        Update a plugin
    pack                          Package currently installed plugins
    unpack                        Unpack packaged plugins
    list                          List all installed plugins
    generate                      Create the foundation for a new plugin

Options:
    -h, --help                    print help

2.1、input详解

https://elkguide.elasticsearch.cn/logstash/plugins/input/

读取文件(File)

分析网站访问日志应该是一个运维工程师最常见的工作了。所以我们先学习一下怎么用 logstash 来处理日志文件。

配置示例

input {
    file {
        path => ["/var/log/*.log", "/var/log/message"]
        type => "system"
        start_position => "beginning"
    }
}

 elasticsearch {
        hosts => ["192.168.84.71:9200"]
        index => "nginxlog"
        flush_size => 20000
        idle_flush_time => 10
  }