Logstash 安装笔记

官方安装文档

采用RPM包安装方式

[visitor@localhost ~]$ yum list logstash

已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.tuna.tsinghua.edu.cn
 * epel: mirrors.yun-idc.com
 * extras: mirrors.tuna.tsinghua.edu.cn
 * updates: mirrors.tuna.tsinghua.edu.cn
错误：没有匹配的软件包可以列出

[visitor@localhost ~]$ sudo vim /etc/yum.repos.d/logstash.repo

[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

[visitor@localhost ~]$ yum list logstash

可安装的软件包
logstash.noarch                                                      1:6.7.2-1                                                       logstash-6.x

[visitor@localhost ~]$ sudo yum -y install logstash

已安装:
  logstash.noarch 1:6.7.2-1                                                                                                                      

完毕！

[visitor@localhost ~]$ logstash 

-bash: logstash: 未找到命令

[visitor@localhost ~]$ sudo vim /etc/profile

export PATH=$PATH:/usr/share/logstash/bin

[visitor@localhost ~]$ source /etc/profile

[visitor@localhost ~]$ logstash 

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults

[visitor@localhost ~]$ logstash -V

logstash 6.7.2

[visitor@localhost ~]$ systemctl status logstash

● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

[visitor@localhost ~]$ sudo systemctl start logstash
[visitor@localhost ~]$ systemctl status logstash

● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)
   Active: active (running) since 二 2019-05-14 11:09:23 CST; 2s ago

[visitor@localhost ~]$  logstash-plugin -h

Subcommands:
    list                          List all installed Logstash plugins
    install                       Install a Logstash plugin
    remove                        Remove a Logstash plugin
    update                        Update a plugin
    pack                          Package currently installed plugins, Deprecated: Please use prepare-offline-pack instead
    unpack                        Unpack packaged plugins, Deprecated: Please use prepare-offline-pack instead
    generate                      Create the foundation for a new plugin
    uninstall                     Uninstall a plugin. Deprecated: Please use remove instead
    prepare-offline-pack          Create an archive of specified plugins to use for offline installation

Options:
    -h, --help                    print help

[visitor@localhost ~]$ logstash-plugin list

logstash-input-jdbc

logstash-output-elasticsearch

[visitor@localhost ~]$ su

密码：

[root@localhost visitor]# logstash -e 'input { stdin { } } output { stdout {} }'

你好 logtash
{
      "@version" => "1",
    "@timestamp" => 2019-05-14T03:20:00.282Z,
       "message" => "你好 logtash",
          "host" => "localhost"
}

logtash mysql数据到elasticsearch

/etc/logstash/conf.d/logstash-mysql-elasticsearch.conf

input {
  jdbc {
    jdbc_driver_library => "/etc/logstash/mysql-connector-java-5.1.47.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/ksy?useUnicode=true&characterEncoding=utf-8"
    jdbc_user => "visitor"
    jdbc_password => "Visitor"
    statement => "SELECT `ID` AS id, `NAME` AS name,  `UPDATE_DATETIME` AS updateDatetime FROM `ksy`.`facility` WHERE `UPDATE_DATETIME` > :sql_last_value"
    lowercase_column_names => "false"
    jdbc_paging_enabled => "true"
    jdbc_page_size => "10000"
    schedule => "*/5 * * * *"

    use_column_value => true
    tracking_column => "updateDatetime"
    tracking_column_type => "timestamp"
    record_last_run => true
    clean_run => false
    last_run_metadata_path => "/etc/logstash/record_last_run/facility.record"
  }
}

filter {
}

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "ksy"
    document_id => "%{id}"
  }
}

/etc/logstash/mysql-connector-java-5.1.47.jar

[visitor@localhost ~]$ sudo mkdir /etc/logstash/record_last_run/

[visitor@localhost ~]$ sudo touch /etc/logstash/record_last_run/facility.record

[visitor@localhost ~]$ sudo chown -R logstash:logstash /etc/logstash/record_last_run/

[visitor@localhost ~]$ sudo ls -al /etc/logstash

drwxr-xr-x.  2 logstash logstash      29 5月  15 12:16 record_last_run

[visitor@localhost ~]$ sudo ls -al /etc/logstash/record_last_run/

-rw-r--r--. 1 logstash logstash   0 5月  15 12:16 facility.record

 手动测试

[root@localhost visitor]# logstash -f /etc/logstash/conf.d/logstash-mysql-elasticsearch.conf 

http://192.168.112.128:9200/_cat/indices?v

health status index uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   ksy   f-jx99TkTISDKoK_RFhn-g   1   1         62            0       48kb           48kb

http://192.168.112.128:9200/ksy/_search

{"took": 35,"timed_out": false,"_shards": {"total": 1,"successful": 1,"skipped": 0,"failed": 0},"hits": {"total": {"value": 62,"relation": "eq"},"max_score": 1,"hits": []}}

开启logtash服务

[visitor@localhost ~]$ sudo systemctl restart logstash

 

配置内存

[visitor@localhost ~]$ sudo vim /etc/logstash/jvm.options

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms500m
-Xmx500m