Logstash与Kibana基础使用(小节2)

---

highlight: a11y-dark

theme: juejin

二、部署logstash：

logstash环境准备及安装：

Logstash是一个开源的数据收集引擎，可以水平伸缩，而且logstash整个ELK当中拥有最多插件的一个组件，其可以接收来自不同来源的数据并统一输出到指定的且可以是多个不同目的地。

logstash1(103)

安装openjdk apt install openjdk-8-jdk -y

安装包：logstash ``` cd /usr/local/src/

安装

dpkg -i logstash-6.8.3.deb logstash帮助 /usr/share/logstash/bin/logstash --help ```

测试logstash：

1.测试标准输入和输出

/usr/share/logstash/bin/logstash -e 'input { stdin{}} output { stdout{ codec => rubydebug }}' ...中间部分省略 [INFO ] 2020-05-08 16:13:07.216 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600} <--Successfully表示成功 hello world! <--输入后、返回值 /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated { "host" => "logstash1", <--拿到了主机名 "message" => "hello world!", <--消息的具体内容 "@timestamp" => 2020-05-08T08:16:06.760Z, <--当前事件的发生时间 "@version" => "1" <--事件版本号，一个事件就是一个ruby对象 } 中文 <--也可以解析中文 { "host" => "logstash1", "message" => "中文", "@timestamp" => 2020-05-08T08:19:38.869Z, "@version" => "1" }

/usr/share/logstash/bin/logstash -e 'input { stdin { type => "syslog" }} output { stdout { codec => rubydebug }}' ...中间部分省略 linux01 /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated { "@version" => "1", "@timestamp" => 2020-05-08T08:28:01.318Z, "message" => "linux01", "type" => "syslog", "host" => "logstash1" } 把配置放到文件目录中 ```

切换目录

cd /etc/logstash/conf.d/

编辑

vim test.conf

输入

input { stdin { type => "syslog" } }

输出

output { stdout { codec => rubydebug } } 检查文件 /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf -t

[INFO ] 2020-05-08 16:47:00.294 [LogStash::Runner] runner - Using config.testandexit mode. Config Validation Result: OK. Exiting Logstash <--表示正常 修改文件 vim test.conf

输入

input { file { path => "/var/log/syslog" startposition => "beginning" statinterval => "3" type => "syslog" } }

输出

output { elasticsearch { hosts => ["http://192.168.37.101:9200"] index => "syslog-37-103-%{+YYYY.MM.dd}" } } 检查文件 /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf -t 启动 /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf 查看文件是否有权限

可以看到权限是640、logstash属于其他、所以没有权限

ll /var/log/syslog -rw-r----- 1 syslog adm 2562408 May 8 17:17 /var/log/syslog 启动服务 systemctl enable logstash systemctl start logstash 需要修改权限

查看日志

tail -f /var/log/logstash/logstash-plain.log ...中间部分省略 [2020-05-08T17:30:02,500][WARN ][filewatch.tailmode.handlers.createinitial] failed to open /var/log/syslog: #<:eacces: permission denied>, ["org/jruby/RubyIO.java:1236:in sysopen'", "org/jruby/RubyFile.java:367:ininitialize'", "org/jruby/RubyIO.java:1155:in `open'"] 编辑启动脚本

修改权限

vim /etc/systemd/system/logstash.service

[Service] User=root <-- Group=root <-- ```

重新加载配置文件、重启服务 systemctl daemon-reload systemctl restart logstash

host1(101)和host2(102)

可以看到一些数据

``` ll /esdata/data/nodes/0/ indices/ node.lock _state/

ll /esdata/data/nodes/0/indices/ drwxr-xr-x 3 elasticsearch elasticsearch 4096 May 8 21:21 ./ drwxr-xr-x 4 elasticsearch elasticsearch 4096 May 8 21:21 ../ drwxr-xr-x 8 elasticsearch elasticsearch 4096 May 8 21:21 NrrrXyllSWmd-Vo3cgEJLA/ ```

host1(101)

安装kibana、安装包：kibana ``` cd /usr/local/src/

安装

dpkg -i kibana-6.8.3-amd64.deb 修改kibana配置信息 vim /etc/kibana/kibana.yml server.port: 5601 <--端口 server.host: "0.0.0.0" <--服务器主机 elasticsearch.hosts: ["http://192.168.37.102:9200"] <--elasticsearch主机(101或102) 启动服务并设为开机启动 systemctl enable kibana systemctl start kibana ```

可以看到一些数据

logstash1(103)

添加一个值、追加到文件最后 echo "111" >> /var/log/syslog

刷新页面后、可以看到新的日志信息、此时就要通过Kibana收集日志

关于sincedb：

``` cat /var/lib/logstash/plugins/inputs/file/.sincedb_f5fdf6ea0ea92860c6a6b2b354bfcbbc 264896 0 2050 2386149 1683639321.210242 /var/log/syslog <--记录了收集文件的inode信息

ll /var/log/syslog -i 264896 -rw-r----- 1 syslog adm 2386149 May 9 21:35 /var/log/syslog <-- ```

2.测试输出到文件

logstash1(103)

```

当前所在目录

pwd /etc/logstash/conf.d

修改配置文件

vim test.conf

输入

input {

stdin {

type => "syslog"

}

file { path => "/var/log/syslog" startposition => "beginning" statinterval => "3" type => "syslog" } }

输出

output {

stdout {

codec => rubydebug

}

elasticsearch { hosts => ["http://192.168.37.101:9200"] index => "syslog-37-103-%{+YYYY.MM.dd}" }

file { path => "/tmp/test.txt" } } ```

重启服务

systemctl restart logstash 添加一个值、追加到文件最后 echo "222" >> /var/log/syslog

刷新页面后、可以看到新的日志信息

ll /tmp/test.txt -rw-r--r-- 1 root root 7086 May 9 22:53 /tmp/test.txt

host1(101)

修改kibana语言为中文

vim /etc/kibana/kibana.yml i18n.locale: "zh-CN"

重启服务

systemctl restart kibana

刷新页面后可看到、页面为中文