大数据学习33：logstash 和 kibanba 安装以及ELK的联调记录

logstash 和 kibanba 安装以及ELK的联调记录

1.Logstash安装

[root@hadoop001 elk]# tar -xzvf logstash-6.0.0.tar.gz 
[root@hadoop001 elk]# cd logstash-6.0.0/
[root@hadoop001 logstash-6.0.0]# ll
total 100
drwxr-xr-x. 2 root root  4096 Nov 25 14:55 bin
drwxr-xr-x. 2 root root  4096 Nov 25 14:55 config
-rw-r--r--. 1 root root  2276 Nov 11 03:59 CONTRIBUTORS
drwxr-xr-x. 2 root root  4096 Nov 11 03:59 data
-rw-r--r--. 1 root root  3959 Nov 11 04:02 Gemfile
-rw-r--r--. 1 root root 21265 Nov 11 03:59 Gemfile.jruby-2.3.lock
drwxr-xr-x. 5 root root  4096 Nov 25 14:55 lib
-rw-r--r--. 1 root root   589 Nov 11 03:59 LICENSE
drwxr-xr-x. 4 root root  4096 Nov 25 14:55 logstash-core
drwxr-xr-x. 3 root root  4096 Nov 25 14:55 logstash-core-plugin-api
drwxr-xr-x. 4 root root  4096 Nov 25 14:55 modules
-rw-rw-r--. 1 root root 26953 Nov 11 04:02 NOTICE.TXT
drwxr-xr-x. 3 root root  4096 Nov 25 14:55 tools
drwxr-xr-x. 4 root root  4096 Nov 25 14:55 vendor
[root@hadoop001 logstash-6.0.0]# 

#创建一个日志收集的规则
#input 的 file 的path 要和 nginx 产生日志的路径一致，否则无法读取到。
#filter 中为匹配规则
#output 中为输出到 elasticsearch 的位置，es中存储数据 index 相当于 rdbms 表空间，type相当于 rdbms 表
[root@hadoop001 logstash-6.0.0]# vi logstash-nginx-access-log.conf
input {
    file {
        path => ["/usr/local/nginx/logs/access.log"]
        type => "nginx_access"
        start_position => "beginning"
    }
}


filter {
  grok {
    match => {
      "message" => '%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:time}\] "%{WORD:request_action} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:response} %{NUMBER:bytes} "%{DATA:referrer}" "%{DATA:agent}"'
    }
  }
  date {
    match => [ "time", "dd/MMM/YYYY:HH:mm:ss Z" ]
    locale => en
  }}


output {
  elasticsearch {
        hosts => ["192.168.137.11:9200"]
        index => "logstash-nginx-access-log"
    }
}


这篇文章详细说了如何使用 grok 去将日志匹配进es 
https://www.cnblogs.com/Orgliny/p/5592186.html
注意：
如果logstash 和 nginx 配合的不好，比如生成日志和 match 后面的写的有问题，会在 es 中导入多条记录，比如刷新一次，会导入5次。


[root@hadoop001 logstash-6.0.0]# nohup bin/logstash -f logstash-nginx-access-log.conf  &
[1] 3547
[root@hadoop001 logstash-6.0.0]# nohup: ignoring input and appending output to `nohup.out'

关闭logstash 
[root@hadoop001 logstash-6.0.0]#  kill -9 $(pgrep -f logstash)

查看日志
[root@hadoop001 logstash-6.0.0]# tail -f nohup.out 

这时，我们没有访问nginx的主页，没有产生新记录，所有 es 上是没有任何 index 和数据的。
这里建议手动先创建一个index ，避免自动创建后 number_of_shards 不好修改

curl -XPUT 'http://192.168.137.11:9200/logstash-nginx-access-log' -d '
{
    "settings" : {
        "index" : {
            "number_of_shards" : 3,
"number_of_replicas" : 1
        }
    }
}' -H 'Content-Type: application/json; charset=UTF-8' 

刷新nginx网页，es中会产生新数据。
这时，我们就确认logstash 配置完成。

清理 es 数据，这里直接删 index 
curl -XDELETE 'http://192.168.137.11:9200/logstash-nginx-access-log' 

2.Kibana安装
[root@hadoop001 elk]# tar -xzvf kibana-6.0.0-linux-x86_64.tar.gz
[root@hadoop001 elk]# chown -R root.root kibana-6.0.0-linux-x86_64/
[root@hadoop001 elk]# cd kibana-6.0.0-linux-x86_64
[root@hadoop001 kibana-6.0.0-linux-x86_64]# ll
total 856
drwxr-xr-x.   2 root root   4096 Nov 11 02:50 bin
drwxrwxr-x.   2 root root   4096 Nov 11 02:50 config
drwxrwxr-x.   2 root root   4096 Nov 11 02:50 data
-rw-rw-r--.   1 root root    562 Nov 11 02:50 LICENSE.txt
drwxrwxr-x.   6 root root   4096 Nov 11 02:50 node
drwxrwxr-x. 620 root root  20480 Nov 11 02:50 node_modules
-rw-rw-r--.   1 root root 799543 Nov 11 02:50 NOTICE.txt
drwxrwxr-x.   3 root root   4096 Nov 11 02:50 optimize
-rw-rw-r--.   1 root root    721 Nov 11 02:50 package.json
drwxrwxr-x.   2 root root   4096 Nov 11 02:50 plugins
-rw-rw-r--.   1 root root   4654 Nov 11 02:50 README.txt
drwxr-xr-x.  14 root root   4096 Nov 11 02:50 src
drwxrwxr-x.   5 root root   4096 Nov 11 02:50 ui_framework
drwxr-xr-x.   2 root root   4096 Nov 11 02:50 webpackShims
[root@hadoop001 kibana-6.0.0-linux-x86_64]# cd config/
[root@hadoop001 config]# ll
total 8
-rw-r--r--. 1 root root 4649 Nov 11 02:50 kibana.yml
[root@hadoop001 config]# vi kibana.yml 
server.host: "192.168.137.11"
elasticsearch.url: "http://192.168.137.11:9200"
[root@hadoop001 config]# cd ../
[root@hadoop001 kibana-6.0.0-linux-x86_64]# bin/kibana
  log   [06:14:46.050] [info][status][plugin:kibana@6.0.0] Status changed from uninitialized to green - Ready
  log   [06:14:46.095] [info][status][plugin:elasticsearch@6.0.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [06:14:46.168] [info][status][plugin:console@6.0.0] Status changed from uninitialized to green - Ready
  log   [06:14:46.209] [info][status][plugin:metrics@6.0.0] Status changed from uninitialized to green - Ready
  log   [06:14:46.407] [info][status][plugin:timelion@6.0.0] Status changed from uninitialized to green - Ready
  log   [06:14:46.413] [info][listening] Server running at http://192.168.137.11:5601
  log   [06:14:46.414] [info][status][ui settings] Status changed from uninitialized to yellow - Elasticsearch plugin is yellow
  log   [06:14:51.198] [info][status][plugin:elasticsearch@6.0.0] Status changed from yellow to yellow - No existing Kibana index found
  log   [06:14:52.409] [info][status][plugin:elasticsearch@6.0.0] Status changed from yellow to green - Kibana index ready
  log   [06:14:52.410] [info][status][ui settings] Status changed from yellow to green - Ready

这时候，可以打开 网页看下 kibana

第一次打开kibana需要创建一个默认的 index pattern 

点击 create ，这时其实就是 kibana 去 es 中的索引，匹配索引中存储的记录建一个模板

刷新nginx 网页，会出现数据

一个基础的联调完成