在centos6上搭建ELK平台

概述

ELK是 elasticsearch + logstash + kibana 的简称。这套组合类似于MVC模型，其中logstash是controller层，数据首先传给它，它负责将数据进行过滤和格式化；elasticsearch是model层，负责数据的存储，建立搜索索引；kibana是view层，负责数据的展示。
所以使用中会涉及到以下几个知识 O(∩_∩)O~
elasticsearch
logstash
当然，如果熟悉ruby就更开心了：）
好了，说了一大堆，开始干活吧 : D

环境

系统：centos 6.x
软件版本：
elasticsearch-5.6.0
kibana-5.6.0
logstash-5.6.0
下载到本地:
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.0.rpm
wget https://artifacts.elastic.co/downloads/logstash/logstash-5.6.0.rpm
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.6.0-x86_64.rpm

安装软件

直接rpm安装

rpm -ivh elasticsearch-5.6.0.rpm
rpm -ivh logstash-5.6.0.rpm
rpm -ivh kibana-5.6.0-x86_64.rpm

配置elasticsearch

主要配置文件/etc/elasticsearch/elasticsearch.yml，添加如下内容

cluster.name: erp #指定个集群名字，等下会用到
node.name: node0 #节点名称也指定下
path.data: /opt/data #指定数据的存放路径
path.logs: /opt/data/logs #指定日志的存放路径，好排查问题
network.host: 172.16.93.237 #指定监听地址
http.port: 9200 #指定端口

如果没data目录，则新建下

mkdir -p /opt/data/logs
chown -R elasticsearch:elasticsearch /opt/data

启动elasticsearch服务

service elasticsearch start

查看运行状态

service elasticsearch status

curl http://ip:9200 如果有类似如下的输出则是正常启动了

# curl http://172.16.93.237:9200
{
  "name" : "node0",
  "cluster_name" : "erp",
  "cluster_uuid" : "Vy0zvSCRQ-y_nAo9YHHRMQ",
  "version" : {
    "number" : "5.6.0",
    "build_hash" : "781a835",
    "build_date" : "2017-09-07T03:09:58.087Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}

在启动的过程中，可能会遇到些错误，导致无法启动。
错误1

max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]

elasticsearch对系统的文件打开数等有要求，需要支持下

vi /etc/security/limits.conf 
添加或修改为如下内容:
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096

vi /etc/security/limits.d/90-nproc.conf
添加或修改为如下内容:
* soft nproc 2048

错误2

system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

说的好清楚了，关闭system_call_filter

vi /etc/elasticsearch/elasticsearch.yml
添加
bootstrap.system_call_filter: false

错误3

memory locking request for elasticsearch process but memory is not locked

尝试关闭bootstrap.memory_lock试试

vi /etc/elasticsearch/elasticsearch.yml
添加
bootstrap.memory_lock: false

错误4

kernel: Out of memory: Kill process
#表现为elasticsearch运行一会儿自动关闭了

是内存超了，如果你的系统内存不够用，可以限制下elasticsearch的内存开销(默认是2g)

vim /etc/elasticsearch/jvm.options
-Xms2g
-Xmx2g
把这两个数值改小一点,比如
-Xms1g
-Xmx1g

配置logstash

默认没有init.d的脚本，需要执行如下命令生成下

/usr/share/logstash/bin/system-install /etc/logstash/startup.options sysv

logstash的配置文件分三部分：input，filter，output。顾名思义，分别就是输入，过滤，输出了。具体的配置语法可以参考文档。
配置文件放置在/etc/logstash/conf.d/目录下。
在配置时，需结合日志格式来配置。比如用ELK来分析nginx日志，nginx的日志配置如下：

log_format main "$http_x_forwarded_for | $time_local | $request | $status | $body_bytes_sent | "
                "$request_body | $content_length | $http_referer | $http_user_agent | nuid  | "
                "$http_cookie | $remote_addr | $hostname | $upstream_addr | $upstream_response_time | $request_time";

则可以在/etc/logstash/conf.d/目录下新建个nginx-access.conf的配置文件，内容如下：

input {
        file {
                path => "/data/log/nginx/*" #nginx日志的路径
                start_position => "beginning"
                sincedb_path => "/data/log/el/nginx_progress"
        }
}
filter {
    ruby {
        init => "@kname = ['http_x_forwarded_for','time_local','request','status','body_bytes_sent','request_body','content_length','http_referer','http_user_agent','nuid','http_cookie','remote_addr','hostname','upstream_addr','upstream_response_time','request_time']"
        code => "
            new_event = LogStash::Event.new(Hash[@kname.zip(event.get('message').split('|'))])
            new_event.remove('@timestamp')
            event.append(new_event)
        "
    }
    if [request] {
        ruby {
            init => "@kname = ['method','uri','verb']"
            code => "
                new_event = LogStash::Event.new(Hash[@kname.zip(event.get('request').split(' '))])
                new_event.remove('@timestamp')
                event.append(new_event)
            "
        }
        if [uri] {
            ruby {
                init => "@kname = ['url_path','url_args']"
                code => "
                    new_event = LogStash::Event.new(Hash[@kname.zip(event.get('uri').split('?'))])
                    new_event.remove('@timestamp')
                    event.append(new_event)
                "
            }
            kv {
                prefix => "url_"
                source => "url_args"
                field_split => "& "
                remove_field => [ "url_args","uri","request" ]
            }
        }
    }
    mutate {
        convert => [
            "body_bytes_sent" , "integer",
            "content_length", "integer",
            "upstream_response_time", "float",
            "request_time", "float"
        ]
    }
    date {
        match => [ "time_local", "dd/MMM/yyyy:hh:mm:ss Z" ]
        locale => "en"
    }
}
output {
  elasticsearch {
    action => "index"          
    hosts  => "172.16.93.237:9200"
    index  => "erp"
  }
}

logstash的配置很丰富，比如我们还可以用json格式的，示例：
nginx配置如下：

log_format  main  '{"data":"$time_local","ip":"$remote_addr","status":"$status","http_referer":"$http_referer","request_length":"$request_length","request":"$request","request_time":"$request_time","Authorization_User":"$http_authorization_user","http_user_agent":"$http_user_agent"}';

logstash的配置如下：

input {
        file {
                path => "/data/log/nginx/*"
                start_position => "beginning"
                sincedb_path => "/data/log/el/nginx_progress"
                codec => "json"
        }
}
output {
  elasticsearch {
    action => "index"          
    hosts  => "172.16.93.237:9200"
    index  => "erp"
  }
}

启动logstash

service logstash start

配置kibana

主要是配置下端口和地址

vim /etc/kibana/kibana.yml
配置或添加如下参数
server.port: 5601
server.host: "172.16.93.237"
server.name: "172.16.93.237"
elasticsearch.url: "http://172.16.93.237:9200"

启动kibana

service kibana start

如果一切正常，浏览器访问http://172.16.93.237:5601即可进入kibana
(PS：kibana默认是没用户名密码的，如果kibana暴露在公网，担心安全问题，可以通过nginx等加一层验证。)
第一次进入需选择一个es的index，这里填入刚才配置集群erp*,点击create即可。之后就愉快地使用吧。

结语

ELK解决日志分析及可视化的问题，它的精华在于logstash，这也是要多花点时间的地方。
当前的场景是基于一台日志服务器搭建的，如果要采集多台服务器的日志时，可使用filebeat + logstash + elasticsearch + kibana 的架构。

参考

https://kibana.logstash.es/content/logstash/examples/nginx-access.html
https://my.oschina.net/shawnplaying/blog/670217
https://my.oschina.net/itblog/blog/547250
https://es.xiaoleilu.com/010_Intro/05_What_is_it.html
https://www.gitbook.com/book/chenryn/elk-stack-guide-cn/details