Centos7 安装ELK

1、概述

• ELK 简介
  ELK 是Elasticsearch+Logstash+Kibana的简称：

  Elasticsearch是一个基于Lucene的搜索服务器。它提供了一个分布式多用户能力的全文搜索引擎，基于java开发
  Logstash是一个接收，处理，转发日志的工具。
  Kibana是一个基于浏览器页面的Elasticsearch前端展示工具。Kibana全部使用HTML语言和Javascript编写的

• 操作系统版本

cat /etc/redhat-release 
CentOS Linux release 7.0.1406 (Core) 

2、 系统配置

• 关闭selinux

 sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config
 setenforce 0

• 设置firewall
  安装firewall（若没有，先安装）

yum install firewalld firewall-config
systemctl start firewalld.service
systemctl enable firewalld.service
systemctl status firewalld

需要开放的端口

服务	需要开放的端口
Elasticsearch	tcp/9200和9300
kibana	tcp/5601
logstash	tcp/5000

firewall-cmd --permanent --add-port={9200/tcp,9300/tcp}
firewall-cmd --permanent --add-port=5601/tcp
firewall-cmd --permanent --add-port=5000/tcp
firewall-cmd --reload
firewall-cmd --state
firewall-cmd --list-all

• 设置FQND

#cat /etc/hostname        
elk

#cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.0.102 elk.zll.com  elk

#hostname -F /etc/hostname

#hostname -f               
elk.zll.com

3、安装Elasticsearch

yum install java-1.7.0-openjdk    (安装java)
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.noarch.rpm
yum  -y localinstall elasticsearch-1.7.1.noarch.rpm

启动服务

systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl start elasticsearch
systemctl status elasticsearch

查看Elasticsearch配置文件

rpm -qc elasticsearch
/etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/logging.yml
/etc/init.d/elasticsearch
/etc/sysconfig/elasticsearch
/usr/lib/sysctl.d/elasticsearch.conf
/usr/lib/systemd/system/elasticsearch.service
/usr/lib/tmpfiles.d/elasticsearch.conf

查看Elasticsearch日志文件

查看Elasticsearch端口（在firewall中放行）

4、安装kibana

下载软件包

wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz
tar zxf kibana-4.1.1-linux-x64.tar.gz -C /usr/local/
cd /usr/local/
mv kibana-4.1.1-linux-x64 kibana

创建kibana.service启动文件。

cat > /etc/systemd/system/kibana.service <<EOF
[Service]
ExecStart=/usr/local/kibana/bin/kibana

[Install]
WantedBy=multi-user.target
EOF

启动kibana服务

systemctl enable kibana
systemctl start kibana
systemctl status kibana

查看kibana端口

web输入 http://ip_address:5601

5、安装Logstash

安装软件包

wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.3-1.noarch.rpm
yum localinstall logstash-1.5.3-1.noarch.rpm 

设置ssl
使用FQDN创建SSL 证书（例：elk.zll.com）

cd /etc/pki/tls
openssl req -subj '/CN=elk.zll.com/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

创建一个01-logstash-initial.conf 文件

cat > /etc/logstash/conf.d/01-logstash-initial.conf << EOF
input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}


filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}
EOF

启动logstash服务

systemctl restart logstash
systemctl status logstash
chkconfig logstash on       (开机启动设置特殊)

查看logstash日志，是否有报错
tail /var/log/logstash/logstash.log

查看logstash端口(firewall中开启)

6、客户端安装Logstash Forwarder

• 安装软件包

wget https://download.elastic.co/logstash-forwarder/binaries/logstash-forwarder-0.4.0-1.x86_64.rpm
yum localinstall logstash-forwarder-0.4.0-1.x86_64.rpm

• 修改配置文件/etc/logstash-forwarder.conf
  修改配置文件中的elk-server

cp /etc/logstash-forwarder.conf /etc/logstash-forwarder.conf.old
cat > /etc/logstash-forwarder.conf << EOF
{
  "network": {
    "servers": [ "elk.zll.com:5000" ],

    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",

    "timeout": 15
  },

  "files": [
    {
      "paths": [
        "/var/log/messages",
        "/var/log/secure"
      ],

      "fields": { "type": "syslog" }
    }
  ]
}
EOF

• 启动服务并设置开启启动

systemctl restart logstash-forwarder
chkconfig logstash-forwarder on
systemctl status logstash-forwarder

登录elk-server的web界面进行配置：http://ip_address:5601

本文参考：陈沙克日志