搭建elasticsearch、Kibana、logstash、Filebeat日志查询系统

一、环境配置

操作系统：CentOS Linux release 7.4.1708

JDK：openjdk version "1.8.0_191"

ES：elasticsearch-6.6.1

kibana：kibana-6.6.1-linux-x86_64

logstash：logstash-6.6.1

filebeat：filebeat-6.2.4-linux-x86_64

测试服务器内网IP：172.1.201.55

测试服务器外网IP：172.1.35.241

测试服务器安装：elasticsearch、kibana、logstash

二、安装ES

下载

创建目录
mkdir /opt/soft
进入目录
cd /opt/soft
下载
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.1.tar.gz

安装

tar -zxvf elasticsearch-6.6.1.tar.gz
mv elasticsearch-6.6.1.tar.gz /tmp/
cd elasticsearch-6.6.1/
mkdir /tmp/elasticsearch
mkdir /tmp/elasticsearch/data
mkdir /tmp/elasticsearch/logs

编辑配置文件

vim /opt/soft/elasticsearch-6.6.1/config/elasticsearch.yml

cluster.name: payment-application
node.name: node-1
path.data: /tmp/elasticsearch/data
path.logs: /tmp/elasticsearch/logs
#当前机器的私有IP地址
network.host: 172.1.201.55
http.port: 9200

sudo vim /etc/sysctl.conf

添加配置 vm.max_map_count=655360

编辑完成，执行命令 sysctl -p

vi /etc/security/limits.conf

在文件最后添加

soft nofile 65536
hard nofile 65536

创建账号

groupadd elsearch
useradd elsearch -g elsearch -p /opt/soft/elasticsearch-6.6.1
chown -R elsearch:elsearch /opt/soft/elasticsearch-6.6.1
chown -R elsearch:elsearch /tmp/elasticsearch/data
chown -R elsearch:elsearch /tmp/elasticsearch/logs

启动ES

切换账号
su elsearch
/opt/soft/elasticsearch-6.6.1/bin/elasticsearch
后台进程的方式启动ES：
/opt/soft/elasticsearch-6.6.1/bin/elasticsearch -d

ES视图

http://172.1.35.241:9200/_cat/health?v

查看所有索引

http://172.1.35.241:9200/_cat/indices

删除索引

curl -XDELETE 'http://172.1.35.241:9200/xproject_tenant2_log4j2_2019.03.01'

ES数据定期删除（还没测试过）

#/bin/bash
#es-index-clear
#只保留15天内的日志索引
LAST_DATA=`date -d "-15 days" "+%Y.%m.%d"`
#删除上个月份所有的索引,
curl -XDELETE 'http://ip:port/*-'${LAST_DATA}'*'

三、安装Kibana

下载

cd /opt/soft
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.6.1-linux-x86_64.tar.gz

安装

tar -zxvf kibana-6.6.1-linux-x86_64.tar.gz 
vim /opt/soft/kibana-6.6.1-linux-x86_64/config/kibana.yml
修改配置：
server.port: 5601
server.host: "172.1.201.55" #当前机器的私有IP地址
elasticsearch.hosts: ["http://172.1.201.55:9200"] #ES机器的IP
kibana.index: ".kibana"

后台启动

nohup /opt/soft/kibana-6.6.1-linux-x86_64/bin/kibana >/dev/null 2>&1 &

kibana访问地址：http://47.1.213.14:5601

四、安装logstash

下载

cd /opt/soft
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.6.1.tar.gz

安装

tar -zxvf logstash-6.6.1.tar.gz
vim /opt/soft/logstash-6.6.1/config/log4j_to_es.conf
配置内容：

input {
 beats {
   port => 5043
  }
}
filter {
}
output {
   stdout {
        codec => rubydebug
    }
    #测试环境
    if[fields][log_source] == "xproject_test" {
        elasticsearch {
          hosts => ["172.1.35.241:9200"] #ES的外网IP
          index => "xproject_test_%{+YYYY.MM.dd}"
          manage_template => false
        }
    }
    #正式环境
    if[fields][log_source] == "xproject_tenant1" {
        elasticsearch {
          hosts => ["172.1.35.241:9200"] #ES的外网IP
          manage_template => false
          index => "xproject_tenant1_%{+YYYY.MM.dd}"
        }
    }
}

启动

nohup /opt/soft/logstash-6.6.1/bin/logstash -f /opt/soft/logstash-6.6.1/config/log4j_to_es.conf >/dev/null 2>&1 &

五、安装Filebeat

下载【测试环境】

在日志收集的服务器上安装
cd /opt/soft
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-linux-x86_64.tar.gz

安装【测试环境】

tar -zvxf filebeat-6.2.4-linux-x86_64.tar.gz
编辑配置文件：
vim /opt/soft/filebeat-6.2.4-linux-x86_64/filebeat.yml

max_procs: 2
queue.mem:
    events: 128
    flush.min_events: 10
    flush.timeout: 10s
filebeat.prospectors:
- input_type: log
  enabled: true
  tail_files: true
  paths:
    - /tmp/logs/XProject_all.log
  fields:
  	log_source: xproject_test
    
output.logstash:
  hosts: ["172.1.35.241:5043"] #logstash的外网IP

启动

nohup /opt/soft/filebeat-6.2.4-linux-x86_64/filebeat -e -c /opt/soft/filebeat-6.2.4-linux-x86_64/filebeat.yml >/dev/null 2>&1 &