elk+filebeat 日志收集搭建

1．环境准备

a)	启动四台linux虚拟机
b)	安装JDK1.8

2. 下载并安装ES(x.x.x.130)

下载地址：

curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.2.tar.gz

安装ES：

tar -xvf elasticsearch-6.5.2.tar.gz

配置ES：

cd  elasticsearch-6.5.1/config/
vim elasticsearch.yml

cluster.name: my-test-es		#es集群的名称
path.data: /application/es/data	#自定义数据存放的文件夹
path.logs: /application/es/logs	#自定义日志存放的文件夹
network.host: x.x.x.130  	#es所在主机的ip地址
http.port: 9200 	#es的端口号

启动ES：

cd elasticsearch-6.5.2/bin
./elasticsearch

3. 下载并安装kibana(x.x.x.129)

下载地址：

https://www.elastic.co/downloads
选择kibana 下载

安装kibana：

tar -xvf kibana-6.5.1-linux-x86_64.tar.gz

配置kibana

server.port: 5601		#kibana的端口号
server.host: "x.x.x.129"		#kibana 所在服务器的ip
server.name: "centos001-kibana"		#kibana 的名称
elasticsearch.url: http://x.x.x.130:9200 		#es的地址

启动Kibana

cd  kibana-6.5.1-linux-x86_64/bin/
./kibana

4. 下载安装logStash(x.x.x.131)

下载地址：

https://www.elastic.co/downloads
选择logStash下载

安装logStash

tar -xvf  logstash-6.5.1.tar.gz

安装logStash插件

cd  logstash-6.5.1
vim Gemfile
修改：source "https://ruby.taobao.org/"，替代source "https://rubygems.org"

. /bin/logstash-plugin install --no-verify

配置logStash

------------------------使用logStash采集日志信息配置开始-----------------------------------------

新建配置文件test.conf
cd config
vim test.conf
输入以下内容：
input{
    file{
            path => "/application/logstash/testLog/test.log"
            start_position => beginning

    }
}
filter{
        grok{
                match=>{
                        "message"=>"%{IP:client} %{WORD:method} %{URIPATHPARAM:reuqest} %{NUMBER:bytes} %{NUMBER:duration}"
                }
        }
}
output{

         elasticsearch {
                 hosts => ["http://x.x.x.130:9200"]
                 index => "logstash-testlog-%{+YYYY.MM.dd}"
                 #user => "elastic"
                 #password => "changeme"
        }

        #stdout{codec=>rubydebug}
}

新建test.log
	55.3.244.1 GET /index.html 15824 0.043

------------------------使用logStash采集日志信息配置结束-----------------------------------------

------------------------使用filebeat采集日志信息，logStash过滤日志信息配置开始--------------

新建配置文件test.conf
cd config
vim test.conf
输入以下内容：
input{
   	 beats {
		port => 5044
	 }
}
filter{
        grok{
                match=>{
                        "message"=>"%{IP:client} %{WORD:method} %{URIPATHPARAM:reuqest} %{NUMBER:bytes} %{NUMBER:duration}"
                }
        }
}
output{

         elasticsearch {
                 hosts => ["http://x.x.x.130:9200"]
                 index => "logstash-testlog-%{+YYYY.MM.dd}"
                 #user => "elastic"
                 #password => "changeme"
        }

        #stdout{codec=>rubydebug}
}

-------------------------使用filebeat采集日志信息，logStash过滤日志信息配----------------------

启动logStash
cd  logstash-6.5.1/
./bin/logstash -f config/test.conf

5. 下载安装filebeat

下载：

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.5.2-linux-x86_64.tar.gz

安装：

tar xzvf filebeat-6.5.2-linux-x86_64.tar.gz

配置：修改文件 filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/test.log
#- c:\programdata\elasticsearch\logs\*
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

 #----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["x.x.x.131:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

新建test.log

55.3.244.1 GET /index.html 15824 0.043

启动filebeat

sudo chown root filebeat.yml 
sudo ./filebeat -e

6.filebeat读取多个log文件，采用不同的过滤方式，输出到es的不同index中

Filebeat中的配置，filebeat.yml修改为：

filebeat.inputs:

- type: log
  enabled: true
  paths:
    #- /var/log/log1.log
     fields:
  #  level: debug
  #  review: 1
    log_source: log1
- type: log
  enabled: true
  paths:
    #- /var/log/log2.log
     fields:
  #  level: debug
  #  review: 1
    log_source: log2
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

 #----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["192.168.6.131:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

logStash中的配置文件修改为：

input{
        beats {
                port => 5044
        }
}
filter{
        if [fields][log_source] == log1" {
                        grok {
                                 match => {
                        "message"=>""#这里写自己的处理
                			}
                        }
                }
        if [fields][log_source] == "log2" {
                 grok {
                match=>{
                        "message"=>""# 这里写自己的处理方式
                }
        	}
          }
        mutate {  #这里删除一些信息 可选
                remove_field =>["message"]
                remove_field =>["beat"]
                remove_field =>["host"]
                #remove_field =>["fields"]
                remove_field =>["input"]
                remove_field =>["prospector"]
        }
}
output{

         if [fields][log_source] == log1" {

                elasticsearch {
                        hosts => ["http://x.x.x.130:9200"]
                        index => "log1"
                        #user => "elastic"
                        #password => "changeme"
                }      
        }

        if [fields][log_source] == "log2" {

                 elasticsearch {
                        hosts => ["http://x.x.x.130:9200"]
                        index => "log2"
                        #user => "elastic"
                        #password => "changeme"
                }
        }
}