ELK基础环境搭建-logstash部署

1        logstash部署

1.1 修改系统参数

1.        hosts文件

vi /etc/hosts

127.0.0.1   localhost

192.168.2.21 logstash

2.        limits及pam添加以下内容

vi /etc/security/limits.conf

* soft nofile 1024

* hard nofile 65536

* soft nproc 2047

* hard nproc 16384

* soft stack 10240

* hard stack 32768

vi /etc/pam.d/login

session required/lib64/security/pam_limits.so

1.2 安装java

rpm -ivh jdk-8u101-linux-x64.rpm

vi /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_101

export PATH=$JAVA_HOME/bin:$PATH

1.3 安装logstash

1.        解压

tar -xvzf logstash-2.4.0.tar.gz

2.        创建相关目录

cd /home/cendish/software/logstash-2.4.0/bin

mkdir conf logs

3.        启动进程

./logstash -e 'input { stdin {} } output { stdout {} }'

[cendish@logstash bin]$./logstash -e 'input { stdin { } } output { stdout {} }'

Settings: Default pipelineworkers: 4

Pipeline main started

hello world

2016-10-09T05:51:11.735Zlogstash hello world

2016-10-09T05:51:11.994Zlogstash

4.        创建pipeline

CTRL-D退出命令行。logstash架构如下：

logstash的pipeline包含input和output两个必须部分，另外filter是可选的。配置文件的结构如下：

#The # character at the beginning of a line indicates a comment. Use

#comments to describe your configuration.

input{

}

#The filter part of this file is commented out to indicate that it is

#optional.

#filter {

#

#}

output{

}

以下是官方的一个例子，读取文件的日志，输出到elasticsearch。参数文件如下：

input{

    file {

        path =>"/home/cendish/logs/*.log"

        start_position => beginning

        ignore_older => 0

    }

}

filter{

    grok {

        match => { "message" =>"%{COMBINEDAPACHELOG}"}

    }

    geoip {

        source => "clientip"

    }

}

output{

    elasticsearch {

        hosts => ["192.168.2.11:9200" ]

    }

}

5.        测试

./logstash -f conf/first-pipeline.conf --configtest

[cendish@logstash bin]$./logstash -f conf/first-pipeline.conf --configtest

Configuration OK

[cendish@logstash bin]$

6.        正式运行

./logstash -f conf/first-pipeline.conf

然后运行查询命令：

[cendish@logstash conf]$ curl-XGET '192.168.2.11:9200/logstash-2016.10.09/_search?pretty&q=response=200'

{

  "took" : 91,

  "timed_out" : false,

  "_shards" : {

    "total" : 5,

    "successful" : 5,

    "failed" : 0

  },

  "hits" : {

    "total" : 98,

    "max_score" : 5.053008,

    "hits" : [ {

      "_index" :"logstash-2016.10.09",

      "_type" : "logs",

      "_id" :"AVeoF0hU7h3h6d65347z",

      "_score" : 5.053008,

      "_source" : {

        "message" : "83.149.9.216- - [04/Jan/2015:05:13:45 +0000] \"GET/presentations/logstash-monitorama-2013/images/frontend-response-codes.pngHTTP/1.1\" 200 52878\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",

        "@version" : "1",

        "@timestamp" :"2016-10-09T06:17:58.042Z",

        "path" :"/home/cendish/logs/logstash-tutorial.log",

        "host" :"logstash",

        "clientip" :"83.149.9.216",

        "ident" : "-",

        "auth" : "-",

        "timestamp" :"04/Jan/2015:05:13:45 +0000",

        "verb" : "GET",

        "request" :"/presentations/logstash-monitorama-2013/images/frontend-response-codes.png",

        "httpversion" : "1.1",

        "response" : "200",

        "bytes" : "52878",

        "referrer" :"\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",

        "agent" :"\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36(KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",

        "geoip" : {

          "ip" :"83.149.9.216",

          "country_code2" :"RU",

          "country_code3" :"RUS",

          "country_name" :"Russian Federation",

          "continent_code" :"EU",

          "region_name" :"48",

          "city_name" :"Moscow",

          "latitude" :55.75219999999999,

          "longitude" : 37.6156,

          "timezone" :"Europe/Moscow",

          "real_region_name" :"Moscow City",

          "location" : [ 37.6156,55.75219999999999 ]

        }

      }

    },

...

过滤查询：

[cendish@logstash conf]$ curl-XGET'192.168.2.11:9200/logstash-2016.10.09/_search?pretty&q=geoip.city_name=Buffalo'

{

  "took" : 29,

  "timed_out" : false,

  "_shards" : {

    "total" : 5,

    "successful" : 5,

    "failed" : 0

  },

  "hits" : {

    "total" : 1,

    "max_score" : 1.1011324,

    "hits" : [ {

      "_index" :"logstash-2016.10.09",

      "_type" : "logs",

      "_id" :"AVeoF0hW7h3h6d65349B",

      "_score" : 1.1011324,

      "_source" : {

        "message" :"108.174.55.234 - - [04/Jan/2015:05:27:45 +0000] \"GET /?flav=rss20HTTP/1.1\" 200 29941 \"-\" \"-\"",

        "@version" : "1",

        "@timestamp" :"2016-10-09T06:17:58.619Z",

        "path" :"/home/cendish/logs/logstash-tutorial.log",

        "host" :"logstash",

        "clientip" :"108.174.55.234",

        "ident" : "-",

        "auth" : "-",

        "timestamp" :"04/Jan/2015:05:27:45 +0000",

        "verb" : "GET",

        "request" :"/?flav=rss20",

        "httpversion" :"1.1",

        "response" : "200",

        "bytes" : "29941",

        "referrer" :"\"-\"",

        "agent" :"\"-\"",

        "geoip" : {

          "ip" :"108.174.55.234",

          "country_code2" :"US",

          "country_code3" :"USA",

          "country_name" :"United States",

          "continent_code" :"NA",

          "region_name" :"NY",

          "city_name" :"Buffalo",

          "postal_code" :"14221",

          "latitude" : 42.9864,

          "longitude" : -78.7279,

          "dma_code" : 514,

          "area_code" : 716,

          "timezone" :"America/New_York",

          "real_region_name" :"New York",

          "location" : [ -78.7279,42.9864 ]

        }

      }

    } ]

  }

}