logstash 采集 Java log4j的日志（全面介绍）

鉴于这方便的资料不是很多，下面详细介绍一下logstash的使用。

下载logstash:

wget https://download.elastic.co/logstash/logstash/logstash-2.4.0.tar.gz
tart zxvf logstash-2.4.0.tar.gz

下载完毕之后，我们启动一下：

bin/logstash -e ‘input { stdin { } } output { stdout {} }’

我们现在可以在命令行下输入一些字符，然后我们将看到logstash的输出内容：

hello world
2016-09-21T01:22:14.405+0000 0.0.0.0 hello world

这里注意我们在命令行中使用了 -e 参数，该参数允许Logstash直接通过命令行接受设置。这点尤其快速的帮助我们反复的测试配置是否正确而不用写配置文件。让我们再试个更有意思的例子。首先我们在命令行下使用CTRL-C命令退出之前运行的Logstash。现在我们重新运行Logstash使用下面的命令：

bin/logstash -e ‘input { stdin { } } output { stdout { codec => rubydebug } }’

我们再输入一些字符，这次我们输入”goodnight moon”：

goodnight moon
{
“message” => “goodnight moon”,
“@timestamp” => “2013-11-20T23:48:05.335Z”,
“@version” => “1”,
“host” => “my-laptop”
}

下一步我们下载elastic serach，将日志通过logstash插入es.

wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/zip/elasticsearch/2.4.0/elasticsearch-2.4.0.zip

uzip elasticsearch-2.4.0.zip

启动es, 不要使用root，否则会报错。鄙视一下 ：

sudo -u es ./elasticsearch-2.4.0/bin/elasticsearch

启动之后，es会出现一个控制台，不要关闭它，重新打开一个shell窗口

netstat -ln
看看 9200 和9300端口在不在，如果在侦听的话，说明es启动成功了。

现在我们Java的log4j日志打印内容，通过logstash输入到es中，假设Java的日志内容为：

2016-08-26 11:28:30,996 [http-nio-18600-exec-72] [ERROR] CLUSTERDOWN The cluster is down - [com.haiziwang.platform.kmem.impl.KMEMCacheImpl.39]
redis.clients.jedis.exceptions.JedisClusterException: CLUSTERDOWN The cluster is down
at redis.clients.jedis.Protocol.processError(Protocol.java:115)
at redis.clients.jedis.Protocol.process(Protocol.java:151)
at redis.clients.jedis.Protocol.read(Protocol.java:205)
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:297)
at redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:216)
at redis.clients.jedis.Connection.getBulkReply(Connection.java:205)
at redis.clients.jedis.Jedis.get(Jedis.java:101)

OK，我们在logstash的bin 目录里面新建一个配置文件 vi logstash-simple.confinput

{ stdin { } }

filter {
   multiline {
            pattern => "^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}"
            negate => true
            what => "previous"
        }

   >grok {
       match => [ "message", "%{NOTSPACE:day} %{NOTSPACE:datetime} %{NOTSPACE:thread-id} %{NOTSPA
CE:level} %{GREEDYDATA:msginfo} " ]
   }
}

output {
  elasticsearch { hosts => ["127.0.0.1:9200"] }
  stdout { codec => rubydebug }
}

使用配置文件的方式启动logstatsh

./logstash -f logstash-simple.conf

在shell窗口中输入：

2016-08-26 11:28:30,996 [http-nio-18600-exec-72] [ERROR] CLUSTERDOWN The cluster is down - [com.haiziwang.platform.kmem.impl.KMEMCacheImpl.39]
redis.clients.jedis.exceptions.JedisClusterException: CLUSTERDOWN The cluster is down
at redis.clients.jedis.Protocol.processError(Protocol.java:115)
at redis.clients.jedis.Protocol.process(Protocol.java:151)
at redis.clients.jedis.Protocol.read(Protocol.java:205)
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:297)
at redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:216)
at redis.clients.jedis.Connection.getBulkReply(Conn “@version” => “1”,
“@timestamp” => “2016-09-02T09:23:20.116Z”,
“host” => “esf-platform-db-01”,
“tags” => [
[0] “multiline”
],
“day” => “2016-08-26”,
“datetime” => “11:28:30,996”,
“thread-id” => “[http-nio-18600-exec-72]”,
“level” => “[ERROR]”,
“msginfo” => “CLUSTERDOWN The cluster is down - [com.haiziwang.platform.kmem.impl.KMEMCacheImpl.39] \nredis.clients.jedis.exceptions.JedisClusterException: CLUSTERDOWN The cluster is down\n\tat redis.clients.jedis.Protocol.processError(Protocol.java:115)\n\tat redis.clients.jedis.Protocol.process(Protocol.java:151)\n\tat redis.clients.jedis.Protocol.read(Protocol.java:205)\n\tat redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:297)\n\tat redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:216)\n\tat redis.clients.jedis.Connection.getBulkReply(Connection.java:205)\n\tat”ection.java:205)
at redis.clients.jedis.Jedis.get(Jedis.java:101)

显示数据：

      "@version" => "1",
    "@timestamp" => "2016-09-02T09:23:20.116Z",
          "host" => "esf-platform-db-01",
          "tags" => [
        [0] "multiline"
    ],
           "day" => "2016-08-26",
      "datetime" => "11:28:30,996",
     "thread-id" => "[http-nio-18600-exec-72]",
         "level" => "[ERROR]",
       "msginfo" => "CLUSTERDOWN The cluster is down - [com.haiziwang.platform.kmem.impl.KMEMCacheImpl.39] \nredis.clients.jedis.exceptions.JedisClusterException: CLUSTERDOWN The cluster is down\n\tat redis.clients.jedis.Protocol.processError(Protocol.java:115)\n\tat redis.clients.jedis.Protocol.process(Protocol.java:151)\n\tat redis.clients.jedis.Protocol.read(Protocol.java:205)\n\tat redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:297)\n\tat redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:216)\n\tat redis.clients.jedis.Connection.getBulkReply(Connection.java:205)\n\tat"

ok,至此数据已经插入了es,我们es的接口也可以查看一下

curl http://localhost:9200/_search?pretty

返回数据：

   "@version" : "1",
    "@timestamp" : "2016-09-02T09:23:20.116Z",
    "host" : "esf-platform-db-01",
    "tags" : [ "multiline" ],
    "day" : "2016-08-26",
    "datetime" : "11:28:30,996",
    "thread-id" : "[http-nio-18600-exec-72]",
    "level" : "[ERROR]",
    "msginfo" : "CLUSTERDOWN The cluster is down - [com.haiziwang.platform.kmem.impl.KMEMCacheImpl.39] \nredis.clients.jedis.exceptions.JedisClusterException: CLUSTERDOWN The cluster is down\n\tat redis.clients.jedis.Protocol.processError(Protocol.java:115)\n\tat redis.clients.jedis.Protocol.process(Protocol.java:151)\n\tat redis.clients.jedis.Protocol.read(Protocol.java:205)\n\tat redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:297)\n\tat redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:216)\n\tat redis.clients.jedis.Connection.getBulkReply(Connection.java:205)\n\tat"

ok,下个章节 我们kibana显示es里面的数据