Logstash安装及使用

Logstash简介

• Logstash是一个开源的，服务端的数据处理管道，用来同时地从大量的数据源收集数据，转换数据，然后将数据发送到你最喜欢的存储地方，我们通常选择Elasticsearch

logstash安装及测试

• 下载包logstash-6.2.1.tar.gz,上传服务器，解压

cd /usr/local/elk

上传logstash-6.2.1.tar.gz

tar -zxvf logstash-6.2.1.tar.gz

• 配置环境变量（不是必须的，如果不配置，则每次启动必须去bin目录下执行）

echo "export PATH=\$PATH:/usr/local/logstash-6.2.1/bin" > /etc/profile.d/logstash.sh

让/etc/profile文件修改后立即生效

. /etc/profile  或者  source /etc/profile

• 执行如下命令，进行快速测试（标准输入–> 标准输出）

logstash -e 'input {stdin{}} output {stdout{}}'

或者  在解压后bin目录下执行：

./logstash -e 'input {stdin{}} output {stdout{}}'

• logstash常用参数

  • -e :指定logstash的配置信息，可以用于快速测试;
  • -f :指定logstash的配置文件；可以用于生产环境;

因为logstash是jvm跑的，资源消耗比较大，启动一个logstash就需要消耗500M左右的内存,所以启动过程可能有点慢，需要20s左右

• 测试：标准输入，返回json。

[root@iZ2ze2lelgjwuyib5l73eaZ bin]# ./logstash -e 'input { stdin { } } output { stdout { codec => rubydebug } }'
my name is weifan
Sending Logstash logs to /usr/local/elk/logstash-6.2.1/logs which is now configured via log4j2.properties.
The stdin plugin is now waiting for input:
[2018-03-12T12:19:21,744][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
[2018-03-12T12:19:21,762][INFO ][logstash.pipeline        ] Pipeline main started
[2018-03-12T12:19:21,830][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
{
    "@timestamp" => 2018-03-12T04:19:21.775Z,
      "@version" => "1",
          "host" => "localhost",
       "message" => "my name is weifan"
}
hello bny
{
    "@timestamp" => 2018-03-12T04:21:00.616Z,
      "@version" => "1",
          "host" => "localhost",
       "message" => "hello bny"
}

• 通过加载配置文件方式启动

  • config新建logstash-simple.conf

  • 编辑输入：

  input { stdin {} }
  output {
     stdout { codec=> rubydebug }
  }

  • 对配置文件进行测试

  测试logstash-simple.conf编写是否正确，-t 代表test：
  
  logstash -f  ../config/logstash-simple.conf  -t
  
  [root@iZ2ze2lelgjwuyib5l73eaZ config]# logstash -f  logstash-simple.conf  -t
  Sending Logstash logs to /usr/local/elk/logstash-5.0.0/logs which is now configured via log4j2.properties.
  Configuration OK  配置正确
  [2018-03-14T17:27:42,914][INFO ][logstash.runner          ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

  • 启动：logstash -f logstash-simple.conf

  • 测试：启动完成，在控制台输入 hello weifan，标准输出如下：

  {
      "@timestamp" => 2018-03-12T05:40:47.165Z,
        "@version" => "1",
            "host" => "localhost",
         "message" => "hello weifan"
  }
  

遇到问题

测试配置文件是否正确时报错，原因： 内存不足，减少启动程序所需内存，或加大内存，如关闭一些程序。

[root@iZ2ze2lelgjwuyib5l73eaZ bin]# ./logstash -f ../config/ecas_logstash.conf -t
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000e5330000, 449642496, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 449642496 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/local/elk/logstash-6.2.1/bin/hs_err_pid18662.log

解决方案：

vi ../config/jvm.options

-Xms2g  
-Xmx2g 

修改为  

-Xms512m  
-Xmx512m  

logstash输出信息存储到redis中

• config目录下新建 ：logstash_to_redis.conf
• 编辑logstash_to_redis.conf：

input { stdin { } }
output {
    stdout { codec => rubydebug }
    redis {
        host => '172.30.1.45'
        port => 5555
        data_type => 'list'
        key => 'logstash:ecas:redis'
    }
}

• 安装redis（略）,并开启redis动态监控

cd redis-2.8.19/src/

./redis-cli  -h 172.30.1.45 -p 5555  monitor

• 测试logstash_to_redis.conf是否生效：

[root@iZ2ze2lelgjwuyib5l73eaZ config]# logstash -f logstash_to_redis.conf 
weifan nihao a aa aaa
Sending Logstash logs to /usr/local/elk/logstash-6.2.1/logs which is now configured via log4j2.properties.
The stdin plugin is now waiting for input:
[2018-03-12T14:04:42,268][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
[2018-03-12T14:04:42,281][INFO ][logstash.pipeline        ] Pipeline main started
[2018-03-12T14:04:42,344][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
{
    "@timestamp" => 2018-03-12T06:04:42.244Z,
      "@version" => "1",
          "host" => "localhost",
       "message" => "weifan nihao a aa aaa"
}

• redis监控输出：

[root@iZ2ze2lelgjwuyib5l73eaZ src]# ./redis-cli -h 172.30.1.45 -p 5555  monitor
OK
1520834683.044455 [0 172.30.1.45:60688] "rpush" "logstash:ecas:redis" "{\"@timestamp\":\"2018-03-12T06:04:42.244Z\",\"@version\":\"1\",\"host\":\"localhost\",\"message\":\"weifan nihao a aa aaa\"}"

redis的监控上有以上信息输出，表明logstash和redis的结合是正常的

• 可以使用以下命令查看redis中存储数据：

连接redis
./redis-cli -h 172.30.1.45 -p 5555

查看所有key
keys *

查看key对应value数量
llen logstash:ecas:redis   

我的测试此处返回：4

列举所有value
lrange logstash:ecas:redis 0 4

返回：
172.30.1.45:5555> lrange logstash:ecas:redis 0 4
1) "{\"@timestamp\":\"2018-03-12T06:04:42.244Z\",\"@version\":\"1\",\"host\":\"localhost\",\"message\":\"weifan nihao a aa aaa\"}"
2) "{\"@timestamp\":\"2018-03-12T06:10:41.365Z\",\"@version\":\"1\",\"host\":\"localhost\",\"message\":\"\"}"
3) "{\"@timestamp\":\"2018-03-12T06:10:41.500Z\",\"@version\":\"1\",\"host\":\"localhost\",\"message\":\"\"}"
4) "{\"@timestamp\":\"2018-03-12T06:10:58.123Z\",\"@version\":\"1\",\"host\":\"localhost\",\"message\":\"yikang\"}"