logstash 插件使用介绍

最新推荐文章于 2024-07-09 08:49:10 发布

zhiliang-chen

最新推荐文章于 2024-07-09 08:49:10 发布

阅读量261

点赞数

分类专栏： ELK 文章标签： logstash plugin

本文链接：https://blog.csdn.net/liangkiller/article/details/102695382

版权

ELK 专栏收录该内容

8 篇文章 0 订阅

订阅专栏

logstash 插件使用介绍

logstash 的配置有input,filter,output三个区;

其中input负责从外部读取数据,转换为logstash事件;

logstash事件,json格式的数据结构,json的key即为"字段";

filter区,对input后的事件进行数据处理,匹配"字段"处理数据;

output区,将事件输入到目标设备;

实例

input { 
    stdin{} 
}

output {
    stdout{}
}

在终端输入

hello

输出为

{
      "@version" => "1",
    "@timestamp" => 2019-10-21T09:59:30.268Z,
          "host" => "vps138",
       "message" => "hello"
}

一般input插件,读取数据到message字段; 一般output插件,也是将message字段写入目标设备;

filter区主要目的是对input的字段内容做处理,比如grok,可以正则匹配message字段的内容,生成新的字段,用于output输出;

input { 
    stdin{} 
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
}

output {
    stdout{}
}

在终端输入

127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"

在终端可以看到输出

{
        "message" => "127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] \"GET /xampp/status.php HTTP/1.1\" 200 3891 \"http://cadenza/xampp/navi.php\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\"",
     "@timestamp" => "2013-12-11T08:01:45.000Z",
       "@version" => "1",
           "host" => "cadenza",
       "clientip" => "127.0.0.1",
          "ident" => "-",
           "auth" => "-",
      "timestamp" => "11/Dec/2013:00:01:45 -0800",
           "verb" => "GET",
        "request" => "/xampp/status.php",
    "httpversion" => "1.1",
       "response" => "200",
          "bytes" => "3891",
       "referrer" => "\"http://cadenza/xampp/navi.php\"",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\""
}

还一类插件,codec插件,可用于input区和output区;
用于input区,就是对输入的内容进行解码,解码为logstash事件;
用于output区,就是在输出前,对事件进行编码;编码为指定的输出格式;

以codec插件json为例,用于input区,就是对输入的json格式进行解码,提取出相应的字段;用于output区,就是将logstash事件编码为json格式;

input {
    file { 
        path => "/var/tmp/test.txt"
        codec=> "json"
    }
}
output {
    stdout{}
}

在新的终端输入

echo  '{"version":1,"timestamp":"2019-10-21T07:14:41.875Z","incre_id":444}' > /var/tmp/test.txt

在原终端可以看到输出

{
       "version" => 1,
          "host" => "vps155",
    "@timestamp" => 2019-10-22T09:05:30.741Z,
     "timestamp" => "2019-10-21T07:14:41.875Z",
      "@version" => "1",
          "path" => "/var/tmp/test.txt",
      "incre_id" => 444
}