Logstash 与 filebeat 配置

 

 

mutate插件可以对事件中的数据进行修改,包括rename、update、replace、convert、split、gsub、uppercase、lowercase、strip、remove_field、join、merge等功能。

1、rename

对于已经存在的字段,重命名其字段名称。

 

filter {
    mutate {
        rename => ["syslog_host", "host"]
    }
}

2、update

更新字段内容,如果字段不存在,不会新建

 

filter {
    mutate {
        update => { "sample" => "My new message" }
    }
}

3、replace

与 update 功能相同,区别在于如果字段不存在则会新建字段

 

filter {
    mutate {
        replace => { "message" => "%{source_host}: My new message" }
    }
}

4、convert

数据类型转换。

 

filter {
    mutate {
        convert => ["request_time", "float"]
    }
}

5、gsub

gsub 提供了通过正则表达式实现文本替换的功能。

 

filter {
    mutate {
        gsub => [
            # replace all forward slashes with underscore
            "fieldname", "/", "_",
            # replace backslashes, question marks, hashes, and minuses
            # with a dot "."
            "fieldname2", "[\\?#-]", "."
        ]
    }
}

6、uppercase/lowercase

大小写转换

 

filter {
    mutate {
        uppercase => [ "fieldname" ]
    }
}

7、split

将提取到的某个字段按照某个字符分割

 

filter {
    mutate {
        split => ["message", "|"]
    }
}

针对字符串 "123|321|adfd|dfjld*=123",可以看到输出结果:

 

{
    "message" => [
        [0] "123",
        [1] "321",
        [2] "adfd",
        [3] "dfjld*=123"
    ],
    "@version" => "1",
    "@timestamp" => "2014-08-20T15:58:23.120Z",
    "host" => "raochenlindeMacBook-Air.local"
}

8、strip

类似 trim,只去除首尾的空白字符

 

filter {
    mutate {
        strip => ["field1", "field2"]
    }
}

9、remove_field

删除字段:

 

filter {
    mutate {
        remove_field => [ "foo_%{somefield}" ]
    }
}

10、join

将类型为 array 的字段中的 array 元素使用指定字符为分隔符聚合成一个字符串。
如我们可以将 split 分割的结果再重新聚合起来:

 

filter {
    mutate {
        split => ["message", "|"]
    }
    mutate {
        join => ["message", ","]
    }
}

输出结果:

 

{
    "message" => "123,321,adfd,dfjld*=123",
    "@version" => "1",
    "@timestamp" => "2014-08-20T16:01:33.972Z",
    "host" => "raochenlindeMacBook-Air.local"
}

11、merge

对于几个类型为 array 或 hash 或 string 的字段,我们可以使用 merge 合并

 

filter {
    mutate {
        merge => [ "dest_field", "added_field" ]
    }
}

需要注意的是,array 和 hash 两个字段是不能 merge 的

注意:建议正则放在单引号内,例如'^\[?[0-9][0-9]:?[0-9][0-9]|^[[:graph:]]+'

样例 描述

单个字符

 

x

单个字符

.

任何字符

[xyz]

字符类

[^xyz]

非字符类

[[:alpha:]]

ASCII字符类

[[:^alpha:]]

非ASCII字符类

\d

Perl字符类

\D

非Perl字符类

\pN

Unicode字符类(一个字母的名称)

\p{Greek}

Unicode字符类

\PN

非Unicode字符类(一个字母的名称)

\P{Greek}

非Unicode字符类

复合类型

 

xy

x|y

重复类型

 

x*

以x开头

x+

一个或者多个x

x?

零或一个x

x{n,m}

n or n+1 or … or m x, prefer more

x{n,}

n or more x, prefer more

x{n}

exactly n x

x*?

zero or more x, prefer fewer

x+?

one or more x, prefer fewer

x??

zero or one x, prefer zero

x{n,m}?

n or n+1 or … or m x, prefer fewer

x{n,}?

n or more x, prefer fewer

x{n}?

exactly n x

分组

 

(re)

numbered capturing group (submatch)

(?P<name>re)

named & numbered capturing group (submatch)

(?:re)

non-capturing group

(?i)abc

set flags within current group, non-capturing

(?i:re)

set flags during re, non-capturing

(?i)PaTTeRN

case-insensitive (default false)

(?m)multiline

multi-line mode: ^ and $ match begin/end line in addition to begin/end text (default false)

(?s)pattern.

let . match \n (default false)

(?U)x*abc

ungreedy: swap meaning of x* and x*?x+ and x+?, etc (default false)

空字符串

 

^

at beginning of text or line (m=true)

$

at end of text (like \z not \Z) or line (m=true)

\A

at beginning of text

\b

at ASCII word boundary (\w on one side and \W\A, or \z on the other)

\B

not at ASCII word boundary

\z

at end of text

转义序列

 

\a

bell (same as \007)

\f

form feed (same as \014)

\t

horizontal tab (same as \011)

\n

newline (same as \012)

\r

carriage return (same as \015)

\v

vertical tab character (same as \013)

\*

literal *, for any punctuation character *

\123

octal character code (up to three digits)

\x7F

two-digit hex character code

\x{10FFFF}

hex character code

\Q...\E

literal text ... even if ... has punctuation

ASCII字符类

 

[[:alnum:]]

alphanumeric (same as [0-9A-Za-z])

[[:alpha:]]

alphabetic (same as [A-Za-z])

[[:ascii:]]

ASCII (same as \x00-\x7F])

[[:blank:]]

blank (same as [\t ])

[[:cntrl:]]

control (same as [\x00-\x1F\x7F])

[[:digit:]]

digits (same as [0-9])

[[:graph:]]

graphical (same as [!-~] == [A-Za-z0-9!"#$%&'()*+,\-./:;<=>?@[\\\]^_` {|}~])

[[:lower:]]

lower case (same as [a-z])

[[:print:]]

printable (same as [ -~] == [ [:graph:]])

[[:punct:]]

punctuation (same as [!-/:-@[-`{-~])

[[:space:]]

whitespace (same as [\t\n\v\f\r ])

[[:upper:]]

upper case (same as [A-Z])

[[:word:]]

word characters (same as [0-9A-Za-z_])

[[:xdigit:]]

hex digit (same as [0-9A-Fa-f])

支持Perl字符类

 

\d

digits (same as [0-9])

\D

not digits (same as [^0-9])

\s

whitespace (same as [\t\n\f\r ])

\S

not whitespace (same as [^\t\n\f\r ])

\w

word characters (same as [0-9A-Za-z_])

\W

not word characters (same as [^0-9A-Za-z_])

 

 

架构一:
filebeat -> logstash1 -> redis -> logstash2 -> elasticsearch(集群) -> kibana
这里就不写安装程序的步骤了相信大家都没有难度:
(软件安装可自行设计)
230,安装filebeat, logstash1 ,elasticsearch
232,安装logstash2, redis, elasticsearch  ,kibana

注意:filebeat文件很注重文件格式
1,配置filebeat文件:
[root@localhost filebeat]# cat /etc/filebeat/filebeat.yml
filebeat:
  prospectors:
   # - #每个日志文件的开始
   #   paths: #定义路径
   #     - /var/www/logs/access.log #绝对路径
   #   input_type: log #日志类型为log
   #   document_type: api4-nginx-accesslog # 此名称要与logstash定义的名称相对应,logstash要使用此名称做type判断使用
    -
      paths:
        - /opt/apps/huhu/logs/ase.log
      input_type: log
      document_type: "ase-ase-log"
      encoding: utf-8
      tail_files: true  #每次最后一行
      multiline.pattern: '^\[' #分割符
      multiline.negate: true
      multiline.match: after    #最后合并
      #tags: ["ase-ase"]

    -
      paths:   #收集json格式日志
        - /var/log/nginx/access.log
      input_type: log
      document_type: "nginx-access-log"
      tail_files: true
      json.keys_under_root: true      
      json.overwrite_keys: true  

  registry_file: /var/lib/filebeat/registry
output:      #输出到230
  logstash:
    hosts: ["192.168.0.230:5044"]

shipper:
  logging:
    to_files: true
    files:
      path: /tmp/mybeat

 2.配置230:logstash-->input-redis
[root@web1 conf.d]# pwd
/etc/logstash/conf.d
[root@web1 conf.d]# cat nginx-ase-input.conf 
input {
        beats {
        port => 5044
        codec => "json"
        }}

output {                         
        if [type] == "nginx-access-log" {
        redis {                            #nginx日志写到redis信息
                data_type => "list"
                key => "nginx-accesslog"
                host => "192.168.0.232"
                port => "6379"
                db => "4"
                password => "123456"
        }}
        if [type] == "ase-ase-log" {
        redis {                            #写到ase日志写到redis信息
                data_type => "list"
                key => "ase-log"
                host => "192.168.0.232"
                port => "6379"
                db => "4"
                password => "123456"
        }}      

}

  3.redis写到elstach里,232服务器配置:logstash-->output-->resid->elasticsearch
[root@localhost conf.d]# pwd
/etc/logstash/conf.d
[root@localhost conf.d]# cat nginx-ase-output.conf 
input {
        redis {
               type => "nginx-access-log"
                data_type => "list"
                key => "nginx-accesslog"
                host => "192.168.0.232"
                port => "6379"
                db => "4"
                password => "123456"
                codec  => "json"
        }

        redis {
                type => "ase-ase-log"
                data_type => "list"
                key => "ase-log"
                host => "192.168.0.232"
                port => "6379"
                db => "4"
                password => "123456"
        }
}

output {
    if [type] == "nginx-access-log" { 
        elasticsearch {  
            hosts => ["192.168.0.232:9200"] 
            index => "nginx-accesslog-%{+YYYY.MM.dd}" 
    }}
    if [type] == "ase-ase-log" {
            elasticsearch {
                hosts => ["192.168.0.232:9200"]
                index => "ase-log-%{+YYYY.MM.dd}"
        }}
}

4,在232上配置elsaticsearch--->kibana
在kibana上找到ELS的索引即可。

架构二:
filebeat -> redis -> logstash --> elsasctic --> kibana  #缺点filebeat写进redis有限制,占时还没找到多个写入。

1.feilebeat配置:
[root@localhost yes_yml]# cat filebeat.yml 
filebeat:
  prospectors:
   # - #每个日志文件的开始
   #   paths: #定义路径
   #     - /var/www/logs/access.log #绝对路径
   #   input_type: log #日志类型为log
   #   document_type: api4-nginx-accesslog # 此名称要与logstash定义的名称相对应,logstash要使用此名称做type判断使用
    -
      paths:
        - /opt/apps/qpq/logs/qpq.log
      input_type: log
      document_type: "qpq-qpq-log"
      encoding: utf-8
      tail_files: true
      multiline.pattern: '^\['
      multiline.negate: true
      multiline.match: after
   #tags: ["qpq-qpq-log"]
  registry_file: /var/lib/filebeat/registry

output:
  redis:
      host: "192.168.0.232"
      port: 6379
      db: 3
      password: "123456"
      timeout: 5
      reconnect_interval: 1
      index: "pqp-pqp-log"

shipper:
  logging:
    to_files: true
    files:
      path: /tmp/mybeat

2.由232redis-->els--kibana
[root@localhost yes_yml]# cat systemlog.conf 
input {
   redis {
        type => "qpq-qpq-log"
        data_type => "list"
        key => "qpq-pqp-log"
        host => "192.168.0.232"
        port => "6379"
        db => "3" 
        password => "123456"
        }}
output {
   if [type] == "qpq-qpq-log"{
      elasticsearch {  
            hosts => ["192.168.0.232:9200"] 
            index => "qpq-qpq-log-%{+YYYY.MM.dd}" 

 }

}
}

3.在232上配置elsaticsearch--->kibana
在kibana上找到ELS的索引即可

 

 

 

 

 

 

 

发布了451 篇原创文章 · 获赞 256 · 访问量 75万+
展开阅读全文

没有更多推荐了,返回首页

©️2019 CSDN 皮肤主题: Age of Ai 设计师: meimeiellie

分享到微信朋友圈

×

扫一扫,手机浏览