logstash grok正则案例

案例一

1.1 日志

Aug 6 2011 20:34:46 HUAWEI %% 01 HWCM/5/EXIT(l)[1]: exit from configure mode
Jan 8 2013 03:58:15+07:00 HUAWEI %% 01 HWCM/5/EXIT(l)[1]: exit from configure mode

1.2 日志grok解析

(?<ss1_date>(%{MONTH} %{MONTHDAY} %{YEAR} %{TIME}) | (%{MONTH} %{MONTHDAY} %{YEAR} %{TIME}%{ISO8601_TIMEZONE})) %{WORD:ss2_word} (?<ss3_cut_str>([\s\S]{2})) %{NUMBER:ss4_number} %{USERNAME:ss5_username}/%{NUMBER:ss6_number}/

1.3 解析结果

{
  "ss6_number": "5",
  "ss3_cut_str": "%%",
  "ss4_number": "01",
  "ss1_date": "Jan 8 2013 03:58:15+07:00",
  "ss5_username": "HWCM",
  "ss2_word": "HUAWEI"
}

1.4 必要说明

\s  空白符
\S  非空白符
[\s\S]  任意字符
[\s\S]*  0个到任意多个字符
[\s\S]*?  0个字符，匹配任何字符前的位置
\w  匹配包括下划线的任何单词字符。等价于[A-Za-z0-9_]
\W  匹配任何非单词字符。等价于[^A-Za-z0-9_]

截取字符串的前两位 (?<ss3_cut_str>([\s\S]{2}))

案例二

2.1 日志

2011-08-18 18:08:56 HUAWEI %%01CLI/5/LOGIN(l):CID=2160731923;VTY0 login from 192.168.0.1 succeed.

2.2 日志grok解析

(?<ss1_date>(%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY})\s+%{TIME}) %{WORD:ss2_word} (?<ss3>([\s\S]{2}))(?<ss4>([\s\S]{2}))%{USERNAME:ss5_username}/%{NUMBER:ss6_number}/

2.3 解析结果

{
  "ss6_number": "5",
  "ss3": "%%",
  "ss4": "01",
  "ss1_date": "2011-08-18 18:08:56",
  "ss5_username": "CLI",
  "ss2_word": "HUAWEI"
}

案例三

3.1 日志

vclound 2015-11-03 03:35:50.283 INFO /usr/lib/python2.6/site-packages/urllib3/connectionpool.py:203 _new_conn - 140192616544000= Starting new HTTP connection (1): 240.10.129.80

3.2 日志grok解析

%{WORD:ss1_word} %{TIMESTAMP_ISO8601:ss2_stamp_time} %{LOGLEVEL:ss3_loglevel} %{PROG:ss4_filepath}:%{NUMBER:ss5_number} %{USERNAME:ss6_username} %{USERNAME:ss7_word} %{NUMBER:ss8_number}= (?<ss9>(.*)): %{IPORHOST:ss10_ip}

3.3 解析结果

{
  "ss2_stamp_time": "2015-11-03 03:35:50.283",
  "ss4_filepath": "/usr/lib/python2.6/site-packages/urllib3/connectionpool.py",
  "ss6_username": "_new_conn",
  "ss3_loglevel": "INFO",
  "ss5_number": "203",
  "ss7_word": "-",
  "ss10_ip": "240.10.129.80",
  "ss8_number": "140192616544000",
  "ss9": "Starting new HTTP connection (1)",
  "ss1_word": "vclound"
}

grok截取指定字符

参考：https://blog.csdn.net/cai750415222/article/details/86614854

表达式：(?<temMsg>(.*)(?=Report)/?) 获取Report之前的字符
表达式：(?<temMsg>(?=Report)(.*)/?) 获取包含Report及之后的字符
表达式：(?<temMsg>(.*)(?<=Report)/?) 获取不含Report及之后的字符

表达式：(?=Report)中的等于【=】符号如果换成【<=】表示不包含本身

表达式：(?<temMsg>(?<=Report).*?(?=msg)) 截取Report和msg之间的值 不包含Report和msg本身
表达式：(?<temMsg>(Report).*?(?=msg)) 截取包含Report但不包含msg
表达式：(?<temMsg>(?<=Report).*?(msg))截取不包含Report但包含msg
表达式：(?<temMsg>(Report).*?(msg|request))输出以Report开头,以msg或者以request结尾的所有包含头尾信息
表达式：(?<temMsg>(Report).*?(?=(msg|request)))输出以Report开头,以msg或者以request结尾的不包含头尾信息

如果是单个字符串(Report).*?(?=msg) 可以写成Report.*?(?=msg)
表达式：(?<file>([\s\S]{50})) 截取日志前50个字符,作为file的值