使用filebeat和logstash解析java的log4j日志

目的:我们使用filebeat来接受日志,并将日志做一个简单的处理,然后发给logstash,使用logstash的filter的grok来匹配日志,将日志解析成json格式并将日志输出在logstash的控制台上.

首先看一下我们的日志样例,这是一个标准的java的log4j日志:

2019-02-28 10:24:48 org.tinyradius.util.RadiusClient.authenticate(RadiusClient.java:81) INFO  (org.tinyradius.util.RadiusClient:81)- send Access-Request packet: Access-Request, ID 7
User-Name: admin
2019-02-28 10:24:48 org.tinyradius.util.RadiusClient.authenticate(RadiusClient.java:85) INFO  (org.tinyradius.util.RadiusClient:85)- received packet: Access-Accept, ID 7
Login-LAT-Group: 0x313233343536
2019-02-28 10:24:49 com.bas.pub.util.PropertiesHelper.getPropFile(PropertiesHelper.java:44) INFO  (com.bas.pub.util.PropertiesHelper:44)- 1-propFile==config.properties
2019-02-28 10:24:49 com.bas.pub.util.PropertiesHelper.getPropFile(PropertiesHelper.java:49) INFO  (com.bas.pub.util.PropertiesHelper:49)- 3-in==java.io.ByteArrayInputStream@32250194
2019-02-28 15:06:24 org.tinyradius.util.RadiusClient.authenticate(RadiusClient.java:81) INFO  (org.tinyradius.util.RadiusClient:81)- send Access-Request packet: Access-Request, ID 8
User-Name: admin
2019-02-28 15:06:24 org.tinyradius.util.RadiusClient.communicate(RadiusClient.java:262) INFO  (org.tinyradius.util.RadiusClient:262)- communication failure, retry 1
2019-02-28 15:06:24 org.tinyradius.util.RadiusClient.communicate(RadiusClient.java:262) INFO  (org.tinyradius.util.RadiusClient:262)- communication failure, retry 2
2019-02-28 15:06:24 org.tinyradius.util.RadiusClient.communicate(RadiusClient.java:262) INFO  (org.tinyradius.util.RadiusClient:262)- communication failure, retry 3
2019-02-28 15:06:24 org.tinyradius.util.RadiusClient.communicate(RadiusClient.java:262) INFO  (org.tinyradius.util.RadiusClient:262)- communication failure, retry 4
2019-02-28 15:06:24 org.tinyradius.util.RadiusClient.communicate(RadiusClient.java:262) INFO  (org.tinyradius.util.RadiusClient:262)- communication failure, retry 5
2019-02-28 15:06:24 org.tinyradius.util.RadiusClient.communicate(RadiusClient.java:255) ERROR (org.tinyradius.util.RadiusClient:255)- communication failure (timeout), no more retries
2019-02-28 15:06:24 com.erp.action.LoginAction.load(LoginAction.java:230) ERROR (com.erp.action.LoginAction:230)- Radius认证异常，请与系统管理员联系
java.net.SocketTimeoutException: Receive timed out
	at java.net.DualStackPlainDatagramSocketImpl.socketReceiveOrPeekData(Native Method)
	at java.net.DualStackPlainDatagramSocketImpl.receive0(DualStackPlainDatagramSocketImpl.java:121)
	at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:146)
	at java.net.DatagramSocket.receive(DatagramSocket.java:816)
	at org.tinyradius.util.RadiusClient.communicate(RadiusClient.java:249)
	at org.tinyradius.util.RadiusClient.authenticate(RadiusClient.java:83)
	at org.tinyradius.util.RadiusClient.authenticate(RadiusClient.java:65)
	at com.erp.action.LoginAction.load(LoginAction.java:207)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at ognl.OgnlRuntime.invokeMethod(OgnlRuntime.java:892)
	at ognl.OgnlRuntime.callAppropriateMethod(OgnlRuntime.java:1294)
	at ognl.ObjectMethodAccessor.callMethod(ObjectMethodAccessor.java:68)
	at com.opensymphony.xwork2.ognl.accessor.XWorkMethodAccessor.callMethodWithDebugInfo(XWorkMethodAccessor.java:117)
	at com.opensymphony.xwork2.ognl.accessor.XWorkMethodAccessor.callMethod(XWorkMethodAccessor.java:108)
	at ognl.OgnlRuntime.callMethod(OgnlRuntime.java:1370)
	at ognl.ASTMethod.getValueBody(ASTMethod.java:91)
	at ognl.SimpleNode.evaluateGetValueBody(SimpleNode.java:212)
	at ognl.SimpleNode.getValue(SimpleNode.java:258)
	at ognl.Ognl.getValue(Ognl.java:467)
	at ognl.Ognl.getValue(Ognl.java:431)
	at com.opensymphony.xwork2.ognl.OgnlUtil$3.execute(OgnlUtil.java:352)
	at com.opensymphony.xwork2.ognl.OgnlUtil.compileAndExecuteMethod(OgnlUtil.java:404)
	at com.opensymphony.xwork2.ognl.OgnlUtil.callMethod(OgnlUtil.java:350)
	at com.opensymphony.xwork2.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:430)
	at com.opensymphony.xwork2.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:290)
	at com