直接进入正题,我在搭建elk日志,使用最简单的log4j2 socket json格式 输出到logstash.
但是logstash报错如下:
[WARN ] 2023-08-30 10:15:17.766 [nioEventLoopGroup-2-2] jsonlines - JSON parse error, original data now in message field {:message=>“incompatible json object type=java.lang.String , only hash map or arrays are supported”, :exception=>LogStash::Json::ParserError, :data=>" “message” : “Started FasterUserApplication in 1.472 seconds (JVM running for 2.232)”,\r"}
总是提示json格式有问题:
logstash配置如图:
input {
tcp {
port => 4560
codec => json_lines
}
}
output {
elasticsearch {
hosts => ["https://192.168.245.140:9200"]
index => "applog-%{+YYYY.MM.dd}"
template_name => "applog"
user => "applog"
password => "applog"
cacert => "/usr/local/service/elasticsearch-8.9.1/config/certs/http_ca.crt"
}
file {
path => "/usr/local/service/test.log"
}
}
log4j2最开始配置如下,下称为配置1:
<Socket name="socket" host="192.168.245.140" port="4560" protocol="TCP" >
<JsonLayout properties="true" >
<KeyValuePair key="project" value="demo1"></KeyValuePair>
</JsonLayout>
</Socket>
以上配置,每次有日志时,logstash就抛出json解析错误。
[WARN ] 2023-08-30 10:15:17.766 [nioEventLoopGroup-2-2] jsonlines - JSON parse error, original data now in message field {:message=>“incompatible json object type=java.lang.String , only hash map or arrays are supported”, :exception=>LogStash::Json::ParserError, :data=>" “message” : “Started FasterUserApplication in 1.472 seconds (JVM running for 2.232)”,\r"}
后经查证,log4j2修改配置如下,下称为配置2:
<Socket name="socket" host="192.168.245.140" port="4560" protocol="TCP" >
<JsonLayout properties="true" complete="false" compact="true" eventEol="true"
locationInfo="false"
includeStacktrace="true" stacktraceAsString="true"
objectMessageAsJsonObject="true">
<KeyValuePair key="project" value="demo1"></KeyValuePair>
</JsonLayout>
</Socket>
错误就不再出现了,下面的配置主要是多了json不再美化输出,每条日志记录输出一行json。
以下为对比:
log4j配置1,json日志输出格式为:
"C:\Program Files\Java\jdk-13.0.1\bin\java.exe" -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:10980,suspend=y,server=n -XX:TieredStopAtLevel=1 -noverify -Dspring.output.ansi.enabled=always -Dcom.sun.management.jmxremote -Dspring.jmx.enabled=true -Dspring.liveBeansView.mbeanDomain -Dspring.application.admin.enabled=true -javaagent:C:\Users\Miste\.IntelliJIdea2019.3\system\captureAgent\debugger-agent.jar -Dfile.encoding=UTF-8 -classpath "D:\workspace\playground\faster-demo\faster-user\target\classes;D:\workspace\mavenLocalRepository\org\springframework\boot\spring-boot-starter-log4j2\2.5.5\spring-boot-starter-log4j2-2.5.5.jar;D:\workspace\mavenLocalRepository\org\apache\logging\log4j\log4j-slf4j-impl\2.14.1\log4j-slf4j-impl-2.14.1.jar;D:\workspace\mavenLocalRepository\org\slf4j\slf4j-api\1.7.32\slf4j-api-1.7.32.jar;D:\workspace\mavenLocalRepository\org\apache\logging\log4j\log4j-api\2.14.1\log4j-api-2.14.1.jar;D:\workspace\mavenLocalRepository\org\apache\logging\log4j\log4j-core\2.14.1\log4j-core-2.14.1.jar;D:\workspace\mavenLocalRepository\org\apache\logging\log4j\log4j-jul\2.14.1\log4j-jul-2.14.1.jar;D:\workspace\mavenLocalRepository\org\slf4j\jul-to-slf4j\1.7.32\jul-to-slf4j-1.7.32.jar;D:\workspace\mavenLocalRepository\com\lmax\disruptor\3.4.2\disruptor-3.4.2.jar;D:\workspace\mavenLocalRepository\org\springframework\boot\spring-boot-starter-web\2.5.5\spring-boot-starter-web-2.5.5.jar;D:\workspace\mavenLocalRepository\org\springframework\boot\spring-boot-starter\2.5.5\spring-boot-starter-2.5.5.jar;D:\workspace\mavenLocalRepository\org\springframework\boot\spring-boot\2.5.5\spring-boot-2.5.5.jar;D:\workspace\mavenLocalRepository\org\springframework\boot\spring-boot-autoconfigure\2.5.5\spring-boot-autoconfigure-2.5.5.jar;D:\workspace\mavenLocalRepository\jakarta\annotation\jakarta.annotation-api\1.3.5\jakarta.annotation-api-1.3.5.jar;D:\workspace\mavenLocalRepository\org\yaml\snakeyaml\1.28\snakeyaml-1.28.jar;D:\workspace\mavenLocalRepository\org\springframework\boot\spring-boot-starter-json\2.5.5\spring-boot-starter-json-2.5.5.jar;D:\workspace\mavenLocalRepository\com\fasterxml\jackson\core\jackson-databind\2.12.5\jackson-databind-2.12.5.jar;D:\workspace\mavenLocalRepository\com\fasterxml\jackson\core\jackson-annotations\2.12.5\jackson-annotations-2.12.5.jar;D:\workspace\mavenLocalRepository\com\fasterxml\jackson\core\jackson-core\2.12.5\jackson-core-2.12.5.jar;D:\workspace\mavenLocalRepository\com\fasterxml\jackson\datatype\jackson-datatype-jdk8\2.12.5\jackson-datatype-jdk8-2.12.5.jar;D:\workspace\mavenLocalRepository\com\fasterxml\jackson\datatype\jackson-datatype-jsr310\2.12.5\jackson-datatype-jsr310-2.12.5.jar;D:\workspace\mavenLocalRepository\com\fasterxml\jackson\module\jackson-module-parameter-names\2.12.5\jackson-module-parameter-names-2.12.5.jar;D:\workspace\mavenLocalRepository\org\springframework\boot\spring-boot-starter-tomcat\2.5.5\spring-boot-starter-tomcat-2.5.5.jar;D:\workspace\mavenLocalRepository\org\apache\tomcat\embed\tomcat-embed-core\9.0.53\tomcat-embed-core-9.0.53.jar;D:\workspace\mavenLocalRepository\org\apache\tomcat\embed\tomcat-embed-el\9.0.53\tomcat-embed-el-9.0.53.jar;D:\workspace\mavenLocalRepository\org\apache\tomcat\embed\tomcat-embed-websocket\9.0.53\tomcat-embed-websocket-9.0.53.jar;D:\workspace\mavenLocalRepository\org\springframework\spring-web\5.3.10\spring-web-5.3.10.jar;D:\workspace\mavenLocalRepository\org\springframework\spring-beans\5.3.10\spring-beans-5.3.10.jar;D:\workspace\mavenLocalRepository\org\springframework\spring-webmvc\5.3.10\spring-webmvc-5.3.10.jar;D:\workspace\mavenLocalRepository\org\springframework\spring-aop\5.3.10\spring-aop-5.3.10.jar;D:\workspace\mavenLocalRepository\org\springframework\spring-context\5.3.10\spring-context-5.3.10.jar;D:\workspace\mavenLocalRepository\org\springframework\spring-expression\5.3.10\spring-expression-5.3.10.jar;D:\workspace\mavenLocalRepository\mysql\mysql-connector-java\8.0.26\mysql-connector-java-8.0.26.jar;D:\workspace\mavenLocalRepository\org\springframework\spring-core\5.3.10\spring-core-5.3.10.jar;D:\workspace\mavenLocalRepository\org\springframework\spring-jcl\5.3.10\spring-jcl-5.3.10.jar;H:\ProframFiles\IntelliJ IDEA 2019.3\lib\idea_rt.jar" com.github.misterchangray.user.FasterUserApplication
Java HotSpot(TM) 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release.
Connected to the target VM, address: '127.0.0.1:0', transport: 'socket'
Java HotSpot(TM) 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.5.5)
{
"instant" : {
"epochSecond" : 1693363254,
"nanoOfSecond" : 88010000
},
"thread" : "main",
"level" : "INFO",
"loggerName" : "com.github.misterchangray.user.FasterUserApplication",
"message" : "Starting FasterUserApplication using Java 13.0.1 on DESKTOP-KVEV94V with PID 13812 (D:\\workspace\\playground\\faster-demo\\faster-user\\target\\classes started by Miste in D:\\workspace\\playground\\faster-demo)",
"endOfBatch" : false,
"loggerFqcn" : "org.apache.commons.logging.LogAdapter$Log4jLog",
"contextMap" : { },
"threadId" : 1,
"threadPriority" : 5,
"project" : "demo1"
}
{
"instant" : {
"epochSecond" : 1693363254,
"nanoOfSecond" : 167298600
},
"thread" : "main",
"level" : "DEBUG",
"loggerName" : "com.github.misterchangray.user.FasterUserApplication",
"message" : "Running with Spring Boot v2.5.5, Spring v5.3.10",
"endOfBatch" : false,
"loggerFqcn" : "org.apache.commons.logging.LogAdapter$Log4jLog",
"contextMap" : { },
"threadId" : 1,
"threadPriority" : 5,
"project" : "demo1"
}
{
"instant" : {
"epochSecond" : 1693363254,
"nanoOfSecond" : 169298100
},
"thread" : "main",
"level" : "INFO",
"loggerName" : "com.github.misterchangray.user.FasterUserApplication",
"message" : "No active profile set, falling back to default profiles: default",
"endOfBatch" : false,
"loggerFqcn" : "org.apache.commons.logging.LogAdapter$Log4jLog",
"contextMap" : { },
"threadId" : 1,
"threadPriority" : 5,
"project" : "demo1"
}
配置2日志输出格式为:
"C:\Program Files\Java\jdk-13.0.1\bin\java.exe" -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:8338,suspend=y,server=n -XX:TieredStopAtLevel=1 -noverify -Dspring.output.ansi.enabled=always -Dcom.sun.management.jmxremote -Dspring.jmx.enabled=true -Dspring.liveBeansView.mbeanDomain -Dspring.application.admin.enabled=true -javaagent:C:\Users\Miste\.IntelliJIdea2019.3\system\captureAgent\debugger-agent.jar -Dfile.encoding=UTF-8 -classpath "D:\workspace\playground\faster-demo\faster-user\target\classes;D:\workspace\mavenLocalRepository\org\springframework\boot\spring-boot-starter-log4j2\2.5.5\spring-boot-starter-log4j2-2.5.5.jar;D:\workspace\mavenLocalRepository\org\apache\logging\log4j\log4j-slf4j-impl\2.14.1\log4j-slf4j-impl-2.14.1.jar;D:\workspace\mavenLocalRepository\org\slf4j\slf4j-api\1.7.32\slf4j-api-1.7.32.jar;D:\workspace\mavenLocalRepository\org\apache\logging\log4j\log4j-api\2.14.1\log4j-api-2.14.1.jar;D:\workspace\mavenLocalRepository\org\apache\logging\log4j\log4j-core\2.14.1\log4j-core-2.14.1.jar;D:\workspace\mavenLocalRepository\org\apache\logging\log4j\log4j-jul\2.14.1\log4j-jul-2.14.1.jar;D:\workspace\mavenLocalRepository\org\slf4j\jul-to-slf4j\1.7.32\jul-to-slf4j-1.7.32.jar;D:\workspace\mavenLocalRepository\com\lmax\disruptor\3.4.2\disruptor-3.4.2.jar;D:\workspace\mavenLocalRepository\org\springframework\boot\spring-boot-starter-web\2.5.5\spring-boot-starter-web-2.5.5.jar;D:\workspace\mavenLocalRepository\org\springframework\boot\spring-boot-starter\2.5.5\spring-boot-starter-2.5.5.jar;D:\workspace\mavenLocalRepository\org\springframework\boot\spring-boot\2.5.5\spring-boot-2.5.5.jar;D:\workspace\mavenLocalRepository\org\springframework\boot\spring-boot-autoconfigure\2.5.5\spring-boot-autoconfigure-2.5.5.jar;D:\workspace\mavenLocalRepository\jakarta\annotation\jakarta.annotation-api\1.3.5\jakarta.annotation-api-1.3.5.jar;D:\workspace\mavenLocalRepository\org\yaml\snakeyaml\1.28\snakeyaml-1.28.jar;D:\workspace\mavenLocalRepository\org\springframework\boot\spring-boot-starter-json\2.5.5\spring-boot-starter-json-2.5.5.jar;D:\workspace\mavenLocalRepository\com\fasterxml\jackson\core\jackson-databind\2.12.5\jackson-databind-2.12.5.jar;D:\workspace\mavenLocalRepository\com\fasterxml\jackson\core\jackson-annotations\2.12.5\jackson-annotations-2.12.5.jar;D:\workspace\mavenLocalRepository\com\fasterxml\jackson\core\jackson-core\2.12.5\jackson-core-2.12.5.jar;D:\workspace\mavenLocalRepository\com\fasterxml\jackson\datatype\jackson-datatype-jdk8\2.12.5\jackson-datatype-jdk8-2.12.5.jar;D:\workspace\mavenLocalRepository\com\fasterxml\jackson\datatype\jackson-datatype-jsr310\2.12.5\jackson-datatype-jsr310-2.12.5.jar;D:\workspace\mavenLocalRepository\com\fasterxml\jackson\module\jackson-module-parameter-names\2.12.5\jackson-module-parameter-names-2.12.5.jar;D:\workspace\mavenLocalRepository\org\springframework\boot\spring-boot-starter-tomcat\2.5.5\spring-boot-starter-tomcat-2.5.5.jar;D:\workspace\mavenLocalRepository\org\apache\tomcat\embed\tomcat-embed-core\9.0.53\tomcat-embed-core-9.0.53.jar;D:\workspace\mavenLocalRepository\org\apache\tomcat\embed\tomcat-embed-el\9.0.53\tomcat-embed-el-9.0.53.jar;D:\workspace\mavenLocalRepository\org\apache\tomcat\embed\tomcat-embed-websocket\9.0.53\tomcat-embed-websocket-9.0.53.jar;D:\workspace\mavenLocalRepository\org\springframework\spring-web\5.3.10\spring-web-5.3.10.jar;D:\workspace\mavenLocalRepository\org\springframework\spring-beans\5.3.10\spring-beans-5.3.10.jar;D:\workspace\mavenLocalRepository\org\springframework\spring-webmvc\5.3.10\spring-webmvc-5.3.10.jar;D:\workspace\mavenLocalRepository\org\springframework\spring-aop\5.3.10\spring-aop-5.3.10.jar;D:\workspace\mavenLocalRepository\org\springframework\spring-context\5.3.10\spring-context-5.3.10.jar;D:\workspace\mavenLocalRepository\org\springframework\spring-expression\5.3.10\spring-expression-5.3.10.jar;D:\workspace\mavenLocalRepository\mysql\mysql-connector-java\8.0.26\mysql-connector-java-8.0.26.jar;D:\workspace\mavenLocalRepository\org\springframework\spring-core\5.3.10\spring-core-5.3.10.jar;D:\workspace\mavenLocalRepository\org\springframework\spring-jcl\5.3.10\spring-jcl-5.3.10.jar;H:\ProframFiles\IntelliJ IDEA 2019.3\lib\idea_rt.jar" com.github.misterchangray.user.FasterUserApplication
Java HotSpot(TM) 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release.
Connected to the target VM, address: '127.0.0.1:0', transport: 'socket'
Java HotSpot(TM) 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.5.5)
{"instant":{"epochSecond":1693362240,"nanoOfSecond":139475300},"thread":"main","level":"INFO","loggerName":"com.github.misterchangray.user.FasterUserApplication","message":"Starting FasterUserApplication using Java 13.0.1 on DESKTOP-KVEV94V with PID 21844 (D:\\workspace\\playground\\faster-demo\\faster-user\\target\\classes started by Miste in D:\\workspace\\playground\\faster-demo)","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","contextMap":{},"threadId":1,"threadPriority":5,"project":"demo1"}
{"instant":{"epochSecond":1693362240,"nanoOfSecond":202819200},"thread":"main","level":"DEBUG","loggerName":"com.github.misterchangray.user.FasterUserApplication","message":"Running with Spring Boot v2.5.5, Spring v5.3.10","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","contextMap":{},"threadId":1,"threadPriority":5,"project":"demo1"}
{"instant":{"epochSecond":1693362240,"nanoOfSecond":203813500},"thread":"main","level":"INFO","loggerName":"com.github.misterchangray.user.FasterUserApplication","message":"No active profile set, falling back to default profiles: default","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","contextMap":{},"threadId":1,"threadPriority":5,"project":"demo1"}
{"instant":{"epochSecond":1693362240,"nanoOfSecond":734127100},"thread":"main","level":"INFO","loggerName":"org.springframework.boot.web.embedded.tomcat.TomcatWebServer","message":"Tomcat initialized with port(s): 8089 (http)","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","contextMap":{},"threadId":1,"threadPriority":5,"project":"demo1"}
{"instant":{"epochSecond":1693362240,"nanoOfSecond":745097900},"thread":"main","level":"INFO","loggerName":"org.apache.coyote.http11.Http11NioProtocol","message":"Initializing ProtocolHandler [\"http-nio-8089\"]","endOfBatch":false,"loggerFqcn":"java.util.logging.Logger","contextMap":{},"threadId":1,"threadPriority":5,"project":"demo1"}