I wanted to log from a log4j process through to Logstash, and have the logging stored in Elastic search. This can be done using the code at https://github.com/logstash/log4j-jsonevent-layout
To make things easy for my test, I put the source code for net.logstash.log4j.JSONEventLayoutV1
and net.logstash.log4j.data.HostData
into my source tree.
I then added json-smart-1.1.1.jar
to the classpath (from https://code.google.com/p/json-smart/downloads/detail?name=json-smart-1.1.1.jar&can=2&q= )
I then created the following log4j appender in log4j.xml
then used it in code:
<appender name= "LogStash"
class= "org.apache.log4j.DailyRollingFileAppender" >
<param name= "File" value= "logs/panCoreSaas.log" />
<param name= "Append" value= "true" />
<layout class= "net.logstash.log4j.JSONEventLayoutV1" />
</appender>
Using this appender produces logs in JSON format:
{ "thread_name" : "http-8080-3@#zUaiBHqDI66O25BntXrghixzzEljOHlHGPB0dNmz4RoX3gNWyB" ,
"message" : "Calling FindEvents" ,
"@timestamp" : "2014-11-11T12:48:35.094Z" ,
"level" : "INFO" ,
"mdc" : {},
"file" : "WebServiceTransactionRunner.java" ,
"class" : "com.pancredit.eaitoolkit.framework.runner.ws.WebServiceTransactionRunner" ,
"line_number" : "638" ,
"logger_name" : "com.pancredit.eaitoolkit.framework.runner.ws.WebServiceTransactionRunner" ,
"method" : "logCall" ,
"@version" : 1 ,
"source_host" : "greenfinch"
}
Create a log4j_file.conf with the following contents:
input {
file {
path => "/apps/logstash/logstash-1.4.2/panCoreSaas.log"
start_position => beginning
}
}
output {
elasticsearch {
host => localhost
}
stdout { codec => rubydebug }
}
and then start logstash to process the file
$ bin/logstash -f log4j_json.conf