Logstash and Log4j

I wanted to log from a log4j process through to Logstash, and have the logging stored in Elastic search. This can be done using the code at https://github.com/logstash/log4j-jsonevent-layout

To make things easy for my test, I put the source code for net.logstash.log4j.JSONEventLayoutV1and net.logstash.log4j.data.HostData into my source tree.

I then added json-smart-1.1.1.jar to the classpath (from https://code.google.com/p/json-smart/downloads/detail?name=json-smart-1.1.1.jar&can=2&q=)

I then created the following log4j appender in log4j.xml then used it in code:

<appender name="LogStash"
    class="org.apache.log4j.DailyRollingFileAppender">
    <param name="File" value="logs/panCoreSaas.log" />
    <param name="Append" value="true" />
    <layout class="net.logstash.log4j.JSONEventLayoutV1"/>
</appender>

Using this appender produces logs in JSON format:

{  "thread_name":"http-8080-3@#zUaiBHqDI66O25BntXrghixzzEljOHlHGPB0dNmz4RoX3gNWyB",
    "message":"Calling FindEvents",
    "@timestamp":"2014-11-11T12:48:35.094Z",
    "level":"INFO",
    "mdc": {},
    "file":"WebServiceTransactionRunner.java",
    "class":"com.pancredit.eaitoolkit.framework.runner.ws.WebServiceTransactionRunner",
    "line_number":"638",
    "logger_name":"com.pancredit.eaitoolkit.framework.runner.ws.WebServiceTransactionRunner",
    "method":"logCall",
    "@version":1,
    "source_host":"greenfinch"
}

Create a log4j_file.conf with the following contents:

input {
  file {
    path => "/apps/logstash/logstash-1.4.2/panCoreSaas.log"
    start_position => beginning
  }
}

output {
  elasticsearch {
    host => localhost
  }
  stdout { codec => rubydebug }
}

and then start logstash to process the file

$ bin/logstash -f log4j_json.conf