数据收集
一、收集模式
数据收集模式可以根据行为模式,分为应用直连上送和中间异步上送。
1、直连上送
以java+log4j2为例:
- 通过socket appender上送
log4j2配置:
<appender>
<Socket name="Logstash_client" host="localhost" port="5044" protocol="TCP">
<PatternLayout pattern="${PATTERN}" />
</Socket>
</appender>
LogStash配置:
input {
tcp {
host =>"127.0.0.1"
port => 5044
}
}
filter {
json {
source => "message"
add_field => ["type", "%{dtype}"]
remove_field => [ "server", "server.fqdn", "timestamp" ]
}
}
output {
stdout { codec => rubydebug }
if "_jsonparsefailure" not in [tags] {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-%{dtype}-%{action}"
}
}
}
这种方式有个缺点是LogStash重启之后不会自动重连,需要应用也重启
- 应用使用logstash-gelf上送
maven:
<dependency>
<groupId>biz.paluch.logging</groupId>
<artifactId>logstash-gelf</artifactId>
<version>1.11.1</version>
</dependency>
log4j2:
<appender>
<Gelf name="gelf" host="udp:localhost" port="12201" version="1.1" extractStackTrace="true"
filterStackTrace="true" mdcProfiling="true" includeFullMdc="true" maximumMessageSize="8192"
originHost="%host{fqdn}" additionalFieldTypes="fieldName1=String,fieldName2=Double,fieldName3=Long">
<Field name="timestamp" pattern="%d{dd MMM yyyy HH:mm:ss,SSS}" />
<Field name="level" pattern="%level" />
<Field name="simpleClassName" pattern="%C{1}" />
<Field name="className" pattern="%C" />
<Field name="server" pattern="%host" />
<Field name="server.fqdn" pattern="%host{fqdn}" />
<!-- This is a static field -->
<Field name="fieldName2" literal="fieldValue2" />
<!-- This is a field using MDC -->
<Field name="mdcField2" mdc="mdcField2" />
<DynamicMdcFields regex="mdc.*" />
<DynamicMdcFields regex="(mdc|MDC)fields" />
</Gelf>
</appender>
<root level="info">
<appender-ref ref="gelf" />
</root>
LogStash配置:
input {
gelf {
port => 12201
}
}
filter {
json {
source => "message"
add_field => ["type", "%{dtype}"]
remove_field => [ "server", "server.fqdn", "timestamp" ]
}
}
output {
stdout { codec => rubydebug }
if "_jsonparsefailure" not in [tags] {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-%{dtype}-%{action}"
}
}
}