基于ELK的分布式日志存储、检索系统之数据收集

最新推荐文章于 2024-08-15 10:05:21 发布

wrke1990

最新推荐文章于 2024-08-15 10:05:21 发布

阅读量232

点赞数

分类专栏： ELK

本文链接：https://blog.csdn.net/wrke1990/article/details/107664212

版权

ELK 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

数据收集

一、收集模式

数据收集模式可以根据行为模式，分为应用直连上送和中间异步上送。

1、直连上送

以java+log4j2为例：

通过socket appender上送
log4j2配置:

<appender>
<Socket name="Logstash_client" host="localhost" port="5044" protocol="TCP">
           <PatternLayout pattern="${PATTERN}" />
       </Socket>
 </appender>

LogStash配置：

input {
  tcp {
  	host =>"127.0.0.1"
    port => 5044
  }
}
filter {
  json {
   source => "message"
   add_field => ["type", "%{dtype}"]
   remove_field => [ "server", "server.fqdn", "timestamp" ]
  }
}
output {
  stdout { codec => rubydebug }
  if "_jsonparsefailure" not in [tags] {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logstash-%{dtype}-%{action}"
    } 
  }
}

这种方式有个缺点是LogStash重启之后不会自动重连，需要应用也重启

应用使用logstash-gelf上送
maven：

<dependency>
    <groupId>biz.paluch.logging</groupId>
    <artifactId>logstash-gelf</artifactId>
    <version>1.11.1</version>
</dependency>

log4j2:

  <appender>
    <Gelf name="gelf" host="udp:localhost" port="12201" version="1.1" extractStackTrace="true"
          filterStackTrace="true" mdcProfiling="true" includeFullMdc="true" maximumMessageSize="8192"
          originHost="%host{fqdn}" additionalFieldTypes="fieldName1=String,fieldName2=Double,fieldName3=Long">
      <Field name="timestamp" pattern="%d{dd MMM yyyy HH:mm:ss,SSS}" />
      <Field name="level" pattern="%level" />
      <Field name="simpleClassName" pattern="%C{1}" />
      <Field name="className" pattern="%C" />
      <Field name="server" pattern="%host" />
      <Field name="server.fqdn" pattern="%host{fqdn}" />

      <!-- This is a static field -->
      <Field name="fieldName2" literal="fieldValue2" />

      <!-- This is a field using MDC -->
      <Field name="mdcField2" mdc="mdcField2" />
      <DynamicMdcFields regex="mdc.*" />
      <DynamicMdcFields regex="(mdc|MDC)fields" />
    </Gelf>
    </appender>


    <root level="info">
         <appender-ref ref="gelf" /> 
    </root>

LogStash配置：

input {
  gelf {
    port => 12201
  }
}
filter {
  json {
   source => "message"
   add_field => ["type", "%{dtype}"]
   remove_field => [ "server", "server.fqdn", "timestamp" ]
  }
}
output {
  stdout { codec => rubydebug }
  if "_jsonparsefailure" not in [tags] {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logstash-%{dtype}-%{action}"
    } 
  }
}