日志采集中的日志打印和logstash配置

一.引言

上一篇文章（https://blog.csdn.net/qq_43216019/article/details/129019631）学习记录关于使用filebeat+kafka+logstash+elasticsearch+kibana,今天主要完成了日志中打印格式的设置和logstash的配置。本来刚开始是使用grok格式匹配，后来发现json基本不需要处理，只要打印的日志是json格式，就可以直接解析。对于不规则的日志采集后续用到会对grok这一logstash中最重要的格式匹配工具做记录。

二.代码中日志打印设置

• 添加采集日志打印类

  @Slf4j
  public class LogRecorder {
      public static void log(Map<String,Object> map){
          log.info("{}", JSON.toJSONString(map));
      }
  }
  

• logback文件设置,只打印json内容

  <appender name="RECORD.INFO" class="ch.qos.logback.core.rolling.RollingFileAppender">
          <file>logs/log-record.log</file>
          <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
              <fileNamePattern>logs/record.%d{yyyy-MM-dd}-%i.log.zip</fileNamePattern>
              <MaxHistory>180</MaxHistory>
              <TimeBasedFileNamingAndTriggeringPolicy
                      class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                  <MaxFileSize>100MB</MaxFileSize>
              </TimeBasedFileNamingAndTriggeringPolicy>
          </rollingPolicy>
          <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
              <!--            <pattern>[%d{yyyy-MM-dd HH:mm:ss}] %-5level %t %c{10}:%line - %msg%n</pattern>-->
              <pattern>%msg%n</pattern>
          </encoder>
      </appender>
            
  <logger name="com.xft.learn.log.LogRecorder" additivity="false">
          <appender-ref ref="RECORD.INFO"/>
          <level value="INFO"/>
      </logger>
  

• 打印内容：

  {"time":"2023-02-14 19:11:59","token":"dsfd"}
  {"time":"2023-02-14 21:54:27","token":"dsfd"}
  

三.logstash格式数据处理配置

• 配置文件first_pipeline.conf

  input {
    kafka {
      bootstrap_servers => "localhost:9092"
      topics => "demo-topic"
      group_id => "test"
      # 读取为json格式
      codec => "json"
    }
  }
  
  filter {
    # 将json体映射为key-value
    json {
      source => "message"
      remove_field => ["version","message","host","ecs","log","input","@version","agent","timestamp"]
    }
  }
  
  output {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "log-record-%{+YYYY.MM.dd}"
    }
    # 控制台查看输出
    # stdout	{
    #  codec => rubydebug
    #}
  }      
  

• 控制台输出

  {
           "event" => {
          "original" => "{\"@timestamp\":\"2023-02-15T01:57:13.869Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.3.3\"},\"host\":{\"name\":\"VM-4-16-centos\"},\"log\":{\"file\":{\"path\":\"/data/project/learn/logs/log-record.log\"},\"offset\":150},\"message\":\"{\\\"time\\\":\\\"2023-02-15 09:57:13\\\",\\\"token\\\":\\\"e4323432\\\"}\",\"input\":{\"type\":\"log\"},\"agent\":{\"ephemeral_id\":\"cea64fa5-f917-49be-a4a5-ea69ffc5887d\",\"id\":\"2f4e036c-d4d8-4209-afba-b93fbd623d13\",\"name\":\"VM-4-16-centos\",\"type\":\"filebeat\",\"version\":\"8.3.3\"},\"ecs\":{\"version\":\"8.0.0\"}}"
      },
      "@timestamp" => 2023-02-15T01:57:13.869Z,
           "token" => "e4323432",
            "time" => "2023-02-15 09:57:13"
  }
  

注意在input中要以josn格式读取，然后在filter处理时将json中的字段直接映射为es中的独立字段.后续通过自定义注解实现接口日志采集以及通过docker搭建日志采集系统。