一.引言
上一篇文章(https://blog.csdn.net/qq_43216019/article/details/129019631)学习记录关于使用filebeat+kafka+logstash+elasticsearch+kibana,今天主要完成了日志中打印格式的设置和logstash的配置。本来刚开始是使用grok格式匹配,后来发现json基本不需要处理,只要打印的日志是json格式,就可以直接解析。对于不规则的日志采集后续用到会对grok这一logstash中最重要的格式匹配工具做记录。
二.代码中日志打印设置
-
添加采集日志打印类
@Slf4j public class LogRecorder { public static void log(Map<String,Object> map){ log.info("{}", JSON.toJSONString(map)); } }
-
logback文件设置,只打印json内容
<appender name="RECORD.INFO" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>logs/log-record.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <fileNamePattern>logs/record.%d{yyyy-MM-dd}-%i.log.zip</fileNamePattern> <MaxHistory>180</MaxHistory> <TimeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP"> <MaxFileSize>100MB</MaxFileSize> </TimeBasedFileNamingAndTriggeringPolicy> </rollingPolicy> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <!-- <pattern>[%d{yyyy-MM-dd HH:mm:ss}] %-5level %t %c{10}:%line - %msg%n</pattern>--> <pattern>%msg%n</pattern> </encoder> </appender> <logger name="com.xft.learn.log.LogRecorder" additivity="false"> <appender-ref ref="RECORD.INFO"/> <level value="INFO"/> </logger>
-
打印内容:
{"time":"2023-02-14 19:11:59","token":"dsfd"} {"time":"2023-02-14 21:54:27","token":"dsfd"}
三.logstash格式数据处理配置
-
配置文件first_pipeline.conf
input { kafka { bootstrap_servers => "localhost:9092" topics => "demo-topic" group_id => "test" # 读取为json格式 codec => "json" } } filter { # 将json体映射为key-value json { source => "message" remove_field => ["version","message","host","ecs","log","input","@version","agent","timestamp"] } } output { elasticsearch { hosts => ["http://localhost:9200"] index => "log-record-%{+YYYY.MM.dd}" } # 控制台查看输出 # stdout { # codec => rubydebug #} }
-
控制台输出
{ "event" => { "original" => "{\"@timestamp\":\"2023-02-15T01:57:13.869Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.3.3\"},\"host\":{\"name\":\"VM-4-16-centos\"},\"log\":{\"file\":{\"path\":\"/data/project/learn/logs/log-record.log\"},\"offset\":150},\"message\":\"{\\\"time\\\":\\\"2023-02-15 09:57:13\\\",\\\"token\\\":\\\"e4323432\\\"}\",\"input\":{\"type\":\"log\"},\"agent\":{\"ephemeral_id\":\"cea64fa5-f917-49be-a4a5-ea69ffc5887d\",\"id\":\"2f4e036c-d4d8-4209-afba-b93fbd623d13\",\"name\":\"VM-4-16-centos\",\"type\":\"filebeat\",\"version\":\"8.3.3\"},\"ecs\":{\"version\":\"8.0.0\"}}" }, "@timestamp" => 2023-02-15T01:57:13.869Z, "token" => "e4323432", "time" => "2023-02-15 09:57:13" }
注意在input中要以josn格式读取,然后在filter处理时将json中的字段直接映射为es中的独立字段.后续通过自定义注解实现接口日志采集以及通过docker搭建日志采集系统。