针对日志产生上报到es的实践

背景

日志检索是在服务治理中用来快速定位错误的一个主要方式，在没有日志检索服务之前，都是需要通过获取相应的日志来进行排查定位。这是一个传统的方式。

为了更快地提升排查效率，现在市场上出现了很多关于日志检索的框架，比较常见的是ELK的搭建方式。

在这里面logStash是相对比较重的软件，目前针对日志，采用filebeat来替换采集。

遇到问题

因为elasticsearch是使用json格式来持久化文档，为了更好地控制日志检索的效果，我们需要特别的处理日志上报的json内容，同时也可以减少在filebeat的处理。

解决步骤

logback的配置

利用logback里面的日志格式转换，来对输入的日志做json转换处理。logback-spring.xml的相关配置。

<!-- Appender to log to file in a JSON format --> <appender name="logstash" class="ch.qos.logback.core.rolling.RollingFileAppender"> <filter class="ch.qos.logback.classic.filter.ThresholdFilter"> <level>INFO</level> </filter> <file>${LOG_HOME}/logstash.json</file> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <fileNamePattern>logstash.json.%d{yyyy-MM-dd}.gz</fileNamePattern> <maxHistory>7</maxHistory> <totalSizeCap>1GB</totalSizeCap> </rollingPolicy> <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder"> <providers> <timestamp> <timeZone>UTC+8</timeZone> </timestamp> <pattern> <pattern> { "date":"%d{yyyy-MM-dd HH:mm:ss.SSS}", "level": "%level", "service": "${springAppName:-}", "trace": "%X{traceid:-}", "pid": "${PID:-}", "thread": "%thread", "class": "%logger", "msg": "%message", "stack_trace": "%exception{20}" } </pattern> </pattern> </providers> </encoder> </appender>

特别注意

因为logback会把所有的info以上的级别都输入到es当中，这里面可能会包含类似，spring tomcat等一些框架或者组件输出的日志，但是这些日志并不是我们需要关注的，为了节省ES的存储空间，需要针对这些框架进行排除

traceId 是分布式跟踪ID 这个根据代码实现

因此，针对logback需要加上以下配置【根据实际情况调配】，事实上这个在实际的工作当中，也是需要配置的，不是说写到es的时候才需要调整

<logger name="org.apache" level="WARN"> <appender-ref ref="WARN" /> <appender-ref ref="ERROR" /> </logger> <logger name="org.mybatis" level="ERROR"> <appender-ref ref="ERROR" /> </logger> <logger name="org.hibernate" level="ERROR"> <appender-ref ref="ERROR" /> </logger> <logger name="org.springframework" level="WARN"> <appender-ref ref="WARN" /> <appender-ref ref="ERROR" /> </logger> <logger name="com.netflix" level="WARN"> <appender-ref ref="WARN" /> <appender-ref ref="ERROR" /> </logger>

针对日志的记录，在代码直接使用slf4j的log直接输出就好。

filebeat配置

filebeat.inputs: - type: log enabled: true paths: - F:\data\logs\java\spring-cloud-eureka-client\*.json fields: ipaddr: ${serverIP} fields_under_root: true processors: - decode_json_fields: fields: ['message'] target: '' overwrite_keys: true - drop_fields: fields: ["message", "ecs", "agent", "log","input"] setup.template.enabled: false setup.ilm.enabled: false output.elasticsearch: hosts: ["192.168.70.130:9200"] index: "client-index-%{+YYYY-MM}" bulk_max_size: 1000

启动filebeat,需要制定启动加载的文件

生成的ES日志

{ "_index" : "client-index-2021-09", "_type" : "_doc", "_id" : "s53vKnwB7cmcpxLLkw5s", "_score" : null, "_source" : { "@timestamp" : "2021-09-28T05:45:06.614Z", "ipaddr" : "192.168.70.1", "trace" : "a6582d76a4bf4c4e88f446ce338dfc44", "pid" : "19888", "stack_trace" : """java.lang.RuntimeException: com.netflix.client.ClientException: Load balancer does not have available server for client: eureka-user-service at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:90) at org.springframework.cloud.sleuth.instrument.web.client.feign.TraceLoadBalancerFeignClient.execute(TraceLoadBalancerFeignClient.java:78) at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:119) at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:89) at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100) at com.sun.proxy.$Proxy149.hello(Unknown Source) at com.zhxw.eureka.client.api.UserClient.callHello(UserClient.java:31) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) Caused by: com.netflix.client.ClientException: Load balancer does not have available server for client: eureka-user-service at com.netflix.loadbalancer.LoadBalancerContext.getServerFromLoadBalancer(LoadBalancerContext.java:483) at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:184) at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180) at rx.Observable.unsafeSubscribe(Observable.java:10327) at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94) at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42) at rx.Observable.unsafeSubscribe(Observable.java:10327) at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber$1.call(OperatorRetryWithPredicate.java:127) at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:73) at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:52) at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:79) at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:45) at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:276) at rx.Subscriber.setProducer(Subscriber.java:209) at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:138) at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:129) at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) """, "date" : "2021-09-28 13:45:06.614", "service" : "eureka-user-client", "thread" : "http-nio-8091-exec-10", "class" : "com.zhxw.common.handler.controller.GlobalExceptionHandler", "host" : { "name" : "zhxw-pc" }, "msg" : "异常本身", "level" : "ERROR" }, "sort" : [ 1632807906614 ] }