logstash、logback、elk集中处理系统日志

公司使用的微服务，有很多系统日志需要查看，为了解决查看不方便的问题啊，就搭建了ELK，集中处理各个系统产生的日志，把搭建过程及遇到的问题记录下来，便于以后查看！

ELK的搭建使用docker-compose搭建，很方便。

日志的采集，使用logback的适配器，将日志转发到rabbit中，然后转存到el里面！

logback的配置文件

input {
	rabbitmq{
		host=>"10.168.31.224"
		port=> 5672
		exchange=>'ex_logstash'
		user=>"guest"
		password=>"guest"
		queue=>"q_logstash"
		durable=> true
		#codec=>json
		type=> "result"
	  }
}

## Add your filters / logstash plugins configuration here
filter{
    json{
      source=>"message"
    }
}
output {
	elasticsearch {
		hosts => "10.168.31.222:9200"
		index => "position-%{+YYYYMMdd}"
	}
	#stdout { codec => rubydebug }
}

logback的配置文件如下

<configuration>
    <property name="LOG_FILE" value="./logs/centerserver"/>
    <property name="LOG_PATTERN" value="%d{yyyy/MM/dd-HH:mm:ss} %-5level [%thread] %logger - %msg%n"/>
    <property name="CONSOLE_LOG_PATTERN" value="${CONSOLE_LOG_PATTERN:-%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}" />
    <conversionRule conversionWord="clr" converterClass="org.springframework.boot.logging.logback.ColorConverter" />
    <conversionRule conversionWord="wex" converterClass="org.springframework.boot.logging.logback.WhitespaceThrowableProxyConverter" />
    <conversionRule conversionWord="wEx" converterClass="org.springframework.boot.logging.logback.ExtendedWhitespaceThrowableProxyConverter" />
    <conversionRule conversionWord="mx" converterClass="com.mti.configuration.CustomMessageConverter" />
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <!--<pattern>${LOG_PATTERN}</pattern>-->
            <pattern>
                ${CONSOLE_LOG_PATTERN}
            </pattern>
        </encoder>
    </appender>
    <appender name="TIME_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOG_FILE}.log</file>
        <encoder>
            <pattern>${LOG_PATTERN}</pattern>
        </encoder>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${LOG_FILE}.%d{yyyy-MM-dd}.%i.log</fileNamePattern>
            <maxHistory>10</maxHistory>
            <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>100MB</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
        </rollingPolicy>
    </appender>
    <appender name="LOGSTASH"
              class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>10.168.31.222:5000</destination>
        <!-- encoder必须配置,有多种可选 -->
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder">
            <customFields>{"appname":"centerserver"}</customFields>
        </encoder>
        <!-- <destination>destination1.domain.com:4560</destination>
            <destination>destination2.domain.com:4560</destination>
            <destination>destination3.domain.com:4560</destination> -->
        <connectionStrategy>
            <roundRobin>
                <connectionTTL>5 minutes</connectionTTL>
            </roundRobin>
        </connectionStrategy>
    </appender>

    <appender name="AMQP"
              class="org.springframework.amqp.rabbit.logback.AmqpAppender">
        <layout>
            <pattern>
                {"time": "%date{ISO8601}","thread": "%thread","level": "%level","class": "%logger{60}","detail": "%mx"}%n
            </pattern>
        </layout>
        <host>10.168.31.224</host>
        <port>5672</port>
        <username>guest</username>
        <password>guest</password>
        <applicationId>ms</applicationId>
        <routingKeyPattern>lgstash</routingKeyPattern>
        <declareExchange>true</declareExchange>
        <exchangeType>direct</exchangeType>
        <exchangeName>ex_logstash</exchangeName>
        <generateId>true</generateId>
        <charset>UTF-8</charset>
        <durable>true</durable>
        <deliveryMode>PERSISTENT</deliveryMode>
    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="TIME_FILE"/>
       <!-- <appender-ref ref="LOGSTASH" />-->
        <appender-ref ref="AMQP" />
    </root>
</configuration>

在配置文件里面自定义自己的转换器CustomMessageConverter，将log以json的格式输出。

import ch.qos.logback.classic.pattern.MessageConverter;
import ch.qos.logback.classic.spi.ILoggingEvent;

/**
 * @ClassName CustomMessageConverter
 * @Description CustomMessageConverter
 * @Author zhaoyj
 * @Date 2019/4/30 11:01
 */
public class CustomMessageConverter extends MessageConverter {

    @Override
    public String convert(ILoggingEvent event) {
        String log =  event.getFormattedMessage();
        log = log.replaceAll("\n","").replaceAll("\"","'");
        return log;
    }
}

替换掉原来日志中的换行符和双引号，不替换的话logstash解析会报错，解析不了！

还有，在项目里面需要引入两个包

 <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-amqp</artifactId>
            <version>2.1.2.RELEASE</version>
        </dependency>

        <dependency>
            <groupId>net.logstash.logback</groupId>
            <artifactId>logstash-logback-encoder</artifactId>
            <version>5.3</version>
        </dependency>