前言
企业开发中,一个应用会有多个测试环境,于是会有一个专门的服务器做日志收集,那就需要保存日志和应用隔离,这里就牵涉到异步存写日志的问题,异步消息队列选取kafka,高性能,日志消息消费我们使用logstash。这里也可以都使用elk(土豪请绕过)。
Kafka的搭建
习惯使用docker-compose启动,脚本如下:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper ## 镜像
ports:
- "2181:2181" ## 对外暴露的端口号
kafka:
image: wurstmeister/kafka ## 镜像
volumes:
- /etc/localtime:/etc/localtime ## 挂载位置(kafka镜像和宿主机器之间时间保持一直)
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: ## 修改:宿主机IP
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 ## kafka运行是基于zookeeper的
kafka-manager:
image: sheepkiller/kafka-manager ## 镜像:开源的web管理kafka集群的界面
environment:
ZK_HOSTS: ## 修改:宿主机IP
ports:
- "9000:9000"
docker-compose up -d 后台方式启动
docker-compose stop 关闭
docker-compose ps 查看有哪些应用
docker-compose build 构建镜像
Logstash的搭建
同样也是用docker-compose启动,脚本如下:
docker-compose.yml->
version: "3"
services:
logstash-1:
image: logstash:7.0.0
container_name: logstash
volumes:
- ${LOGSTASH_CONFIG_DIR}/logstash.conf:/usr/share/logstash/pipeline/logstash.conf:rw
- ${LOGSTASH_CONFIG_DIR}/logstash.yml:/usr/share/logstash/config/logstash.yml:rw
network_mode: "host"
logstash.config
input {
kafka {
bootstrap_servers => "IP:9092"
topics => ["abklog_topic"]
group_id => "abklog_topic"
}
}
output {
file {
file_mode => 0777
dir_mode => 0777
path => "/path/to/%{+yyyy-MM-dd-HH}/%{host}.log"
}
stdout {
codec => rubydebug
}
}
logstash.yml
# set now host ip to http.host
http.host: 192.168.56.121
# set the es-tribe-node host. let logstash monitor the es.
#xpack.monitoring.elasticsearch.hosts:
#- http://10.2.114.110:9204
# enable or disable the logstash monitoring the es.
#xpack.monitoring.enabled: true
Java日志配置
在你改好的springboot工程中整好,加入这个依赖。
<!-- https://mvnrepository.com/artifact/com.github.danielwegener/logback-kafka-appender -->
<dependency>
<groupId>com.github.danielwegener</groupId>
<artifactId>logback-kafka-appender</artifactId>
<version>0.2.0-RC2</version>
</dependency>
logback.xml配置
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<!-- springProfile用于指定当前激活的环境,如果spring.profile.active的值是哪个,就会激活对应节点下的配置 -->
<springProfile name="dev">
<!-- configuration to be enabled when the "staging" profile is active -->
<springProperty scope="context" name="module" source="spring.application.name"
defaultValue="undefinded"/>
<!-- 该节点会读取Environment中配置的值,在这里我们读取application.yml中的值 -->
<springProperty scope="context" name="bootstrapServers" source="spring.kafka.bootstrap-servers"
defaultValue="IP:9092"/>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are assigned the type
ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder>
<pattern>%boldYellow(${module}) | %d | %highlight(%-5level)| %cyan(%logger{15}) - %msg %n</pattern>
</encoder>
</appender>
<!-- kafka的appender配置 -->
<appender name="kafka" class="com.github.danielwegener.logback.kafka.KafkaAppender">
<encoder>
<pattern>${module} | %d | %-5level| %logger{15} - %msg</pattern>
</encoder>
<topic>abklog_topic</topic>
<keyingStrategy class="com.github.danielwegener.logback.kafka.keying.NoKeyKeyingStrategy"/>
<deliveryStrategy class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy"/>
<!-- Optional parameter to use a fixed partition -->
<!-- <partition>0</partition> -->
<!-- Optional parameter to include log timestamps into the kafka message -->
<!-- <appendTimestamp>true</appendTimestamp> -->
<!-- each <producerConfig> translates to regular kafka-client config (format: key=value) -->
<!-- producer configs are documented here: https://kafka.apache.org/documentation.html#newproducerconfigs -->
<!-- bootstrap.servers is the only mandatory producerConfig -->
<producerConfig>bootstrap.servers=${bootstrapServers}</producerConfig>
<!-- 如果kafka不可用则输出到控制台 -->
<appender-ref ref="STDOUT"/>
</appender>
<!-- 指定项目中的logger -->
<!--<logger name="org.springframework.test" level="INFO" >
<appender-ref ref="kafka" />
</logger>-->
<logger name="com.fast.cloud.fastelk.controller" level="INFO" >
<appender-ref ref="kafka" />
</logger>
<root level="info">
<appender-ref ref="STDOUT" />
</root>
</springProfile>
</configuration>
记得将kafka的ip换成你真实的ip