springcloud+docker+elk搭建日志分析系统

springcloud+docker+elk搭建日志分析系统

elk架构

搭建步骤

1. 创建对应配置文件（注：编排文件与logstash和kibana在同一文件夹下）
   

2. logstash.conf配置

input {
	tcp {
		mode => "server"
		#接收服务发送过来的日志端口
		port => 5000
		codec => json_lines
	}
}
output {
	elasticsearch {
		hosts => ["elasticsearch:9200"]
		action => "index"
		#索引 kibana创建索引时使用
		index => "piclog-%{+YYYY.MM.dd}"
	}
	stdout {
		codec => rubydebug
	}
}

3. kibana.yml配置

## Default Kibana configuration from Kibana base image.
## https://github.com/elastic/kibana/blob/master/src/dev/build/tasks/os_packages/docker_generator/templates/kibana_yml.template.js
#
server.name: kibana.example.org
server.host: "0"
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
xpack.monitoring.ui.container.elasticsearch.enabled: true
#配置kibana界面汉化
i18n.locale: "zh-CN"

4. elk编排docker-compose.yml配置

version: '3.1'

services:
  elasticsearch:
    image: elasticsearch:7.5.1
    environment:
      discovery.type: single-node
    volumes:
      - ./elasticsearch:/usr/share/elasticsearch
    ports:
      - 9200:9200
      - 9300:9300
    networks:
      - elk

  kibana:
    image: kibana:7.5.1
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    ports:
      - 5601:5601
    networks:
      - elk
    depends_on:
      - elasticsearch

  logstash:
    image: logstash:7.5.1
    environment:
      XPACK_MONITORING_ENABLED: "false"
      XPACK_MONITORING_ELASTICSEARCH_HOSTS: http://elasticsearch:9200
    volumes:
      - ./logstash/config:/usr/share/logstash/pipeline
    ports:
      - 5000:5000
    networks:
      - elk
    depends_on:
      - elasticsearch

networks:
  elk:
    driver: bridge

注:因为需要挂载elasticsearch数据则先默认配置启动将/usr/share/elasticsearch拷贝到宿主机
docker pull elasticsearch:7.5.1
docker run -d --name elasticsearch elasticsearch:7.5.1
docker cp elasticsearch:/usr/share/elasticsearch /data/elk
删除默认配置的容器
docker stop elasticsearch
docker rm elasticsearch

5. 进入docker-compose.yml所在包下执行docker-compose up -d 开始部署
   
6. 服务中配置发送日志到logstash
   加入logback依赖

        <dependency>
            <groupId>net.logstash.logback</groupId>
            <artifactId>logstash-logback-encoder</artifactId>
            <version>5.1</version>
        </dependency>

创建logback-spring.xml （这里使用logback.xml也可以但是logback.xml加载比application的配置文件快所以无法读取配置文件的配置，这里建议使用logback-spring.xml方便读取application配置）

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false" scan="true" scanPeriod="1 seconds">
    <springProperty scope="context" name="applicationName" source="spring.application.name" defaultValue=""/>
    <include resource="org/springframework/boot/logging/logback/base.xml" />
    <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>logstash服务地址:5000</destination>
        <!-- encoder必须配置,有多种可选 -->
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder" />
    </appender>

    <root level="info">
        <appender-ref ref="stash" />
    </root>
</configuration>

配置文件application.yml中加载logback-spring.xml配置（如果是logback.xml则不需要）

logging:
  config: classpath:logback-spring.xml

到此全部搭建完毕

使用

1. 启动配置好的服务,加载日志数据到Elasticsearch中 也可以后面启动,但如果不启动kibana会找不到日志索引

2. 打开kibana图形界面 kibana服务器地址:5601
   

3. kiana如果要查看需创建索引模式