日志入库+分库分表

今日面试时问到，当前微服务的流行且导致应用分布在不同的机器上，日志不集中，对日志查看很不方便。虽有ELK等方案处理，但复杂度和投入可不是一般的公司能承受！那有没有其它折衷方案呢？！有，我的其中一个方案就是利用最新流行的开源分库分表框架ShardingSphere存储日志，设计如下：

话不多说，现描述实现步骤如下：

一、在应用端添加配置

1、pom.xml中添加依赖

		<!-- logback-redis-appender -->
		<dependency>
			<groupId>com.cwbase</groupId>
			<artifactId>logback-redis-appender</artifactId>
			<version>1.1.6</version>
		</dependency>

2、在logback.xml中配置Redis队列

    <appender name="LOGSTASH" class="com.cwbase.logback.RedisAppender">
        <source>mySource</source>
        <sourcePath>mySourcePath</sourcePath>
        <type>songjy-app</type>
        <tags>production</tags>
        <host>127.0.0.1</host>
        <port>6379</port>
        <password>123456</password>
        <!-- 此处偷懒，用当前系统用户名(songjy)做队列名 -->
        <key>${user.name}</key>
        <layout class="com.cwbase.logback.JSONEventLayout">
            <source>mySource</source>
            <sourcePath>mySourcePath</sourcePath>
            <type>songjy-app</type>
            <locationInfo>true</locationInfo>
        </layout>
    </appender>

配置完毕后，启动应用，可在Redis队列(songjy)中看到日志，如下：

二、使用spring boot+MyBatis搭建分库分表（ShardingSphere）应用

1、搭建普通spring boot项目并在pom.xml中引入如下依赖

        <!-- sharding-jdbc-spring-boot-starter -->
        <dependency>
            <groupId>org.apache.shardingsphere</groupId>
            <artifactId>sharding-jdbc-spring-boot-starter</artifactId>
            <version>4.1.1</version>
        </dependency>

2、分库分表配置

2.1 物理表tb_log_store结构

 

CREATE TABLE `tb_log_store2020_10_16` (
  `id` BIGINT(20) NOT NULL AUTO_INCREMENT COMMENT '自增主键',
  `log_date` INT(11) NOT NULL COMMENT '日志日期，yyyymmdd',
  `log_time` TIME NOT NULL COMMENT '日志时间，hh:mm:ss',
  `log_source` VARCHAR(32) NOT NULL DEFAULT '' COMMENT '日志来源，即所属应用',
  `log_ip` VARCHAR(16) DEFAULT '' COMMENT '日志机器IP',
  `clazz` VARCHAR(256) NOT NULL DEFAULT '' COMMENT '日志打印所在类',
  `method` VARCHAR(64) NOT NULL DEFAULT '' COMMENT '日志打印所在方法',
  `line` INT(11) NOT NULL DEFAULT '0' COMMENT '日志打印所在行号',
  `message` TEXT NOT NULL COMMENT '日志内容',
  PRIMARY KEY (`id`)
) ENGINE=INNODB DEFAULT CHARSET=utf8mb4 COMMENT='日志记录'

 2.2 对应Java的DO类

package com.songjy.log.sharding.sphere.model;

import lombok.Getter;
import lombok.Setter;

import java.io.Serializable;
import java.util.Date;

/**
 * @author songjy
 */
@Getter
@Setter
public class LogStore implements Serializable {
    private static final long serialVersionUID = -7467751988618132971L;

    /**
     * 主键
     */
    private Long id;
    /**
     * 日志日期，yyyyMMdd，整型
     */
    private Integer logDate;
    /**
     * 日志时间，hh:mm:ss
     */
    private Date logTime;
    /**
     * 日志来源，即所属应用
     */
    private String logSource;
    /**
     * 日志机器IP
     */
    private String logIp;
    /**
     * 日志打印所在类
     */
    private String clazz;
    /**
     * 日志打印所在方法
     */
    private String method;
    /**
     * 日志打印所在行号
     */
    private Integer line;
    /**
     * 日志内容
     */
    private String message;
}

2.3 分库分表

# 指定所有数据库
spring.shardingsphere.datasource.names=ds0,ds1

spring.shardingsphere.datasource.ds0.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.ds0.driver-class-name=com.mysql.cj.jdbc.Driver
#spring.shardingsphere.datasource.ds0.url=jdbc:mysql://localhost:3306/log_store0?useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=UTC
spring.shardingsphere.datasource.ds0.url=jdbc:mysql://localhost:3306/log_store0?useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=Asia/Shanghai
spring.shardingsphere.datasource.ds0.username=root
spring.shardingsphere.datasource.ds0.password=123456

spring.shardingsphere.datasource.ds1.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.ds1.driver-class-name=com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.ds1.url=jdbc:mysql://localhost:3306/log_store1?useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=UTC
spring.shardingsphere.datasource.ds1.username=root
spring.shardingsphere.datasource.ds1.password=123456

# 默认数据库，针对未分库分表处理
spring.shardingsphere.sharding.default-data-source-name=ds0
# 分库策略
spring.shardingsphere.sharding.default-database-strategy.inline.sharding-column=log_date
spring.shardingsphere.sharding.default-database-strategy.inline.algorithm-expression=ds$->{log_date % 2}
# 分表策略:按天分表
spring.shardingsphere.sharding.tables.tb_log_store.actual-data-nodes=ds$->{0..1}.tb_log_store$->{2020..2030}_$->{1..12}_$->{1..31}
spring.shardingsphere.sharding.tables.tb_log_store.table-strategy.complex.sharding-columns=log_date
spring.shardingsphere.sharding.tables.tb_log_store.table-strategy.complex.algorithm-class-name=com.songjy.log.sharding.sphere.config.ShardingTableAlgorithm
# 是否打印SQL
spring.shardingsphere.props.sql.show=true

分库策略：除2取模，因为指定了2个库。

分表策略：自定义，即日志按天分表写入数据库，如下：

package com.songjy.log.sharding.sphere.config;

import org.apache.shardingsphere.api.sharding.complex.ComplexKeysShardingAlgorithm;
import org.apache.shardingsphere.api.sharding.complex.ComplexKeysShardingValue;

import java.time.LocalDate;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.Map;

/**
 * 分表策略
 *
 * @author songjy
 */
public class ShardingTableAlgorithm implements ComplexKeysShardingAlgorithm<Integer> {

    /**
     * 时间格式
     */
    private DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyyMMdd");

    @Override
    public Collection<String> doSharding(Collection<String> tableNames,
                                         ComplexKeysShardingValue<Integer> shardingValues) {

        Map<String, Collection<Integer>> map = shardingValues.getColumnNameAndShardingValuesMap();
        Collection<Integer> orderIdValues = map.get("log_date");
        List<String> shardingSuffix = new ArrayList<>(8);


        for (Integer logDate : orderIdValues) {

            LocalDate localDate = LocalDate.parse(logDate.toString(), formatter);

            String suffix = localDate.getYear() + "_" + localDate.getMonthValue() + "_" + localDate.getDayOfMonth();
            for (String tableName : tableNames) {
                if (tableName.endsWith(suffix)) {
                    shardingSuffix.add(tableName);
                }
            }
        }

        return shardingSuffix;
    }
}

3、消费Redis队列，即日志数据写入MySQL

package com.songjy.log.sharding.sphere.service;

import cn.hutool.core.util.NetUtil;
import com.alibaba.fastjson.JSON;
import com.songjy.log.sharding.sphere.commons.AsyncManager;
import com.songjy.log.sharding.sphere.config.JedisUtil;
import com.songjy.log.sharding.sphere.dto.LogStoreDto;
import com.songjy.log.sharding.sphere.mapper.LogStoreMapper;
import com.songjy.log.sharding.sphere.mapstruct.LogStoreConverter;
import com.songjy.log.sharding.sphere.model.LogStore;
import com.songjy.log.sharding.sphere.utils.DateUtils;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang3.SystemUtils;
import org.springframework.beans.factory.InitializingBean;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.time.LocalDateTime;
import java.util.List;

/**
 * @author songjy
 * @date 2020-10-16
 */
@Service
@Slf4j
public class LogStoreServiceImpl implements ILogStoreService, InitializingBean {
    @Autowired
    private JedisUtil jedisUtil;
    @Autowired
    private LogStoreMapper logStoreMapper;

    @Override
    public void afterPropertiesSet() {
        handleLog();
    }

    /**
     * 处理日志入库
     */
    public void handleLog() {
        String ip = NetUtil.getLocalhostStr();
        AsyncManager.getInstance().execute(() -> {
            try {
                while (System.currentTimeMillis() > Integer.MIN_VALUE) {
                    /*Redis队列中取出日志(JSON)，*/
                    List<String> brpop = jedisUtil.brpop(SystemUtils.USER_NAME);
                    String json = brpop.get(1);
                    LogStoreDto logStoreDto = JSON.parseObject(json, LogStoreDto.class);
                    LocalDateTime localDateTime = DateUtils.toLocalDateTime(logStoreDto.getTimestamp());
                    LogStore logStore = LogStoreConverter.INSTANCE.toDO(logStoreDto);
                    /*此处特殊处理，日期转成整型:yyyyMMdd*/
                    logStore.setLogDate(Integer.parseInt(StringUtils.remove(localDateTime.toLocalDate().toString(), '-')));
                    logStore.setLogTime(logStoreDto.getTimestamp());
                    logStoreMapper.insertSelective(logStore);
                }
            } catch (Exception e) {
                log.error(e.getMessage(), e);
            }
        });
    }
}

4、启动分库分表应用查看MySQL，如下：

完毕！

源码地址：https://github.com/songjygithub/log-sharding-sphere.git

其它说明：因为是按天记录日志，即每天一张表，需要定时任务自动建表，你懂的！