Spring Boot 集成百度 Uidgenerator分布式ID生成器

UidGenerator是Java实现的, 基于Snowflake算法的唯一ID生成器。UidGenerator以组件形式工作在应用项目中, 支持自定义workerId位数和初始化策略, 从而适用于docker等虚拟化环境下实例自动重启、漂移等场景。 在实现上, UidGenerator通过借用未来时间来解决sequence天然存在的并发限制; 采用RingBuffer来缓存已生成的UID, 并行化UID的生产和消费, 同时对CacheLine补齐，避免了由RingBuffer带来的硬件级「伪共享」问题. 最终单机QPS可达600万。

官方地址：https://github.com/baidu/uid-generator

uid-generator提供了两种生成器: DefaultUidGenerator、CachedUidGenerator。如对UID生成性能有要求, 请使用CachedUidGenerator

CachedUidGenerator

RingBuffer环形数组，数组每个元素成为一个slot。RingBuffer容量，默认为Snowflake算法中sequence最大值，且为2^N。可通过boostPower配置进行扩容，以提高RingBuffer 读写吞吐量。

Tail指针、Cursor指针用于环形数组上读写slot：

• Tail指针
  表示Producer生产的最大序号(此序号从0开始，持续递增)。Tail不能超过Cursor，即生产者不能覆盖未消费的slot。当Tail已赶上curosr，此时可通过rejectedPutBufferHandler指定PutRejectPolicy
• Cursor指针
  表示Consumer消费到的最小序号(序号序列与Producer序列相同)。Cursor不能超过Tail，即不能消费未生产的slot。当Cursor已赶上tail，此时可通过rejectedTakeBufferHandler指定TakeRejectPolicy

CachedUidGenerator采用了双RingBuffer，Uid-RingBuffer用于存储Uid、Flag-RingBuffer用于存储Uid状态(是否可填充、是否可消费)

由于数组元素在内存中是连续分配的，可最大程度利用CPU cache以提升性能。但同时会带来「伪共享」FalseSharing问题，为此在Tail、Cursor指针、Flag-RingBuffer中采用了CacheLine 补齐方式。

RingBuffer填充时机

• 初始化预填充
  RingBuffer初始化时，预先填充满整个RingBuffer.
• 即时填充
  Take消费时，即时检查剩余可用slot量(tail - cursor)，如小于设定阈值，则补全空闲slots。阈值可通过paddingFactor来进行配置，请参考Quick Start中CachedUidGenerator配置
• 周期填充
  通过Schedule线程，定时补全空闲slots。可通过scheduleInterval配置，以应用定时填充功能，并指定Schedule时间间隔

1、导入依赖

<!-- https://mvnrepository.com/artifact/com.xfvape.uid/uid-generator -->
<dependency>
    <groupId>com.xfvape.uid</groupId>
    <artifactId>uid-generator</artifactId>
    <version>0.0.4-RELEASE</version>
</dependency>

2、创建表WORKER_NODE

运行sql脚本以导入表WORKER_NODE, 脚本如下:

DROP DATABASE IF EXISTS `xxxx`;
CREATE DATABASE `xxxx` ;
use `xxxx`;
DROP TABLE IF EXISTS WORKER_NODE;
CREATE TABLE WORKER_NODE
(
ID BIGINT NOT NULL AUTO_INCREMENT COMMENT 'auto increment id',
HOST_NAME VARCHAR(64) NOT NULL COMMENT 'host name',
PORT VARCHAR(64) NOT NULL COMMENT 'port',
TYPE INT NOT NULL COMMENT 'node type: ACTUAL or CONTAINER',
LAUNCH_DATE DATE NOT NULL COMMENT 'launch date',
MODIFIED TIMESTAMP NOT NULL COMMENT 'modified time',
CREATED TIMESTAMP NOT NULL COMMENT 'created time',
PRIMARY KEY(ID)
)
 COMMENT='DB WorkerID Assigner for UID Generator',ENGINE = INNODB;

3、maven依赖

 <dependencies>
        <dependency>
            <groupId>com.xiaobear.distributedid</groupId>
            <artifactId>distributedid-core</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-jdbc</artifactId>
        </dependency>

        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>org.mybatis.spring.boot</groupId>
            <artifactId>mybatis-spring-boot-starter</artifactId>
            <version>2.1.0</version>
        </dependency>
        <!--必须放在最后-->
        <!-- https://mvnrepository.com/artifact/com.xfvape.uid/uid-generator -->
        <dependency>
            <groupId>com.xfvape.uid</groupId>
            <artifactId>uid-generator</artifactId>
            <version>0.0.4-RELEASE</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>log4j-over-slf4j</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>ch.qos.logback</groupId>
                    <artifactId>logback-classic</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-api</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.mybatis</groupId>
                    <artifactId>mybatis-spring</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.mybatis</groupId>
                    <artifactId>mybatis</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>

4、配置文件

spring:
  datasource:
    type: com.zaxxer.hikari.HikariDataSource
    url: jdbc:mysql://127.0.0.1:3306/distributedid?serverTimezone=Asia/Shanghai&characterEncoding=utf8&useUnicode=true&useSSL=false&autoReconnect=true
    username: root
    password: 密码
    driver-class-name: com.mysql.cj.jdbc.Driver
server:
  port: 8088

mybatis:
  type-aliases-package: com.xiaobear.distributedid.core.domain
  mapper-locations: classpath*:mapper/*.xml

5、主启动类

@SpringBootApplication
@MapperScan("com.xiaobear.distributedid.mapper")
public class UidgeneratorApplication {

    public static void main(String[] args) {
        SpringApplication.run(UidgeneratorApplication.class);
        System.out.println("百度生成ID 启动成功");
    }
}

6、业务类

1、实体类

public class WorkerNode {

    private Long id;

    private String hostName;

    private String port;

    private Integer type;

    private Date launchDate;

    private Date modified;

    private Date created;

    public Long getId() {
        return id;
    }

    public void setId(Long id) {
        this.id = id;
    }

    public String getHostName() {
        return hostName;
    }

    public void setHostName(String hostName) {
        this.hostName = hostName;
    }

    public String getPort() {
        return port;
    }

    public void setPort(String port) {
        this.port = port;
    }

    public Integer getType() {
        return type;
    }

    public void setType(Integer type) {
        this.type = type;
    }

    public Date getLaunchDate() {
        return launchDate;
    }

    public void setLaunchDate(Date launchDate) {
        this.launchDate = launchDate;
    }

    public Date getModified() {
        return modified;
    }

    public void setModified(Date modified) {
        this.modified = modified;
    }

    public Date getCreated() {
        return created;
    }

    public void setCreated(Date created) {
        this.created = created;
    }

    @Override
    public String toString() {
        return "WorkerNode{" +
                "id=" + id +
                ", hostName='" + hostName + '\'' +
                ", port='" + port + '\'' +
                ", type=" + type +
                ", launchDate=" + launchDate +
                ", modified=" + modified +
                ", created=" + created +
                '}';
    }
}

2、数据层接口

@Mapper
public interface WorkerNodeMapper {

    /**
     * 添加对象
     * @param workerNodeEntity
     * @return
     */
    int addWorkerNode(WorkerNode workerNodeEntity);


    /**
     * 通过host port 获取ID
     * @param host
     * @param port
     * @return
     */
    WorkerNode getWorkerNodeByHostPort(@Param("host") String host, @Param("port") String port);
}

mapper.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.xiaobear.distributedid.mapper.WorkerNodeMapper">
    <resultMap id="workerNodeRes" type="com.xiaobear.distributedid.core.domain.WorkerNode">
        <id column="ID" jdbcType="BIGINT" property="id"/>
        <result column="HOST_NAME" jdbcType="VARCHAR" property="hostName"/>
        <result column="PORT" jdbcType="VARCHAR" property="port"/>
        <result column="TYPE" jdbcType="INTEGER" property="type"/>
        <result column="LAUNCH_DATE" jdbcType="DATE" property="launchDate"/>
        <result column="MODIFIED" jdbcType="TIMESTAMP" property="modified"/>
        <result column="CREATED" jdbcType="TIMESTAMP" property="created"/>
    </resultMap>

    <insert id="addWorkerNode" useGeneratedKeys="true" keyProperty="id"
            parameterType="com.xiaobear.distributedid.core.domain.WorkerNode">
        INSERT INTO WORKER_NODE
        (HOST_NAME,
         PORT,
         TYPE,
         LAUNCH_DATE,
         MODIFIED,
         CREATED)
        VALUES (
                   #{hostName},
                   #{port},
                   #{type},
                   #{launchDate},
                   NOW(),
                   NOW())
    </insert>

    <select id="getWorkerNodeByHostPort" resultMap="workerNodeRes">
        SELECT
            ID,
            HOST_NAME,
            PORT,
            TYPE,
            LAUNCH_DATE,
            MODIFIED,
            CREATED
        FROM
            WORKER_NODE
        WHERE
            HOST_NAME = #{host} AND PORT = #{port}
    </select>
</mapper>

3、实现层接口

public interface IWorkerNodeService {

    /**
     * 获取ID
     * @return
     */
    public long genUid();
}

4、实现层实现类

@Service
public class WorkerNodeServiceImpl implements IWorkerNodeService {
    @Resource
    private UidGenerator uidGenerator;

    /**
     * 获取百度生成ID
     * @return
     */
    @Override
    public long genUid() {
        return uidGenerator.getUID();
    }
}

5、控制层接口

@RestController
public class WorkerNodeController {

    @Resource
    private IWorkerNodeService workerNodeService;

    /**
     *使用百度 Uidgenerator获取ID
     * @return
     */
    @GetMapping("/Uidgenerator")
    public RestResponse getIdByBaiDuUid(){
        long id = workerNodeService.genUid();
        return RestResponse.success(id);
    }
}

7、测试

http://localhost:8088/Uidgenerator

{"code":200,"msg":"成功","result":7067247636209745920}

8、uid-generator核心对象装配为spring的bean

1、重写WorkerIdAssigner接口

public class DisposableWorkerIdAssigner implements WorkerIdAssigner{
    @Resource
    private WorkerNodeMapper workerNodeMapper;

    @Override
    @Transactional(rollbackFor = Exception.class)
    public long assignWorkerId() {
        WorkerNode workerNode = buildWorkerNode();

        workerNodeMapper.addWorkerNode(workerNode);

        return workerNode.getId();
    }

    private WorkerNode buildWorkerNode() {
        WorkerNode workNode = new WorkerNode();
        if (DockerUtils.isDocker()) {
            workNode.setType(WorkerNodeType.CONTAINER.value());
            workNode.setHostName(DockerUtils.getDockerHost());
            workNode.setPort(DockerUtils.getDockerPort());
        } else {
            workNode.setType(WorkerNodeType.ACTUAL.value());
            workNode.setHostName(NetUtils.getLocalAddress());
            workNode.setPort(System.currentTimeMillis() + "-" + RandomUtils.nextInt(100000));
        }
        workNode.setLaunchDate(new Date());

        return workNode;
    }
}

2、自动转配bean

@Configuration
public class WorkerNodeConfig {

    @Bean(name = "disposableWorkerIdAssigner")
    public DisposableWorkerIdAssigner disposableWorkerIdAssigner(){
        return new DisposableWorkerIdAssigner();
    }

    @Bean(name = "cachedUidGenerator")
    public UidGenerator uidGenerator(DisposableWorkerIdAssigner disposableWorkerIdAssigner){
        CachedUidGenerator cachedUidGenerator = new CachedUidGenerator();
        cachedUidGenerator.setWorkerIdAssigner(disposableWorkerIdAssigner);
        return cachedUidGenerator;
    }
}

3、使用，在实现类指定生成ID的bean

@Service
public class WorkerNodeServiceImpl implements IWorkerNodeService {
    @Resource(name = "cachedUidGenerator")
    private UidGenerator uidGenerator;

    /**
     * 获取百度生成ID
     * @return
     */
    @Override
    public long genUid() {
        return uidGenerator.getUID();
    }
}