Id Generator

Id Generator

一、介绍

在复杂分布式系统中，往往需要对大量的数据和消息进行唯一标识。全局唯一ID需要具备一下特点：

• 全局唯一性
• 趋势递增：MySQL InnoDB引擎中使用的是聚集索引，由于B-tree的数据结构来存储索引数据，应该尽量使用有序的主键保证写入性能
• 单调递增
• 信息安全：如果ID是连续的，利于恶意用户扒取，直接按照顺序下载指定URL即可，如果ID是订单号，竞对可以知道一天的单量。所以需要ID无规则、不规则。
• 含时间戳

一个ID生成系统大致需要：

1. 低延时
2. 高可用
3. 高QPS

二、常见方式

1.UUID

UUID包含32个16进制数字，以-分为五段，形式为8-4-4-4-12的36个字符，

fd4ce1e6-a080-4aa7-9457-78816b8dbc3b

优点：

• 性能非常高：本地生成，没有网络消耗

缺点：

• 不易于存储：UUID太长，16字节128位，通常以36长度的字符串表示
• 信息不安全：基于MAC地址生成UUID的算法可能会造成MAC地址泄露
• 不适合作为数据库主键：主键要尽量越短越好；由于UUID无序性可能会引起数据位置频繁变动，严重影响性能

Mysql官网：https://dev.mysql.com/doc/refman/5.7/en/innodb-index-types.html

2.数据库自增

创建数据库：

CREATE TABLE "id_table" (
  "id" bigint NOT NULL AUTO_INCREMENT,
  "stub" varchar(255) NOT NULL,
  PRIMARY KEY ("id"),
  UNIQUE KEY "idx_stub" ("stub")
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=utf8

replace into：

首先尝试插入数据到表中，如果表中已经有此行数据则先删除此行数据，然后插入新的数据

否则，直接插入新数据

begin;
REPLACE INTO id_table(stub) VALUES ('a');
SELECT LAST_INSERT_ID();
commit;

优点：

• 简单
• ID号单调自增

缺点：

• 当DB异常时整个系统不可用
• ID性能瓶颈限制在单台MySQL的读写性能

3.Snowflake

1bit-不用表示正负，ID永远为正数

41-bit的时间可以表示（1L<<41）/ (1000L360024*365)=69年的时间

10-bit机器可以分别表示1024台机器。还可以将10-bit分5-bit给IDC，分5-bit给工作机器

12个自增序列号可以表示2^12个ID，理论上snowflake方案的QPS约为409.6w/s

@Slf4j
@Component
public class IdGeneratorSnowFlake {

    private long workerId = 0;

    private final long datacenterId = 1;

    private final Snowflake snowflake = IdUtil.getSnowflake(workerId, datacenterId);

    @PostConstruct
    public void init(){
        try {
            workerId = NetUtil.ipv4ToLong(NetUtil.getLocalhostStr());
            log.info("workerId: " + workerId);
        } catch (Exception e) {
            e.printStackTrace();
            log.warn("未获取到workerId, exception: " + e);
            workerId = NetUtil.getLocalhostStr().hashCode();
        }
    }

    public synchronized long getSnowflakeId(){
        return snowflake.nextId();
    }

    public synchronized long getSnowflakeId(long workerId, long datacenterId){
        Snowflake snowflake = IdUtil.getSnowflake(workerId, datacenterId);
        return snowflake.nextId();
    }
}

@SpringBootTest
public class SnowflakeTests {

    @Autowired
    private IdGeneratorSnowFlake idGeneratorSnowFlake;

    @Test
    void test(){

        ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(5,
                5,
                0,
                TimeUnit.MILLISECONDS,
                new LinkedBlockingDeque<>(),
                Executors.defaultThreadFactory(),
                new ThreadPoolExecutor.CallerRunsPolicy());

        for (int i = 1; i < 20; i++) {
            threadPoolExecutor.submit(() -> {
                System.out.println(Thread.currentThread().getName() + "获取snowflakeId: " + idGeneratorSnowFlake.getSnowflakeId());
            });
        }

        threadPoolExecutor.shutdown();
    }
}

/*
pool-1-thread-1获取snowflakeId: 1577204575893585920
pool-1-thread-2获取snowflakeId: 1577204575897780224
pool-1-thread-3获取snowflakeId: 1577204575897780225
pool-1-thread-3获取snowflakeId: 1577204575897780226
pool-1-thread-4获取snowflakeId: 1577204575897780227
pool-1-thread-5获取snowflakeId: 1577204575897780228
pool-1-thread-2获取snowflakeId: 1577204575897780229
pool-1-thread-4获取snowflakeId: 1577204575897780232
pool-1-thread-3获取snowflakeId: 1577204575897780231
pool-1-thread-1获取snowflakeId: 1577204575897780230
pool-1-thread-3获取snowflakeId: 1577204575897780236
pool-1-thread-4获取snowflakeId: 1577204575897780235
pool-1-thread-2获取snowflakeId: 1577204575897780234
pool-1-thread-5获取snowflakeId: 1577204575897780233
pool-1-thread-2获取snowflakeId: 1577204575897780240
pool-1-thread-4获取snowflakeId: 1577204575897780239
pool-1-thread-3获取snowflakeId: 1577204575897780238
pool-1-thread-1获取snowflakeId: 1577204575897780237
pool-1-thread-5获取snowflakeId: 1577204575897780241
*/

优点：

• 毫秒数在高位，自增序列在低位，整个ID都是趋势递增的
• 不依赖数据库等第三方系统，以服务的方式部署，稳定性更高，生成ID的性能也是非常高的
• 可以自定义分配bit位，非常灵活

缺点：

• 强依赖机器时钟，如果机器上时钟回拨，会导致发号重复或者服务会处于不可用状态

4.UidGenerator

github：https://github.com/baidu/uid-generator/blob/master/README.zh_cn.md

UidGenerator是Java实现的, 基于Snowflake算法的唯一ID生成器，UidGenerator支持自定义workerId位数和初

始化策略， 从而适用于docker等虚拟化环境下实例自动重启、漂移等场景，最终单机QPS可达600万

采用RingBuffer来缓存已生成的UID, 并行化UID的生产和消费。RingBuffer是一个环形数组，数组每个元素成

为一个slot。RingBuffer容量，默认为sequence的值，可通过boostPower配置进行扩容，以提高读写吞吐量

Tail指针、Cursor指针用于环形数组上读写slot：

• Tail指针
  表示Producer生产的序号(此序号从0开始，持续递增)。Tail不能超过Cursor，即生产者不能覆盖未消费的slot。当Tail已赶上curosr，此时可通过rejectedPutBufferHandler指定局拒绝生产策略
• Cursor指针
  表示Consumer消费到的序号(序号序列与Producer序列相同)。Cursor不能超过Tail，即不能消费未生产的slot。当Cursor已赶上tail，此时可通过rejectedTakeBufferHandler指定拒绝消费策略

生成64 bits的唯一ID：

• sign(1bit)：生成的UID为正数
• delta seconds (28 bits)：当前时间，相对于"2016-05-20"的增量值，单位：秒，最多可支持约8.7年
• worker id (22 bits)：机器id，最多可支持约420w次机器启动。内置实现为在启动时由数据库分配，默认分配策略为用后即弃，后续可提供复用策略
• sequence (13 bits)： 每秒下的并发序列，13 bits可支持每秒8192(2 ^ 13)并发

整合springboot：

创建库表：

DROP TABLE IF EXISTS WORKER_NODE;
CREATE TABLE WORKER_NODE
(
    ID BIGINT NOT NULL AUTO_INCREMENT COMMENT 'auto increment id',
    HOST_NAME VARCHAR(64) NOT NULL COMMENT 'host name',
    PORT VARCHAR(64) NOT NULL COMMENT 'port',
    TYPE INT NOT NULL COMMENT 'node type: ACTUAL or CONTAINER',
    LAUNCH_DATE DATE NOT NULL COMMENT 'launch date',
    MODIFIED DATETIME NOT NULL COMMENT 'modified time',
    CREATED DATETIME NOT NULL COMMENT 'created time',
    PRIMARY KEY(ID)
)
COMMENT='DB WorkerID Assigner for UID Generator',ENGINE = INNODB;

导入依赖：

<dependency>
    <groupId>com.xfvape.uid</groupId>
    <artifactId>uid-generator</artifactId>
    <version>0.0.4-RELEASE</version>
    <!--排除mybatis依赖,避免冲突-->
    <exclusions>
         <exclusion>
             <groupId>org.slf4j</groupId>
             <artifactId>log4j-over-slf4j</artifactId>
         </exclusion>
         <exclusion>
             <groupId>ch.qos.logback</groupId>
             <artifactId>logback-classic</artifactId>
         </exclusion>
         <exclusion>
             <groupId>org.slf4j</groupId>
             <artifactId>slf4j-api</artifactId>
         </exclusion>
          <exclusion>
             <groupId>org.mybatis</groupId>
             <artifactId>mybatis-spring</artifactId>
         </exclusion>
         <exclusion>
             <groupId>org.mybatis</groupId>
             <artifactId>mybatis</artifactId>
         </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>com.baomidou</groupId>
    <artifactId>mybatis-plus-boot-starter</artifactId>
    <version>3.4.3.4</version>
</dependency>
<dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <optional>true</optional>
</dependency>
<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

分析源码：

WorkerNodeMapper接口：

@Mapper
public interface WorkerNodeMapper {

    WorkerNodeEntity getWorkerNodeByHostPort(@Param("host") String var1, @Param("port") String var2);

    void addWorkerNode(WorkerNodeEntity var1);
}

workerNodeMapper.xml：

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.qingsongxyz.mybatisplus.mapper.WorkerNodeMapper">
   <resultMap id="workerNodeRes"
            type="com.xfvape.uid.worker.entity.WorkerNodeEntity">
      <id column="ID" jdbcType="BIGINT" property="id" />
      <result column="HOST_NAME" jdbcType="VARCHAR" property="hostName" />
      <result column="PORT" jdbcType="VARCHAR" property="port" />
      <result column="TYPE" jdbcType="INTEGER" property="type" />
      <result column="LAUNCH_DATE" jdbcType="DATE" property="launchDate" />
      <result column="MODIFIED" jdbcType="DATE" property="modified" />
      <result column="CREATED" jdbcType="DATE" property="created" />
   </resultMap>

   <insert id="addWorkerNode" useGeneratedKeys="true" keyProperty="id"
      parameterType="com.xfvape.uid.worker.entity.WorkerNodeEntity">
      INSERT INTO WORKER_NODE
      (HOST_NAME,
      PORT,
      TYPE,
      LAUNCH_DATE,
      MODIFIED,
      CREATED)
      VALUES (
      #{hostName},
      #{port},
      #{type},
      #{launchDate},
      NOW(),
      NOW())
   </insert>

   <select id="getWorkerNodeByHostPort" resultMap="workerNodeRes">
      SELECT
      ID,
      HOST_NAME,
      PORT,
      TYPE,
      LAUNCH_DATE,
      MODIFIED,
      CREATED
      FROM
      WORKER_NODE
      WHERE
      HOST_NAME = #{host} AND PORT = #{port}
   </select>
</mapper>

模仿DisposableWorkerIdAssigner编写自己的WorkerIdAssigner实现类：

@Data
@NoArgsConstructor
@Component
public class MyDisposableWorkerIdAssigner implements WorkerIdAssigner {

    private static final Logger LOGGER = LoggerFactory.getLogger(MyDisposableWorkerIdAssigner.class);

    @Autowired
    private WorkerNodeMapper workerNodeMapper;

    @Override
    public long assignWorkerId() {
        WorkerNodeEntity workerNodeEntity = this.buildWorkerNode();
        this.workerNodeMapper.addWorkerNode(workerNodeEntity);
        LOGGER.info("Add worker node:" + workerNodeEntity);
        return workerNodeEntity.getId();
    }

    private WorkerNodeEntity buildWorkerNode() {
        WorkerNodeEntity workerNodeEntity = new WorkerNodeEntity();
        if (DockerUtils.isDocker()) {
            workerNodeEntity.setType(WorkerNodeType.CONTAINER.value());
            workerNodeEntity.setHostName(DockerUtils.getDockerHost());
            workerNodeEntity.setPort(DockerUtils.getDockerPort());
        } else {
            workerNodeEntity.setType(WorkerNodeType.ACTUAL.value());
            workerNodeEntity.setHostName(NetUtils.getLocalAddress());
            workerNodeEntity.setPort(System.currentTimeMillis() + "-" + RandomUtils.nextInt(100000));
        }

        return workerNodeEntity;
    }
}

application.yaml：

uid-generator:
  time-bits: 29
  worker-bits: 21
  seq-bits: 13
  epoch-str: 2016-09-20
  boost-power: 3
  padding-factor: 50
  schedule-interval: 60

UidGenerator配置类：

@Data
@Configuration
//绑定配置文件,前缀为uid-generator
@ConfigurationProperties(prefix = "uid-generator")
public class UidGeneratorConfig {

    private int timeBits;

    private int workerBits;

    private int seqBits;

    private String epochStr;

    private int boostPower;

    private int paddingFactor;

    private int scheduleInterval;

    @Resource
    private WorkerNodeMapper WorkerNodeMapper;

    @Bean
    public MyDisposableWorkerIdAssigner disposableWorkerIdAssigner(){
        MyDisposableWorkerIdAssigner disposableWorkerIdAssigner = new MyDisposableWorkerIdAssigner();
        disposableWorkerIdAssigner.setWorkerNodeMapper(WorkerNodeMapper);
        return disposableWorkerIdAssigner;
    }

    @Bean
    public CachedUidGenerator cachedUidGenerator() {
        CachedUidGenerator cachedUidGenerator = new CachedUidGenerator();
        cachedUidGenerator.setWorkerIdAssigner(disposableWorkerIdAssigner());

        cachedUidGenerator.setTimeBits(timeBits);
        cachedUidGenerator.setWorkerBits(workerBits);
        cachedUidGenerator.setSeqBits(seqBits);
        cachedUidGenerator.setEpochStr(epochStr);
        cachedUidGenerator.setBoostPower(boostPower);
        cachedUidGenerator.setPaddingFactor(paddingFactor);
        cachedUidGenerator.setScheduleInterval(scheduleInterval);

        return cachedUidGenerator;
    }
}

WorkerNodeService：

public interface WorkerNodeService {

    long genUid();
}

WorkerNodeServiceImpl：

@Service
public class WorkerNodeServiceImpl implements WorkerNodeService {

    @Autowired
    private UidGenerator uidGenerator;

    @Override
    public long genUid() {
        return uidGenerator.getUID();
    }
}

WorkerNodeController：

@RestController
public class WorkerNodeController {

    @Autowired
    private WorkerNodeService workerNodeService;

    @GetMapping("/baidu/uid")
    public String baiduUid(){
        return String.valueOf(workerNodeService.genUid());
    }
}

测试：