分布式唯一ID生成器UidGenerator
一、UidGenerator简介
UidGenerator是百度开源的Java语言实现,基于Snowflake算法的唯一ID生成器。
它是分布式的,并克服了雪花算法的并发限制。
单个实例的QPS能超过6000000。
需要的环境:JDK8+,MySQL(用于分配WorkerId)
二、雪花算法snowflake
由下图可知,雪花算法的几个核心组成部分:
1为sign标识位;
41位时间戳;
10位workId(数据中心+工作机器,可以其他组成方式);
12位自增序列;
但是百度对这些组成部分稍微调整了一下:
由上图可知,UidGenerator的时间部分只有28位,这就意味着UidGenerator默认只能承受8.5年(2^28-1/86400/365)
也可以根据你业务的需求,UidGenerator可以适当调整delta seconds、worker node id和sequence占用位数。
三、如何使用UidGenerator 生成全局唯一ID
开发环境: springboot,mybatis
由于UidGenerator没有上传jar包到maven仓库上,需要从GitHub上上面下载源码,自己打成jar包安装到maven本地库中。
官方文档
UidGenerator源码地址
UidGenerator官方中文文档
1.新建springboot项目
如果不会springboot,请参考springboot教程。
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>demo9-springboot</artifactId>
<version>1.0-SNAPSHOT</version>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.3.RELEASE</version>
<relativePath />
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>
2.添加数据库和mybatis依赖
<dependency>
<groupId>org.mybatis.spring.boot</groupId>
<artifactId>mybatis-spring-boot-starter</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.20</version>
</dependency>
3.最后添加uid-generator 依赖
<dependency>
<groupId>com.baidu.fsg</groupId>
<artifactId>uid-generator</artifactId>
<version>1.0.0-SNAPSHOT</version>
</dependency>
4. 新建表结构
在数据库中创建uid-generator 依赖的表结构
sql语句如下:
CREATE TABLE WORKER_NODE
(
ID BIGINT NOT NULL AUTO_INCREMENT COMMENT 'auto increment id',
HOST_NAME VARCHAR(64) NOT NULL COMMENT 'host name',
PORT VARCHAR(64) NOT NULL COMMENT 'port',
TYPE INT NOT NULL COMMENT 'node type: ACTUAL or CONTAINER',
LAUNCH_DATE DATE NOT NULL COMMENT 'launch date',
MODIFIED datetime NOT NULL COMMENT 'modified time',
CREATED datetime NOT NULL COMMENT 'created time',
PRIMARY KEY(ID)
)
COMMENT='DB WorkerID Assigner for UID Generator',ENGINE = INNODB;
5.复制WorkerNodeDAO
从uid-generator 源码中复制WorkerNodeDAO到自己项目,改名WorkerNodeMapper,并去掉@Repository注解,添加@Mapper
@Mapper
public interface WorkerNodeMapper {
WorkerNodeEntity getWorkerNodeByHostPort(@Param("host") String host, @Param("port") String port);
void addWorkerNode(WorkerNodeEntity workerNodeEntity);
}
6.复制WORKER_NODE.xml文件
把WORKER_NODE.xml文件拷贝到resource的目录下,改名为WorkerNodeMapper.xml,并把里面的namespace="com.baidu.fsg.uid.worker.dao.WorkerNodeDAO"改为自己项目中的WorkerNodeMapper。
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.demo9.mapper.WorkerNodeMapper">
<resultMap id="workerNodeRes"
type="com.baidu.fsg.uid.worker.entity.WorkerNodeEntity">
<id column="ID" jdbcType="BIGINT" property="id" />
<result column="HOST_NAME" jdbcType="VARCHAR" property="hostName" />
<result column="PORT" jdbcType="VARCHAR" property="port" />
<result column="TYPE" jdbcType="INTEGER" property="type" />
<result column="LAUNCH_DATE" jdbcType="DATE" property="launchDate" />
<result column="MODIFIED" jdbcType="TIMESTAMP" property="modified" />
<result column="CREATED" jdbcType="TIMESTAMP" property="created" />
</resultMap>
<insert id="addWorkerNode" useGeneratedKeys="true" keyProperty="id"
parameterType="com.baidu.fsg.uid.worker.entity.WorkerNodeEntity">
INSERT INTO WORKER_NODE
(HOST_NAME,
PORT,
TYPE,
LAUNCH_DATE,
MODIFIED,
CREATED)
VALUES (
#{hostName},
#{port},
#{type},
#{launchDate},
NOW(),
NOW())
</insert>
<select id="getWorkerNodeByHostPort" resultMap="workerNodeRes">
SELECT
ID,
HOST_NAME,
PORT,
TYPE,
LAUNCH_DATE,
MODIFIED,
CREATED
FROM
WORKER_NODE
WHERE
HOST_NAME = #{host} AND PORT = #{port}
</select>
</mapper>
7.复制DisposableWorkerIdAssigner类
复制DisposableWorkerIdAssigner类到自己的工程中,并把里面依赖的WorkerNodeDAO 改为本地新增的WorkerNodeMapper,并添加@Component 注解
@Component
public class DisposableWorkerIdAssigner implements WorkerIdAssigner {
private static final Logger LOGGER = LoggerFactory.getLogger(com.baidu.fsg.uid.worker.DisposableWorkerIdAssigner.class);
@Autowired
private WorkerNodeMapper workerNodeDAO;
@Transactional
public long assignWorkerId() {
// build worker node entity
WorkerNodeEntity workerNodeEntity = buildWorkerNode();
// add worker node for new (ignore the same IP + PORT)
workerNodeDAO.addWorkerNode(workerNodeEntity);
LOGGER.info("Add worker node:" + workerNodeEntity);
return workerNodeEntity.getId();
}
private WorkerNodeEntity buildWorkerNode() {
WorkerNodeEntity workerNodeEntity = new WorkerNodeEntity();
if (DockerUtils.isDocker()) {
workerNodeEntity.setType(WorkerNodeType.CONTAINER.value());
workerNodeEntity.setHostName(DockerUtils.getDockerHost());
workerNodeEntity.setPort(DockerUtils.getDockerPort());
} else {
workerNodeEntity.setType(WorkerNodeType.ACTUAL.value());
workerNodeEntity.setHostName(NetUtils.getLocalAddress());
workerNodeEntity.setPort(System.currentTimeMillis() + "-" + RandomUtils.nextInt(100000));
}
return workerNodeEntity;
}
}
8.修改mybatis配置
在application.yml配置文件中修改mybatis的配置,添加com.baidu.fsg.uid.worker.entity 扫描
mybatis.mapper-locations: classpath:/mapper/*.xml
mybatis.typeAliasesPackage: com.demo9.entity,com.baidu.fsg.uid.worker.entity
9.添加UidGenerator配置,并添加扫描路径
@Configuration
@ComponentScan(basePackages = {"com.baidu.fsg.uid"})
public class CachedUidGeneratorConfig {
@Bean
public DefaultUidGenerator defaultUidGenerator(WorkerIdAssigner disposableWorkerIdAssigner) {
DefaultUidGenerator defaultUidGenerator = new DefaultUidGenerator();
defaultUidGenerator.setWorkerIdAssigner(disposableWorkerIdAssigner);
defaultUidGenerator.setTimeBits(32);
defaultUidGenerator.setWorkerBits(22);
defaultUidGenerator.setSeqBits(9);
defaultUidGenerator.setEpochStr("2020-01-01");
return defaultUidGenerator;
}
@Bean
public CachedUidGenerator cachedUidGenerator(WorkerIdAssigner disposableWorkerIdAssigner) {
CachedUidGenerator cachedUidGenerator = new CachedUidGenerator();
cachedUidGenerator.setWorkerIdAssigner(disposableWorkerIdAssigner);
cachedUidGenerator.setTimeBits(32);
cachedUidGenerator.setWorkerBits(22);
cachedUidGenerator.setSeqBits(9);
cachedUidGenerator.setEpochStr("2020-01-01");
cachedUidGenerator.setBoostPower(3);
cachedUidGenerator.setScheduleInterval(60L);
return cachedUidGenerator;
}
}
10.添加单元测试类
只需引入cachedUidGenerator bean实例,直接调用cachedUidGenerator.getUID()方法即可生成uid
@RunWith(SpringRunner.class)
@SpringBootTest(classes = DemoApplication.class)
public class UidGeneratorTest {
@Autowired
@Qualifier("cachedUidGenerator")
private UidGenerator uidGenerator;
@Test
public void getUidTest() {
Long uid = uidGenerator.getUID();
System.out.println(uid);
}
}
每生成一个uid都会往WORKER_NODE表中插入一条记录
四、UidGenerator的说明
UidGenerator提供两种方式实现:DefaultUidGenerator和CachedUidGenerator。
DefaultUidGenerator
DefaultUidGenerator是UidGenerator 默认的实现方式
参数 | 说明 |
---|---|
timeBits | 相对于时间基点"2016-05-20"的增量值,单位:秒,可使用的时间为2^timeBis 秒例如:timeBits=30,则可使用230秒,约34年,timeBits=31,则可使用231秒,约68年 |
workerBits | 机器id,最多可支持2^22约420w次机器启动。内置实现为在启动时由数据库分配,默认分配策略为用后即弃,每次启动都会重新生成一批ID,因此重启次数也是会有限制的,后续可提供复用策略。 |
seqBits | 每秒下的并发序列,9 bits可支持每台服务器每秒512个并发。 |
epochStr | 指集成UidGenerator生成分布式ID服务第一次上线的时间,可配置,也一定要根据你的上线时间进行配置,因为默认的epoch时间可是2016-09-20,不配置的话,会浪费好几年的可用时间。 |
CachedUidGenerator
CachedUidGenerator是UidGenerator的重要改进实现
参数 | 说明 |
---|---|
boostPower | RingBuffer size扩容参数, 可提高UID生成的吞吐量默认:3, 原bufferSize=8192, 扩容后bufferSize= 8192 << 3 = 65536 |
paddingFactor | 指定何时向RingBuffer中填充UID, 取值为百分比(0, 100), 默认为50举例: bufferSize=1024, paddingFactor=50 -> threshold=1024 * 50 / 100 = 512.当环上可用UID数量 < 512时, 将自动对RingBuffer进行填充补全 |
scheduleInterval | 另外一种RingBuffer填充时机, 在Schedule线程中, 周期性检查填充默认:不配置此项, 即不实用Schedule线程. 如需使用, 请指定Schedule线程时间间隔, 单位:秒 |
----------------------------------欢迎大家点赞留言讨论---------------------------------