UidGenerator
是Java实现的, 基于Snowflake算法的唯一ID生成器。UidGenerator
以组件形式工作在应用项目中, 支持自定义workerId位数和初始化策略, 从而适用于docker等虚拟化环境下实例自动重启、漂移等场景。 在实现上,UidGenerator
通过借用未来时间来解决sequence天然存在的并发限制; 采用RingBuffer来缓存已生成的UID, 并行化UID的生产和消费, 同时对CacheLine补齐,避免了由RingBuffer带来的硬件级「伪共享」问题. 最终单机QPS可达600万。官方地址:https://github.com/baidu/uid-generator
uid-generator提供了两种生成器: DefaultUidGenerator、CachedUidGenerator。如对UID生成性能有要求, 请使用CachedUidGenerator
CachedUidGenerator
RingBuffer环形数组,数组每个元素成为一个slot。RingBuffer容量,默认为Snowflake算法中sequence最大值,且为2^N。可通过boostPower
配置进行扩容,以提高RingBuffer 读写吞吐量。
Tail指针、Cursor指针用于环形数组上读写slot:
- Tail指针
表示Producer生产的最大序号(此序号从0开始,持续递增)。Tail不能超过Cursor,即生产者不能覆盖未消费的slot。当Tail已赶上curosr,此时可通过rejectedPutBufferHandler
指定PutRejectPolicy - Cursor指针
表示Consumer消费到的最小序号(序号序列与Producer序列相同)。Cursor不能超过Tail,即不能消费未生产的slot。当Cursor已赶上tail,此时可通过rejectedTakeBufferHandler
指定TakeRejectPolicy
CachedUidGenerator采用了双RingBuffer,Uid-RingBuffer用于存储Uid、Flag-RingBuffer用于存储Uid状态(是否可填充、是否可消费)
由于数组元素在内存中是连续分配的,可最大程度利用CPU cache以提升性能。但同时会带来「伪共享」FalseSharing问题,为此在Tail、Cursor指针、Flag-RingBuffer中采用了CacheLine 补齐方式。
RingBuffer填充时机
- 初始化预填充
RingBuffer初始化时,预先填充满整个RingBuffer. - 即时填充
Take消费时,即时检查剩余可用slot量(tail
-cursor
),如小于设定阈值,则补全空闲slots。阈值可通过paddingFactor
来进行配置,请参考Quick Start中CachedUidGenerator配置 - 周期填充
通过Schedule线程,定时补全空闲slots。可通过scheduleInterval
配置,以应用定时填充功能,并指定Schedule时间间隔
1、导入依赖
<!-- https://mvnrepository.com/artifact/com.xfvape.uid/uid-generator -->
<dependency>
<groupId>com.xfvape.uid</groupId>
<artifactId>uid-generator</artifactId>
<version>0.0.4-RELEASE</version>
</dependency>
2、创建表WORKER_NODE
运行sql脚本以导入表WORKER_NODE, 脚本如下:
DROP DATABASE IF EXISTS `xxxx`;
CREATE DATABASE `xxxx` ;
use `xxxx`;
DROP TABLE IF EXISTS WORKER_NODE;
CREATE TABLE WORKER_NODE
(
ID BIGINT NOT NULL AUTO_INCREMENT COMMENT 'auto increment id',
HOST_NAME VARCHAR(64) NOT NULL COMMENT 'host name',
PORT VARCHAR(64) NOT NULL COMMENT 'port',
TYPE INT NOT NULL COMMENT 'node type: ACTUAL or CONTAINER',
LAUNCH_DATE DATE NOT NULL COMMENT 'launch date',
MODIFIED TIMESTAMP NOT NULL COMMENT 'modified time',
CREATED TIMESTAMP NOT NULL COMMENT 'created time',
PRIMARY KEY(ID)
)
COMMENT='DB WorkerID Assigner for UID Generator',ENGINE = INNODB;
3、maven依赖
<dependencies>
<dependency>
<groupId>com.xiaobear.distributedid</groupId>
<artifactId>distributedid-core</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.mybatis.spring.boot</groupId>
<artifactId>mybatis-spring-boot-starter</artifactId>
<version>2.1.0</version>
</dependency>
<!--必须放在最后-->
<!-- https://mvnrepository.com/artifact/com.xfvape.uid/uid-generator -->
<dependency>
<groupId>com.xfvape.uid</groupId>
<artifactId>uid-generator</artifactId>
<version>0.0.4-RELEASE</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>log4j-over-slf4j</artifactId>
</exclusion>
<exclusion>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>org.mybatis</groupId>
<artifactId>mybatis-spring</artifactId>
</exclusion>
<exclusion>
<groupId>org.mybatis</groupId>
<artifactId>mybatis</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
4、配置文件
spring:
datasource:
type: com.zaxxer.hikari.HikariDataSource
url: jdbc:mysql://127.0.0.1:3306/distributedid?serverTimezone=Asia/Shanghai&characterEncoding=utf8&useUnicode=true&useSSL=false&autoReconnect=true
username: root
password: 密码
driver-class-name: com.mysql.cj.jdbc.Driver
server:
port: 8088
mybatis:
type-aliases-package: com.xiaobear.distributedid.core.domain
mapper-locations: classpath*:mapper/*.xml
5、主启动类
@SpringBootApplication
@MapperScan("com.xiaobear.distributedid.mapper")
public class UidgeneratorApplication {
public static void main(String[] args) {
SpringApplication.run(UidgeneratorApplication.class);
System.out.println("百度生成ID 启动成功");
}
}
6、业务类
1、实体类
public class WorkerNode {
private Long id;
private String hostName;
private String port;
private Integer type;
private Date launchDate;
private Date modified;
private Date created;
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public String getHostName() {
return hostName;
}
public void setHostName(String hostName) {
this.hostName = hostName;
}
public String getPort() {
return port;
}
public void setPort(String port) {
this.port = port;
}
public Integer getType() {
return type;
}
public void setType(Integer type) {
this.type = type;
}
public Date getLaunchDate() {
return launchDate;
}
public void setLaunchDate(Date launchDate) {
this.launchDate = launchDate;
}
public Date getModified() {
return modified;
}
public void setModified(Date modified) {
this.modified = modified;
}
public Date getCreated() {
return created;
}
public void setCreated(Date created) {
this.created = created;
}
@Override
public String toString() {
return "WorkerNode{" +
"id=" + id +
", hostName='" + hostName + '\'' +
", port='" + port + '\'' +
", type=" + type +
", launchDate=" + launchDate +
", modified=" + modified +
", created=" + created +
'}';
}
}
2、数据层接口
@Mapper
public interface WorkerNodeMapper {
/**
* 添加对象
* @param workerNodeEntity
* @return
*/
int addWorkerNode(WorkerNode workerNodeEntity);
/**
* 通过host port 获取ID
* @param host
* @param port
* @return
*/
WorkerNode getWorkerNodeByHostPort(@Param("host") String host, @Param("port") String port);
}
mapper.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.xiaobear.distributedid.mapper.WorkerNodeMapper">
<resultMap id="workerNodeRes" type="com.xiaobear.distributedid.core.domain.WorkerNode">
<id column="ID" jdbcType="BIGINT" property="id"/>
<result column="HOST_NAME" jdbcType="VARCHAR" property="hostName"/>
<result column="PORT" jdbcType="VARCHAR" property="port"/>
<result column="TYPE" jdbcType="INTEGER" property="type"/>
<result column="LAUNCH_DATE" jdbcType="DATE" property="launchDate"/>
<result column="MODIFIED" jdbcType="TIMESTAMP" property="modified"/>
<result column="CREATED" jdbcType="TIMESTAMP" property="created"/>
</resultMap>
<insert id="addWorkerNode" useGeneratedKeys="true" keyProperty="id"
parameterType="com.xiaobear.distributedid.core.domain.WorkerNode">
INSERT INTO WORKER_NODE
(HOST_NAME,
PORT,
TYPE,
LAUNCH_DATE,
MODIFIED,
CREATED)
VALUES (
#{hostName},
#{port},
#{type},
#{launchDate},
NOW(),
NOW())
</insert>
<select id="getWorkerNodeByHostPort" resultMap="workerNodeRes">
SELECT
ID,
HOST_NAME,
PORT,
TYPE,
LAUNCH_DATE,
MODIFIED,
CREATED
FROM
WORKER_NODE
WHERE
HOST_NAME = #{host} AND PORT = #{port}
</select>
</mapper>
3、实现层接口
public interface IWorkerNodeService {
/**
* 获取ID
* @return
*/
public long genUid();
}
4、实现层实现类
@Service
public class WorkerNodeServiceImpl implements IWorkerNodeService {
@Resource
private UidGenerator uidGenerator;
/**
* 获取百度生成ID
* @return
*/
@Override
public long genUid() {
return uidGenerator.getUID();
}
}
5、控制层接口
@RestController
public class WorkerNodeController {
@Resource
private IWorkerNodeService workerNodeService;
/**
*使用百度 Uidgenerator获取ID
* @return
*/
@GetMapping("/Uidgenerator")
public RestResponse getIdByBaiDuUid(){
long id = workerNodeService.genUid();
return RestResponse.success(id);
}
}
7、测试
http://localhost:8088/Uidgenerator
{"code":200,"msg":"成功","result":7067247636209745920}
8、uid-generator核心对象装配为spring的bean
1、重写WorkerIdAssigner接口
public class DisposableWorkerIdAssigner implements WorkerIdAssigner{
@Resource
private WorkerNodeMapper workerNodeMapper;
@Override
@Transactional(rollbackFor = Exception.class)
public long assignWorkerId() {
WorkerNode workerNode = buildWorkerNode();
workerNodeMapper.addWorkerNode(workerNode);
return workerNode.getId();
}
private WorkerNode buildWorkerNode() {
WorkerNode workNode = new WorkerNode();
if (DockerUtils.isDocker()) {
workNode.setType(WorkerNodeType.CONTAINER.value());
workNode.setHostName(DockerUtils.getDockerHost());
workNode.setPort(DockerUtils.getDockerPort());
} else {
workNode.setType(WorkerNodeType.ACTUAL.value());
workNode.setHostName(NetUtils.getLocalAddress());
workNode.setPort(System.currentTimeMillis() + "-" + RandomUtils.nextInt(100000));
}
workNode.setLaunchDate(new Date());
return workNode;
}
}
2、自动转配bean
@Configuration
public class WorkerNodeConfig {
@Bean(name = "disposableWorkerIdAssigner")
public DisposableWorkerIdAssigner disposableWorkerIdAssigner(){
return new DisposableWorkerIdAssigner();
}
@Bean(name = "cachedUidGenerator")
public UidGenerator uidGenerator(DisposableWorkerIdAssigner disposableWorkerIdAssigner){
CachedUidGenerator cachedUidGenerator = new CachedUidGenerator();
cachedUidGenerator.setWorkerIdAssigner(disposableWorkerIdAssigner);
return cachedUidGenerator;
}
}
3、使用,在实现类指定生成ID的bean
@Service
public class WorkerNodeServiceImpl implements IWorkerNodeService {
@Resource(name = "cachedUidGenerator")
private UidGenerator uidGenerator;
/**
* 获取百度生成ID
* @return
*/
@Override
public long genUid() {
return uidGenerator.getUID();
}
}