背景:
工作中经常需要用到分库分表,其中shardingsphere上手快且成本低(当当网开源,后由apache维护),但是为了在业务场景中使用起来更加灵活,比如想对数据源进行热更新,这样就不用每次更新数据源都需要重启项目,适用于项目中数据源更新频繁的情况,本章借助nacos实现热更新分库分表数据源热更新功能。
一、理论
理论知识大家可以直接去shardingsphere的官网查看。
官方文档:Overview :: ShardingSphere
二、demo
理论知识网上已经很多前辈总结了,在此不在赘述,直接进行实战。
1、准备工作
首先准备一下数据库以及要引入的依赖
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.10</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>druid</artifactId>
<version>1.1.6</version>
<exclusions>
<exclusion>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>sharding-jdbc-orchestration-spring-boot-starter</artifactId>
<version>4.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>sharding-core-common</artifactId>
<version>4.1.1</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.48</version>
</dependency>
<dependency>
<groupId>org.mybatis.spring.boot</groupId>
<artifactId>mybatis-spring-boot-starter</artifactId>
<version>2.0.1</version>
</dependency>
<dependency>
<groupId>com.alibaba.nacos</groupId>
<artifactId>nacos-client</artifactId>
<version>1.4.2</version>
</dependency>
2、application.properties配置
server.port=8090
spring.application.name=sharding-jdbc-demo
#分三个数据库
spring.shardingsphere.datasource.names=ds-master-0,ds-master-1,ds-master-2
spring.shardingsphere.datasource.ds-master-0.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.ds-master-0.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.ds-master-0.url=XXXXXX
spring.shardingsphere.datasource.ds-master-0.username=XXXXXX
spring.shardingsphere.datasource.ds-master-0.password=XXXXXX
spring.shardingsphere.datasource.ds-master-1.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.ds-master-1.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.ds-master-1.url=XXXXXX
spring.shardingsphere.datasource.ds-master-1.username=XXXXXX
spring.shardingsphere.datasource.ds-master-1.password=XXXXXX
spring.shardingsphere.datasource.ds-master-2.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.ds-master-2.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.ds-master-2.url=XXXXXX
spring.shardingsphere.datasource.ds-master-2.username=XXXXXX
spring.shardingsphere.datasource.ds-master-2.password=XXXXXX
#三个数据库 每个数据库两个表 通过id分片选择路由到哪张表
spring.shardingsphere.sharding.tables.test_user.actual-data-nodes=ds-master-$->{0..2}.test_user_$->{0..1}
spring.shardingsphere.sharding.tables.test_user.table-strategy.inline.sharding-column=id
spring.shardingsphere.sharding.tables.test_user.table-strategy.inline.algorithm-expression=test_user_$->{id % 2}
#id生成方式可选择雪花算法
#spring.shardingsphere.sharding.tables.test_user.key-generator.column=id
#spring.shardingsphere.sharding.tables.test_user.key-generator.type=SNOWFLAKE
spring.shardingsphere.sharding.default-database-strategy.inline.sharding-column=id
#数据库分片选择id取余
spring.shardingsphere.sharding.default-database-strategy.inline.algorithm-expression=ds-master-$->{id % 3}
spring.shardingsphere.props.sql.show=true
spring.main.allow-bean-definition-overriding=true
management.endpoints.web.exposure.include=*
mybatis.type-aliases-package=com.calvin.sharding.pojo
mybatis.mapper-locations=classpath:mapper/*.xml
mybatis.configuration.map-underscore-to-camel-case=true
mybatis-plus.configuration.jdbc-type-for-null=null
mybatis-plus.configuration.log-impl=org.apache.ibatis.logging.stdout.StdOutImpl
mybatis-plus.global-config.banner=false
#nacos配置中心
spring.shardingsphere.orchestration.config_center_sharding.orchestration-type=config_center
spring.shardingsphere.orchestration.config_center_sharding.instance-type=nacos
spring.shardingsphere.orchestration.config_center_sharding.server-lists=XXXXXX
spring.shardingsphere.orchestration.config_center_sharding.props.access-key=XXXXXX
spring.shardingsphere.orchestration.config_center_sharding.props.secret-key=XXXXXX
spring.shardingsphere.orchestration.config_center_sharding.props.overwrite=true
#nacos注册中心
spring.shardingsphere.orchestration.registry_center_sharding.orchestration-type=registry_center
spring.shardingsphere.orchestration.registry_center_sharding.instance-type=nacos
spring.shardingsphere.orchestration.registry_center_sharding.server-lists=XXXXXX
spring.shardingsphere.orchestration.registry_center_sharding.props.access-key=XXXXXX
spring.shardingsphere.orchestration.registry_center_sharding.props.secret-key=XXXXXX
spring.shardingsphere.orchestration.registry_center_sharding.props.overwrite=true
3、mybatis配置
import org.apache.ibatis.session.SqlSessionFactory;
import org.mybatis.spring.SqlSessionFactoryBean;
import org.mybatis.spring.SqlSessionTemplate;
import org.mybatis.spring.annotation.MapperScan;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.core.io.Resource;
import javax.sql.DataSource;
@Configuration
@MapperScan(
basePackages = {"com.alibaba.ascp.plan.dao.mysql.mapper"},
sqlSessionFactoryRef = "sqlSessionFactory",
sqlSessionTemplateRef = "sqlSessionTemplate"
)
public class MybatisConfig {
@Primary
@Bean("sqlSessionFactory")
public SqlSessionFactory sqlSessionFactory(@Autowired DataSource dataSource,
@Value("classpath:/mybatis-config.xml") Resource configLocation,
@Value("classpath:mapper/*.xml") Resource[] mapperLocations
) throws Exception {
SqlSessionFactoryBean sqlSessionFactoryBean = new SqlSessionFactoryBean();
sqlSessionFactoryBean.setDataSource(dataSource);
sqlSessionFactoryBean.setConfigLocation(configLocation);
sqlSessionFactoryBean.setMapperLocations(mapperLocations);
return sqlSessionFactoryBean.getObject();
}
@Primary
@Bean(name = "sqlSessionTemplate")
public SqlSessionTemplate sqlSessionTemplate(SqlSessionFactory sqlSessionFactory) {
return new SqlSessionTemplate(sqlSessionFactory);
}
}
4、mybatis-config.xml
<?xml version="1.0" encoding="UTF-8" ?>
<!-- mybatis的配置文件 -->
<!DOCTYPE configuration PUBLIC "-//mybatis.org//DTD Config 3.0//EN" "http://mybatis.org/dtd/mybatis-3-config.dtd">
<configuration>
<settings>
<!-- 映射失败时抛出异常,阻止执行 -->
<setting name="autoMappingUnknownColumnBehavior" value="FAILING"/>
<!-- 支持下划线分隔命名的属性 -->
<setting name="mapUnderscoreToCamelCase" value="true"/>
<!-- 支持show_sql -->
<!-- <setting name="logPrefix" value="ascp.plan.mybatis."/>-->
<!-- 允许插入null -->
<setting name="jdbcTypeForNull" value="NULL"/>
</settings>
<typeAliases>
</typeAliases>
</configuration>
5、TestUser实体类
import java.io.Serializable;
public class TestUser implements Serializable ,Comparable{
private static final long serialVersionUID = -1205226416664488559L;
private Long id;
private String username;
private String password;
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public String getUsername() {
return username;
}
public void setUsername(String username) {
this.username = username;
}
public String getPassword() {
return password;
}
public void setPassword(String password) {
this.password = password;
}
@Override
public String toString() {
return "User{" +
"id=" + id +
", username='" + username + '\'' +
", password='" + password + '\'' +
'}';
}
@Override
public int compareTo(Object o) {
TestUser u= (TestUser) o;
return this.id.compareTo(u.id);
}
}
6、UserMapper
package com.calvin.sharding.mapper;
import com.calvin.sharding.pojo.TestUser;
import org.apache.ibatis.annotations.Mapper;
import java.util.List;
@Mapper
public interface UserMapper {
int insert(TestUser testUser);
List<TestUser> selectList();
int deleteById(Integer id);
}
7、UserMapper.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.calvin.sharding.mapper.UserMapper">
<resultMap id="BaseResultMap" type="com.calvin.sharding.pojo.TestUser">
<result column="id" jdbcType="BIGINT" property="id"/>
<result column="username" jdbcType="VARCHAR" property="username"/>
<result column="password" jdbcType="VARCHAR" property="password"/>
</resultMap>
<sql id="table_name">test_user</sql>
<sql id="Base_Column_List">
id,
<include refid="Column_List_Insert"/>
</sql>
<sql id="Column_List_Insert">
username, password
</sql>
<sql id="Field_List_Insert">
#{item.username,jdbcType=VARCHAR},
#{item.password,jdbcType=VARCHAR}
</sql>
<insert id="insert" useGeneratedKeys="true" keyColumn="id" keyProperty="id"
parameterType="com.calvin.sharding.pojo.TestUser">
INSERT INTO test_user(id,username,password) values(#{id},#{username},#{password})
</insert>
<select id="selectList" resultType="com.calvin.sharding.pojo.TestUser">
select id,username,password from test_user limit 100
</select>
<select id="deleteById" resultType="java.lang.Integer">
delete from test_user where id=${_parameter}
</select>
</mapper>
8、UserService接口
import com.calvin.sharding.pojo.TestUser;
import java.util.List;
public interface UserService {
Integer addUser(TestUser user);
List<TestUser> getUsers();
TestUser getUser(Integer id);
boolean deleteOne(Integer id);
}
9、UserServiceIImpl实现类
import com.calvin.sharding.mapper.UserMapper;
import com.calvin.sharding.pojo.TestUser;
import com.calvin.sharding.service.UserService;
import org.springframework.stereotype.Service;
import javax.annotation.Resource;
import java.util.List;
@Service
public class UserServiceImpl implements UserService {
@Resource
private UserMapper userMapper;
@Override
public Integer addUser(TestUser user) {
return userMapper.insert(user);
}
@Override
public List<TestUser> getUsers() {
return userMapper.selectList();
}
@Override
public TestUser getUser(Integer id) {
return null;
}
@Override
public boolean deleteOne(Integer id) {
int i = userMapper.deleteById(id);
return i > 0;
}
}
10、NacosProperties
import org.apache.shardingsphere.underlying.common.properties.TypedProperties;
import java.util.Properties;
public final class NacosProperties extends TypedProperties<NacosPropertyKey> {
public NacosProperties(final Properties props) {
super(NacosPropertyKey.class, props);
}
}
11、NacosPropertyKey
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import org.apache.shardingsphere.underlying.common.properties.TypedPropertyKey;
/**
* Typed property key of Nacos.
*/
@RequiredArgsConstructor
@Getter
public enum NacosPropertyKey implements TypedPropertyKey {
/**
* Nacos config service group name.
*/
GROUP("group", "SHARDING_SPHERE_DEFAULT_GROUP", String.class),
/**
* Nacos get config data timeout value.
*/
TIMEOUT("timeout", String.valueOf(3000), long.class);
private final String key;
private final String defaultValue;
private final Class<?> type;
}
12、NacosRegistryCenter
import com.alibaba.nacos.api.NacosFactory;
import com.alibaba.nacos.api.config.ConfigService;
import com.alibaba.nacos.api.config.listener.Listener;
import com.alibaba.nacos.api.exception.NacosException;
import lombok.Getter;
import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
import org.apache.shardingsphere.orchestration.center.ConfigCenterRepository;
import org.apache.shardingsphere.orchestration.center.RegistryCenterRepository;
import org.apache.shardingsphere.orchestration.center.config.CenterConfiguration;
import org.apache.shardingsphere.orchestration.center.listener.DataChangedEvent;
import org.apache.shardingsphere.orchestration.center.listener.DataChangedEventListener;
import org.apache.shardingsphere.orchestration.center.util.ConfigKeyUtils;
import org.springframework.util.Assert;
import java.util.List;
import java.util.Properties;
import java.util.concurrent.Executor;
@Slf4j
public final class NacosRegistryCenter implements ConfigCenterRepository, RegistryCenterRepository {
private ConfigService configService;
private NacosProperties nacosProperties;
@Getter
@Setter
private Properties properties = new Properties();
public static final String ACCESS_KEY = "access-key";
public static final String SECRET_KEY = "secret-key";
@Override
public void persistEphemeral(String s, String s1) {
System.out.println("persistEphemeral: " + s + "," + s1);
}
@Override
public void init(CenterConfiguration config) {
try {
final String accessKey = config.getProperties().getProperty(ACCESS_KEY);
final String secretKey = config.getProperties().getProperty(SECRET_KEY);
Assert.notNull(accessKey, "access-key can not be null");
Assert.notNull(secretKey, "secret-key can not be null");
nacosProperties = new NacosProperties(properties);
Properties properties = new Properties();
properties.put("serverAddr", config.getServerLists());
properties.put("namespace", null == config.getNamespace() ? "" : config.getNamespace());
properties.put("accessKey", accessKey);
properties.put("secretKey", secretKey);
configService = NacosFactory.createConfigService(properties);
} catch (final NacosException ex) {
log.debug("exception for: {}", ex.toString());
}
}
/**
* Get data from nacos instance.
*
* @param key key of data
* @return value of data
*/
@Override
public String get(final String key) {
try {
String dataId = ConfigKeyUtils.path2Key(key);
String group = nacosProperties.getValue(NacosPropertyKey.GROUP);
long timeoutMs = nacosProperties.getValue(NacosPropertyKey.TIMEOUT);
return configService.getConfig(dataId, group, timeoutMs);
} catch (final NacosException ex) {
log.debug("Nacos get config value exception for: {}", ex.toString());
return null;
}
}
/**
* Get node's sub-nodes list.
*
* @param key key of data
* @return sub-nodes name list
*/
@Override
public List<String> getChildrenKeys(final String key) {
return null;
}
/**
* Persist data.
*
* @param key key of data
* @param value value of data
*/
@Override
public void persist(final String key, final String value) {
try {
String dataId = ConfigKeyUtils.path2Key(key);
String group = nacosProperties.getValue(NacosPropertyKey.GROUP);
configService.publishConfig(dataId, group, value);
} catch (final NacosException ex) {
log.debug("Nacos persist config exception for: {}", ex.toString());
}
}
/**
* Watch key or path of the config server.
*
* @param key key of data
* @param dataChangedEventListener data changed event listener
*/
@Override
public void watch(final String key, final DataChangedEventListener dataChangedEventListener) {
try {
System.out.println("watch key:" + key);
String dataId = ConfigKeyUtils.path2Key(key);
String group = nacosProperties.getValue(NacosPropertyKey.GROUP);
configService.addListener(dataId, group, new Listener() {
@Override
public Executor getExecutor() {
return null;
}
@Override
public void receiveConfigInfo(final String configInfo) {
dataChangedEventListener.onChange(new DataChangedEvent(key, configInfo, DataChangedEvent.ChangedType.UPDATED));
}
});
} catch (final NacosException ex) {
log.debug("Nacos watch key exception for: {}", ex.toString());
}
}
@Override
public void close() {
}
/**
* Get algorithm type.
*
* @return type
*/
@Override
public String getType() {
return "nacos";
}
//@Override
//public String get(final String key) {
// return getDirectly(key);
//}
//
//@Override
//public String getDirectly(final String key) {
// try {
// String dataId = key.replace("/", ".");
// String group = properties.getProperty("group", "SHARDING_SPHERE_DEFAULT_GROUP");
// long timeoutMs = Long.parseLong(properties.getProperty("timeout", "3000"));
// return configService.getConfig(dataId, group, timeoutMs);
// } catch (final NacosException ex) {
// log.debug("exception for: {}", ex.toString());
// return null;
// }
//}
//
//@Override
//public boolean isExisted(final String key) {
// return !Strings.isNullOrEmpty(getDirectly(key));
//}
//
//@Override
//public List<String> getChildrenKeys(final String key) {
// return null;
//}
//
//@Override
//public void persist(final String key, final String value) {
// update(key, value);
//}
//
//@Override
//public void update(final String key, final String value) {
// try {
// String dataId = key.replace("/", ".");
// String group = properties.getProperty("group", "SHARDING_SPHERE_DEFAULT_GROUP");
// configService.publishConfig(dataId, group, value);
// } catch (final NacosException ex) {
// log.debug("exception for: {}", ex.toString());
// }
//}
//
//@Override
//public void persistEphemeral(final String key, final String value) {
//
//}
//
//@Override
//public void watch(final String key, final DataChangedEventListener dataChangedEventListener) {
// try {
// String dataId = key.replace("/", ".");
// String group = properties.getProperty("group", "SHARDING_SPHERE_DEFAULT_GROUP");
// configService.addListener(dataId, group, new Listener() {
//
// @Override
// public Executor getExecutor() {
// return null;
// }
//
// @Override
// public void receiveConfigInfo(final String configInfo) {
// dataChangedEventListener.onChange(new DataChangedEvent(key, configInfo, DataChangedEvent.ChangedType.UPDATED));
// }
// });
// } catch (final NacosException ex) {
// log.debug("exception for: {}", ex.toString());
// }
//}
//
//@Override
//public void close() {
//}
//
//
//@Override
//public String getType() {
// return "nacos";
//}
}
最后使用SPI调用的方式,加载nacos的注册中心和配置中心,具体做法是在resource包下的META-INF包下新建services包,然后新建两个文件分别为:
org.apache.shardingsphere.orchestration.center.ConfigCenterRepository、
org.apache.shardingsphere.orchestration.center.RegistryCenterRepository
文件内容都为我们前面配置的NacosRegistryCenter类的全路径:
com.calvin.sharding.reg.nacos.NacosRegistryCenter
三、数据分片算法
shardingsphere还提供主从配置、加密配置和影子表配置,以及可以自定义选择数据分片的算法,下面介绍两种分片算法:
1、标准分片算法
- 基于分片键的范围进行分片,如按照用户ID的范围分片。
- 可以通过配置实现基于某个字段的范围分片,如将用户ID在1-10000的数据存储在一个数据库实例中,用户ID在10001-20000的数据存储在另一个数据库实例中。
一下是自定义的分片算法demo:
精确分片算法
import org.apache.shardingsphere.api.sharding.standard.PreciseShardingAlgorithm;
import org.apache.shardingsphere.api.sharding.standard.PreciseShardingValue;
import org.springframework.stereotype.Component;
import java.util.Collection;
/**
* @author StrongerB
* 精确分片算法
*/
@Component
public class CustomPreciseShardingAlgorithm implements PreciseShardingAlgorithm<Long> {
@Override
public String doSharding(Collection<String> availableTargetNames, PreciseShardingValue<Long> preciseShardingValue) {
//用主键id作为分片键
Long id = preciseShardingValue.getValue();
String tableName = preciseShardingValue.getLogicTableName();
for (String param : availableTargetNames){
if (param.endsWith(tableName + "_info" + (id % 2))) {
return param;
}
}
throw new RuntimeException("database doesn't exist");
}
}
范围分片算法
import com.google.common.collect.Range;
import com.google.common.collect.Sets;
import org.apache.shardingsphere.api.sharding.standard.RangeShardingAlgorithm;
import org.apache.shardingsphere.api.sharding.standard.RangeShardingValue;
import org.springframework.stereotype.Component;
import java.util.Collection;
import java.util.Set;
/**
* @author StrongerB
* 范围分片算法
*/
@Component
public class CustomRangeShardingAlgorithm implements RangeShardingAlgorithm<Long> {
@Override
public Collection<String> doSharding(Collection<String> availableTargetNames, RangeShardingValue<Long> rangeShardingValue) {
String tableName = rangeShardingValue.getLogicTableName();
Range<Long> valueRange = rangeShardingValue.getValueRange();
Set<String> result = rangeShardingHandler(tableName, valueRange.lowerEndpoint(), valueRange.upperEndpoint(), availableTargetNames.size());
if (result.size() == 0){
throw new RuntimeException("range sharding error , please check table");
}
return result;
}
private Set<String> rangeShardingHandler(String logicTableName , Long lowerEndPoint , Long upperEndPoint , int targetNamesQuantity){
Set<String> result = Sets.newHashSet();
for (int i = 0; i < targetNamesQuantity; i++) {
String tableName = logicTableName.concat("_").concat("" + (i + 1));
result.add(tableName);
}
return result;
}
}
2、暗示分片算法
- 通过Hint API手动指定分片键值,适用于无法通过SQL分析得到分片键的场景。
- 例如,在某些业务场景中,分片键可能并不直接包含在SQL语句中,此时可以使用Hint分片算法来手动指定分片键值。
import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
import org.apache.shardingsphere.api.hint.HintManager;
import org.apache.shardingsphere.api.sharding.hint.HintShardingAlgorithm;
import org.apache.shardingsphere.api.sharding.hint.HintShardingValue;
import org.springframework.stereotype.Component;
import java.util.Collection;
import java.util.HashSet;
import java.util.List;
/**
* @author StrongerB
* 暗示分片算法
*/
@Component
public class CustomHintShardingAlgorithm implements HintShardingAlgorithm<Long> {
@Override
public Collection<String> doSharding(Collection<String> availableTargetNames, HintShardingValue<Long> hintShardingValue) {
HashSet<String> result = Sets.newHashSet();
Collection<Long> values = hintShardingValue.getValues();
for (String each : availableTargetNames){
for (Long shardingValue : values){
if (each.endsWith(String.valueOf(shardingValue % 2))) {
result.add(each);
}
}
}
if (result.size() == 0) {
throw new RuntimeException("hint sharding is null");
}
return result;
}
public void test(){
List<Integer> list = Lists.newArrayList(1, 2, 3, 4, 5);
for (Integer each : list) {
HintManager manager = HintManager.getInstance();
manager.addDatabaseShardingValue("table" , each);
manager.addTableShardingValue("table" , each);
//添加暗示条件后,执行业务处理(数据库操作)
}
}
}
在application.properties配置文件中,可以通过属性来选择使用什么分片算法:
spring.shardingsphere.sharding.tables.message_info.table-strategy.standard.sharding-column=id
spring.shardingsphere.sharding.tables.message_info.table-strategy.standard.precise-algorithm-class-name=com.shardingsphere.algorithm.CustomPreciseShardingAlgorithm
最后在nacos的控制台中即可动态修改数据源的信息以及分库分表的分片算法等,即可实现不用重新部署项目进行热更新配置。