Sharding-JDBC系列
2、Sharding-JDBC分库分表之SpringBoot分片策略
3、Sharding-JDBC分库分表之SpringBoot主从配置
4、SpringBoot集成Sharding-JDBC-5.3.0分库分表
5、SpringBoot集成Sharding-JDBC-5.3.0实现按月动态建表分表
8、【源码】Sharding-JDBC源码分析之Yaml分片配置文件解析原理
9、【源码】Sharding-JDBC源码分析之Yaml分片配置原理(一)
10、【源码】Sharding-JDBC源码分析之Yaml分片配置原理(二)
11、【源码】Sharding-JDBC源码分析之Yaml分片配置转换原理
前言
在《Sharding-JDBC系列》中用了三篇介绍了Yaml分片配置。在ShardingSphere的源码设计中,对于Yaml中最核心的配置信息rules和mode,解析后的对象的类名为YamlXxxConfiguration,且都实现了YamlConfiguration。
Yaml中的配置都是文本信息,在使用配置信息时,需要对一些配置信息进行转换(如通过type类型,匹配对应的类),对于每个YamlXxxConfiguration类,都有一个对应的配置转换后的类,类名为XxxConfiguration(如:YamlModeConfiguration对应ModeConfiguration)。
每个YamlXxxConfiguration都有一个对应的YamlXxxConfigurationSwapper,这些类都实现YamlConfigurationSwapper,用于YamlXxxConfiguration与XxxConfiguration的相互转换。
dataSources配置转换DataSource对象回顾
在【源码】Sharding-JDBC源码分析之Yaml分片配置原理(一)-CSDN博客中介绍了dataSources的配置以及转换为对应的DataSource对象。
1)通过YamlDataSourceConfigurationSwapper,将dataSources配置信息转换为DataSourceProperties。在DataSourceProperties中,对不同的dataSourceClassName进行兼容性处理;
2)通过DataSourcePoolCreator,使用反射机制,将DataSourceProperties创建为对应的DataSource对象。在DataSource对象中,添加了配置的数据源信息以及对应数据源的默认配置信息;
通过dataSourceClassName找到对应DataSourcePoolMetaData,在其中指定了默认配置项。如HikariDataSourcePoolMetaData。
rules配置转换
在YamlShardingSphereDataSourceFactory的createDataSource(final byte[] yamlBytes)方法中,先解析Yaml配置文件,得到YamlRootConfiguration对象。然后使用YamlDataSourceConfigurationSwapper将dataSources配置转换为DataSource对象。之后调用同名方法,对mode配置及rules配置进行转换。
3.1 YamlShardingSphereDataSourceFactory
YamlShardingSphereDataSourceFactory源码如下:
package org.apache.shardingsphere.driver.api.yaml;
@NoArgsConstructor(access = AccessLevel.PRIVATE)
public final class YamlShardingSphereDataSourceFactory {
// Yaml中rules配置转换引擎
private static final YamlRuleConfigurationSwapperEngine SWAPPER_ENGINE = new YamlRuleConfigurationSwapperEngine();
// yaml中dataSources配置信息转换器
private static final YamlDataSourceConfigurationSwapper DATA_SOURCE_SWAPPER = new YamlDataSourceConfigurationSwapper();
/**
* 创建ShardingSphereDataSource对象
* @param yamlBytes yaml配置字符串
*/
public static DataSource createDataSource(final byte[] yamlBytes) throws SQLException, IOException {
// 解析yaml配置文件,获取YamlRootConfiguration
YamlRootConfiguration rootConfig = YamlEngine.unmarshal(yamlBytes, YamlRootConfiguration.class);
// 调用DATA_SOURCE_SWAPPER.swapToDataSources(rootConfig.getDataSources(),
// 将Yaml中配置的dataSources信息转换为对应的DataSources对象
return createDataSource(DATA_SOURCE_SWAPPER.swapToDataSources(rootConfig.getDataSources()), rootConfig);
}
/**
* 创建ShardingSphereDataSource
*/
private static DataSource createDataSource(final Map<String, DataSource> dataSourceMap, final YamlRootConfiguration rootConfig) throws SQLException {
// mode配置转换
ModeConfiguration modeConfig = null == rootConfig.getMode() ? null : new YamlModeConfigurationSwapper().swapToObject(rootConfig.getMode());
// rules配置转换
Collection<RuleConfiguration> ruleConfigs = SWAPPER_ENGINE.swapToRuleConfigurations(rootConfig.getRules());
// 创建ShardingSphereDataSource
return ShardingSphereDataSourceFactory.createDataSource(rootConfig.getDatabaseName(), modeConfig, dataSourceMap, ruleConfigs, rootConfig.getProps());
}
}
通过YamlRuleConfigurationSwapperEngine.swapToRuleConfigurations()方法,将YamlRootConfiguration中的rules配置信息转换为RuleConfiguration集合。
3.2 YamlRuleConfigurationSwapperEngine
YamlRuleConfigurationSwapperEngine的源码如下:
package org.apache.shardingsphere.infra.yaml.config.swapper.rule;
/**
* Yaml中rules配置转换引擎
*/
public final class YamlRuleConfigurationSwapperEngine {
/**
* 将RuleConfiguration集合转换为YamlRuleConfiguration集合
*/
@SuppressWarnings("unchecked")
public Collection<YamlRuleConfiguration> swapToYamlRuleConfigurations(final Collection<RuleConfiguration> ruleConfigs) {
return OrderedSPIRegistry.getRegisteredServices(YamlRuleConfigurationSwapper.class, ruleConfigs).entrySet().stream()
.map(entry -> (YamlRuleConfiguration) entry.getValue().swapToYamlConfiguration(entry.getKey())).collect(Collectors.toList());
}
/**
* 将YamlRuleConfiguration集合转换为RuleConfiguration集合
*/
@SuppressWarnings("rawtypes")
public Collection<RuleConfiguration> swapToRuleConfigurations(final Collection<YamlRuleConfiguration> yamlRuleConfigs) {
Collection<RuleConfiguration> result = new LinkedList<>();
// 获取配置的规则类型。如数据分片规则YamlShardingRuleConfiguration,对应的类型为ShardingRuleConfiguration
Collection<Class<?>> ruleConfigTypes = yamlRuleConfigs.stream().map(YamlRuleConfiguration::getRuleConfigurationType).collect(Collectors.toList());
// 通过SPI,获取能够转换对应规则类型的转换器。如数据分片规则的转换器为YamlShardingRuleConfigurationSwapper,
// 该类的getTypeClass()方法返回ShardingRuleConfiguration.class
for (Entry<Class<?>, YamlRuleConfigurationSwapper> entry : OrderedSPIRegistry.getRegisteredServicesByClass(YamlRuleConfigurationSwapper.class, ruleConfigTypes).entrySet()) {
// 使用转换器,将YamlXxxRuleConfiguration转换为XxxRuleConfiguration,添加到result结合中
result.addAll(swapToRuleConfigurations(yamlRuleConfigs, entry.getKey(), entry.getValue()));
}
return result;
}
/**
* 从yamlRuleConfigs集合中找到YamlRuleConfiguration.getRuleConfigurationType()和ruleConfigType相等的YamlRuleConfiguration,
* 使用swapper转换器转换为对应的XxxRuleConfiguration
*/
@SuppressWarnings({"unchecked", "rawtypes"})
private Collection<RuleConfiguration> swapToRuleConfigurations(final Collection<YamlRuleConfiguration> yamlRuleConfigs,
final Class<?> ruleConfigType, final YamlRuleConfigurationSwapper swapper) {
return yamlRuleConfigs.stream()
.filter(each -> each.getRuleConfigurationType().equals(ruleConfigType)).map(each -> (RuleConfiguration) swapper.swapToObject(each)).collect(Collectors.toList());
}
}
在swapToRuleConfigurations(Collection<YamlRuleConfiguration> yamlRuleConfigs)方法中,执行如下:
1)通过YamlRuleConfiguration.getRuleConfigurationType()获取配置的规则类型。如数据分片规则YamlShardingRuleConfiguration,对应的类型为ShardingRuleConfiguration。添加到集合中;
2)通过Java SPI,获取能够转换对应规则类型的转换器。如数据分片规则的转换器为YamlShardingRuleConfigurationSwapper,该类的getTypeClass()方法返回ShardingRuleConfiguration.class;
Java SPI详见:【源码】Sharding-JDBC源码分析之SPI机制-CSDN博客
3)遍历规则类型转换器,从YamlRuleConfiguration集合中通过getRuleConfigurationType()查找匹配的YamlRuleConfiguration,执行转换器swapper.swapToObject(YamlRuleConfiguration),将YamlRuleConfiguration转换为RuleConfiguration;
3.3 YamlShardingRuleConfigurationSwapper
ShardingSphere有十几种规则,以下以数据分片规则为例,分析YamlRuleConfiguration通过Swapper转换为RuleConfiguration的过程。数据分片规则的转换器为YamlShardingRuleConfigurationSwapper,源码如下:
package org.apache.shardingsphere.sharding.yaml.swapper;
/**
* yaml中sharding规则配置转换器
*/
public final class YamlShardingRuleConfigurationSwapper implements YamlRuleConfigurationSwapper<YamlShardingRuleConfiguration, ShardingRuleConfiguration> {
// Yaml中分片表规则配置转换器
private final YamlShardingTableRuleConfigurationSwapper tableSwapper = new YamlShardingTableRuleConfigurationSwapper();
// yaml分片算法配置转换器
private final YamlShardingStrategyConfigurationSwapper shardingStrategySwapper = new YamlShardingStrategyConfigurationSwapper();
// yaml主键生成器算法配置转换器
private final YamlKeyGenerateStrategyConfigurationSwapper keyGenerateStrategySwapper = new YamlKeyGenerateStrategyConfigurationSwapper();
// yaml中算法配置转换器
private final YamlAlgorithmConfigurationSwapper algorithmSwapper = new YamlAlgorithmConfigurationSwapper();
// yaml分片审计算法配置转换器
private final YamlShardingAuditStrategyConfigurationSwapper auditStrategySwapper = new YamlShardingAuditStrategyConfigurationSwapper();
// yaml中分片autoTable规则配置转换器
private final YamlShardingAutoTableRuleConfigurationSwapper autoTableYamlSwapper = new YamlShardingAutoTableRuleConfigurationSwapper();
/**
* 将ShardingRuleConfiguration转换为YamlShardingRuleConfiguration
*/
@Override
public YamlShardingRuleConfiguration swapToYamlConfiguration(final ShardingRuleConfiguration data) {
YamlShardingRuleConfiguration result = new YamlShardingRuleConfiguration();
// ShardingTableRuleConfiguration转换为YamlTableRuleConfiguration
data.getTables().forEach(each -> result.getTables().put(each.getLogicTable(), tableSwapper.swapToYamlConfiguration(each)));
// ShardingAutoTableRuleConfiguration转换为YamlShardingAutoTableRuleConfiguration
data.getAutoTables().forEach(each -> result.getAutoTables().put(each.getLogicTable(), autoTableYamlSwapper.swapToYamlConfiguration(each)));
// 添加关联表
result.getBindingTables().addAll(data.getBindingTableGroups().stream().map(YamlShardingTableReferenceRuleConfigurationConverter::convertToYamlString).collect(Collectors.toList()));
// 添加广播表
result.getBroadcastTables().addAll(data.getBroadcastTables());
// 默认策略转换
setYamlStrategies(data, result);
// 算法转换(分片算法、主键生成算法、审计员生成算法)
setYamlAlgorithms(data, result);
// 设置默认分片键
result.setDefaultShardingColumn(data.getDefaultShardingColumn());
return result;
}
/**
* 默认策略转换。通过对应策略的转换器,将ShardingStrategyConfiguration转换为YamlShardingStrategyConfiguration
*/
private void setYamlStrategies(final ShardingRuleConfiguration data, final YamlShardingRuleConfiguration yamlConfig) {
if (null != data.getDefaultDatabaseShardingStrategy()) {
yamlConfig.setDefaultDatabaseStrategy(shardingStrategySwapper.swapToYamlConfiguration(data.getDefaultDatabaseShardingStrategy()));
}
if (null != data.getDefaultTableShardingStrategy()) {
yamlConfig.setDefaultTableStrategy(shardingStrategySwapper.swapToYamlConfiguration(data.getDefaultTableShardingStrategy()));
}
if (null != data.getDefaultKeyGenerateStrategy()) {
yamlConfig.setDefaultKeyGenerateStrategy(keyGenerateStrategySwapper.swapToYamlConfiguration(data.getDefaultKeyGenerateStrategy()));
}
if (null != data.getDefaultAuditStrategy()) {
yamlConfig.setDefaultAuditStrategy(auditStrategySwapper.swapToYamlConfiguration(data.getDefaultAuditStrategy()));
}
}
/**
* 算法深度转换
*/
private void setYamlAlgorithms(final ShardingRuleConfiguration data, final YamlShardingRuleConfiguration yamlConfig) {
if (null != data.getShardingAlgorithms()) {
data.getShardingAlgorithms().forEach((key, value) -> yamlConfig.getShardingAlgorithms().put(key, algorithmSwapper.swapToYamlConfiguration(value)));
}
if (null != data.getKeyGenerators()) {
data.getKeyGenerators().forEach((key, value) -> yamlConfig.getKeyGenerators().put(key, algorithmSwapper.swapToYamlConfiguration(value)));
}
if (null != data.getAuditors()) {
data.getAuditors().forEach((key, value) -> yamlConfig.getAuditors().put(key, algorithmSwapper.swapToYamlConfiguration(value)));
}
}
/**
* 将YamlShardingRuleConfiguration转换为ShardingRuleConfiguration
*/
@Override
public ShardingRuleConfiguration swapToObject(final YamlShardingRuleConfiguration yamlConfig) {
ShardingRuleConfiguration result = new ShardingRuleConfiguration();
// 分表转换
for (Entry<String, YamlTableRuleConfiguration> entry : yamlConfig.getTables().entrySet()) {
YamlTableRuleConfiguration tableRuleConfig = entry.getValue();
tableRuleConfig.setLogicTable(entry.getKey());
result.getTables().add(tableSwapper.swapToObject(tableRuleConfig));
}
// 自动表转换
for (Entry<String, YamlShardingAutoTableRuleConfiguration> entry : yamlConfig.getAutoTables().entrySet()) {
YamlShardingAutoTableRuleConfiguration tableRuleConfig = entry.getValue();
tableRuleConfig.setLogicTable(entry.getKey());
result.getAutoTables().add(autoTableYamlSwapper.swapToObject(tableRuleConfig));
}
result.getBindingTableGroups().addAll(yamlConfig.getBindingTables().stream().map(YamlShardingTableReferenceRuleConfigurationConverter::convertToObject).collect(Collectors.toList()));
result.getBroadcastTables().addAll(yamlConfig.getBroadcastTables());
setStrategies(yamlConfig, result);
setAlgorithms(yamlConfig, result);
result.setDefaultShardingColumn(yamlConfig.getDefaultShardingColumn());
return result;
}
/**
* 默认策略转换
*/
private void setStrategies(final YamlShardingRuleConfiguration yamlConfig, final ShardingRuleConfiguration ruleConfig) {
if (null != yamlConfig.getDefaultDatabaseStrategy()) {
ruleConfig.setDefaultDatabaseShardingStrategy(shardingStrategySwapper.swapToObject(yamlConfig.getDefaultDatabaseStrategy()));
}
if (null != yamlConfig.getDefaultTableStrategy()) {
ruleConfig.setDefaultTableShardingStrategy(shardingStrategySwapper.swapToObject(yamlConfig.getDefaultTableStrategy()));
}
if (null != yamlConfig.getDefaultKeyGenerateStrategy()) {
ruleConfig.setDefaultKeyGenerateStrategy(keyGenerateStrategySwapper.swapToObject(yamlConfig.getDefaultKeyGenerateStrategy()));
}
if (null != yamlConfig.getDefaultAuditStrategy()) {
ruleConfig.setDefaultAuditStrategy(auditStrategySwapper.swapToObject(yamlConfig.getDefaultAuditStrategy()));
}
}
/**
* 算法转换。分片算法、主键生成算法、审计员生成算法)
*/
private void setAlgorithms(final YamlShardingRuleConfiguration yamlConfig, final ShardingRuleConfiguration ruleConfig) {
if (null != yamlConfig.getShardingAlgorithms()) {
yamlConfig.getShardingAlgorithms().forEach((key, value) -> ruleConfig.getShardingAlgorithms().put(key, algorithmSwapper.swapToObject(value)));
}
if (null != yamlConfig.getKeyGenerators()) {
yamlConfig.getKeyGenerators().forEach((key, value) -> ruleConfig.getKeyGenerators().put(key, algorithmSwapper.swapToObject(value)));
}
if (null != yamlConfig.getAuditors()) {
yamlConfig.getAuditors().forEach((key, value) -> ruleConfig.getAuditors().put(key, algorithmSwapper.swapToObject(value)));
}
}
@Override
public Class<ShardingRuleConfiguration> getTypeClass() {
return ShardingRuleConfiguration.class;
}
@Override
public String getRuleTagName() {
return "SHARDING";
}
@Override
public int getOrder() {
return ShardingOrder.ORDER;
}
}
如果YamlXxxConfiguration类中有其他YamlYyyConfiguration属性对象,在对应的Swapper转换器中,会定义能够转换YamlYyyConfiguration的转换器,逐层将YamlXxxConfiguration转换为XxxConfiguration对象。
如上面的YamlShardingRuleConfiguration,该规则配置包含分表配置、分片策略、分片算法等,对应的类型为YamlTableRuleConfiguration、YamlShardingStrategyConfiguration、YamlAlgorithmConfiguration。
在YamlShardingRuleConfigurationSwapper中进行YamlShardingRuleConfiguration和ShardingRuleConfiguration的互转时,会调用YamlShardingTableRuleConfigurationSwapper,进行YamlTableRuleConfiguration和ShardingTableRuleConfiguration的互转。
mode配置转换
在上面的3.1中YamlShardingSphereDataSourceFactory的createDataSource()方法中,如果配置了mode,则通过YamlModeConfigurationSwapper.swapToObject()方法,将YamlModeConfiguration转换为ModeConfiguration。
4.1 YamlModeConfigurationSwapper
YamlModeConfigurationSwapper的源码如下:
package org.apache.shardingsphere.infra.yaml.config.swapper.mode;
/**
* Yaml中的mode配置转换器。YamlModeConfiguration和ModeConfiguration的复转
*/
@SuppressWarnings("unchecked")
public final class YamlModeConfigurationSwapper implements YamlConfigurationSwapper<YamlModeConfiguration, ModeConfiguration> {
/**
* ModeConfiguration转为YamlModeConfiguration
*/
@Override
public YamlModeConfiguration swapToYamlConfiguration(final ModeConfiguration data) {
YamlModeConfiguration result = new YamlModeConfiguration();
result.setType(data.getType());
// 持久化转换
if (null != data.getRepository()) {
// 通过Java SPI获取对应配置的type的YamlPersistRepositoryConfigurationSwapper对象。
// 如Standalone为StandaloneYamlPersistRepositoryConfigurationSwapper
// Cluster为ClusterYamlPersistRepositoryConfigurationSwapper
YamlPersistRepositoryConfigurationSwapper<PersistRepositoryConfiguration> swapper = TypedSPIRegistry.getRegisteredService(
YamlPersistRepositoryConfigurationSwapper.class, data.getType());
result.setRepository(swapper.swapToYamlConfiguration(data.getRepository()));
}
return result;
}
/**
* YamlModeConfiguration转换为ModeConfiguration
*/
@Override
public ModeConfiguration swapToObject(final YamlModeConfiguration yamlConfig) {
if (null == yamlConfig.getRepository()) {
return new ModeConfiguration(yamlConfig.getType(), null);
}
// 根据配置的mode的类型,找到对应的持久化存储转换器,将YamlPersistRepositoryConfiguration转换为XxxPersistRepositoryConfiguration
YamlPersistRepositoryConfigurationSwapper<PersistRepositoryConfiguration> swapper = TypedSPIRegistry.getRegisteredService(
YamlPersistRepositoryConfigurationSwapper.class, yamlConfig.getType());
return new ModeConfiguration(yamlConfig.getType(), swapper.swapToObject(yamlConfig.getRepository()));
}
}
如配置的type为Standalone,则对应的持久化存储转换器为StandaloneYamlPersistRepositoryConfigurationSwapper。
4.2 StandaloneYamlPersistRepositoryConfigurationSwapper
StandaloneYamlPersistRepositoryConfigurationSwapper源码如下:
package org.apache.shardingsphere.mode.manager.standalone.yaml;
/**
* mode类型为Standalone的持久化配置转换器
*/
public final class StandaloneYamlPersistRepositoryConfigurationSwapper implements YamlPersistRepositoryConfigurationSwapper<StandalonePersistRepositoryConfiguration> {
@Override
public YamlPersistRepositoryConfiguration swapToYamlConfiguration(final StandalonePersistRepositoryConfiguration data) {
YamlPersistRepositoryConfiguration result = new YamlPersistRepositoryConfiguration();
result.setType(data.getType());
result.setProps(data.getProps());
return result;
}
@Override
public StandalonePersistRepositoryConfiguration swapToObject(final YamlPersistRepositoryConfiguration yamlConfig) {
return new StandalonePersistRepositoryConfiguration(yamlConfig.getType(), yamlConfig.getProps());
}
@Override
public String getType() {
return "Standalone";
}
}
小结
以上为本篇分析的全部内容,以下做一个小结:
1)Yaml中最核心的配置信息rules和mode,解析后的对象的类名为YamlXxxConfiguration,且对应一个XxxConfiguration类,通过对应的YamlXxxConfigurationSwapper进行互转;
2)dataSources配置,通过YamlDataSourceConfigurationSwapper转换器,采用反射机制,创建对应的DataSource对象;
3)rules配置,通过YamlRuleConfigurationSwapperEngine,进行YamlRuleConfiguration集合和RuleConfiguration集合的互转;
4)mode配置信息通过YamlModeConfigurationSwapper,将YamlModeConfiguraton转换为ModeConfiguration对象;
关于本篇内容你有什么自己的想法或独到见解,欢迎在评论区一起交流探讨下吧。