1、概述
当单表的记录条数达到千万级别后,就要考虑使用分库分表技术框架了。本文介绍当当分库分表技术框架Sharding-JDBC,主页http://shardingjdbc.io。下载1.5.4.1源码,https://github.com/shardingjdbc/sharding-jdbc/releases。
2、开发环境
win7、JDK1.7、IntelliJ IDEA 2017.2.5。
3、sharding-jdbc-example-mybatis
File-Open...,选择sharding-jdbc-1.5.4.1下的sharding-jdbc-example,先从sharding-jdbc-example-mybatis项目开始,如图01。
3.1 准备工作
修改shardingContext.xml里访问MySQL的用户名和密码,我的密码是:123456。
<bean id="ds_0" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url" value="jdbc:mysql://localhost:3306/ds_0"/>
<property name="username" value="root"/>
<property name="password" value="123456"/>
</bean>
<bean id="ds_1" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url" value="jdbc:mysql://localhost:3306/ds_1"/>
<property name="username" value="root"/>
<property name="password" value="123456"/>
</bean>
根据mysql.sql,在ds_0和ds_1里分别建表:t_order_0、t_order_1、t_order_item_0、t_order_item_1。本例不涉及t_order_item_0、t_order_item_1。
要认真看看shardingContext.xml里的内容,并和传统Web应用里的数据源配置做下对比。通俗地讲,id为"transactionManager"的bean引用了数据源"shardingDataSource",而后者根据某种算法,依据user_id的值选择真正的数据源"ds_0"或"ds_1"。之后,再根据某种算法,依据order_id的值选择插入到t_order_0或t_order_1。
上面提到的两种算法就是shardingContext.xml里的SingleKeyModuloDatabaseShardingAlgorithm.java(以下简称01类)和SingleKeyModuloTableShardingAlgorithm.java(以下简称02类)。对上述两类中的三个核心方法添加方法参数打印语句,先感性认识再理性理解!
public final class SingleKeyModuloDatabaseShardingAlgorithm implements SingleKeyDatabaseShardingAlgorithm<Integer> {
@Override
public String doEqualSharding(final Collection<String> availableTargetNames, final ShardingValue<Integer> shardingValue) {
System.out.println("###### 01 doEqualSharding(),availableTargetNames=" + availableTargetNames + ",shardingValue=" + shardingValue);
for (String each : availableTargetNames) {
if (each.endsWith(shardingValue.getValue() % 2 + "")) {
System.out.println("###### 01 doEqualSharding(),命中=" + each);
return each;
}
}
throw new UnsupportedOperationException();
}
@Override
public Collection<String> doInSharding(final Collection<String> availableTargetNames, final ShardingValue<Integer> shardingValue) {
System.out.println("###### 01 doInSharding(),availableTargetNames=" + availableTargetNames + ",shardingValue=" + shardingValue);
Collection<String> result = new LinkedHashSet<>(availableTargetNames.size());
Collection<Integer> values = shardingValue.getValues();
for (Integer value : values) {
for (String each : availableTargetNames) {
if (each.endsWith(value % 2 + "")) {
result.add(each);
}
}
}
System.out.println("###### 01 doInSharding(),命中=" + result);
return result;
}
@Override
public Collection<String> doBetweenSharding(final Collection<String> availableTargetNames, final ShardingValue<Integer> shardingValue) {
System.out.println("###### 01 doBetweenSharding(),availableTargetNames=" + availableTargetNames + ",shardingValue=" + shardingValue);
Collection<String> result = new LinkedHashSet<>(availableTargetNames.size());
Range<Integer> range = shardingValue.getValueRange();
for (Integer value = range.lowerEndpoint(); value <= range.upperEndpoint(); value++) {
for (String each : availableTargetNames) {
if (each.endsWith(value % 2 + "")) {
result.add(each);
}
}
}
System.out.println("###### 01 doBetweenSharding(),命中=" + result);
return result;
}
}
public final class SingleKeyModuloTableShardingAlgorithm implements SingleKeyTableShardingAlgorithm<Long> {
@Override
public String doEqualSharding(final Collection<String> availableTargetNames, final ShardingValue<Long> shardingValue) {
System.out.println("###### 02 doEqualSharding(),availableTargetNames=" + availableTargetNames + ",shardingValue=" + shardingValue);
for (String each : availableTargetNames) {
if (each.endsWith(shardingValue.getValue() % 2 + "")) {
System.out.println("###### 02 doEqualSharding(),命中=" + each);
return each;
}
}
throw new UnsupportedOperationException();
}
@Override
public Collection<String> doInSharding(final Collection<String> availableTargetNames, final ShardingValue<Long> shardingValue) {
System.out.println("###### 02 doInSharding(),availableTargetNames=" + availableTargetNames + ",shardingValue=" + shardingValue);
Collection<String> result = new LinkedHashSet<>(availableTargetNames.size());
Collection<Long> values = shardingValue.getValues();
for (Long value : values) {
for (String each : availableTargetNames) {
if (each.endsWith(value % 2 + "")) {
result.add(each);
}
}
}
System.out.println("###### 02 doInSharding(),命中=" + result);
return result;
}
@Override
public Collection<String> doBetweenSharding(final Collection<String> availableTargetNames, final ShardingValue<Long> shardingValue) {
System.out.println("###### 02 doBetweenSharding(),availableTargetNames=" + availableTargetNames + ",shardingValue=" + shardingValue);
Collection<String> result = new LinkedHashSet<>(availableTargetNames.size());
Range<Long> range = shardingValue.getValueRange();
for (Long i = range.lowerEndpoint(); i <= range.upperEndpoint(); i++) {
for (String each : availableTargetNames) {
if (each.endsWith(i % 2 + "")) {
result.add(each);
}
}
}
System.out.println("###### 02 doBetweenSharding(),命中=" + result);
return result;
}
}
3.2 执行Main类中的main()方法
分析Dao操作与对应的日志输出。
orderService.clear();不涉及上述两个算法。
orderService.fooService();
###### 01 doEqualSharding(),availableTargetNames=[ds_0, ds_1],shardingValue=ShardingValue(logicTableName=t_order, columnName=user_id, value=10, values=[], valueRange=null)
###### 01 doEqualSharding(),命中=ds_0
###### 02 doEqualSharding(),availableTargetNames=[t_order_1, t_order_0],shardingValue=ShardingValue(logicTableName=t_order, columnName=order_id, value=149948054085042176, values=[], valueRange=null)
###### 02 doEqualSharding(),命中=t_order_0
Generated key1 of order_id:149948054085042176
###### 01 doEqualSharding(),availableTargetNames=[ds_0, ds_1],shardingValue=ShardingValue(logicTableName=t_order, columnName=user_id, value=11, values=[], valueRange=null)
###### 01 doEqualSharding(),命中=ds_1
###### 02 doEqualSharding(),availableTargetNames=[t_order_1, t_order_0],shardingValue=ShardingValue(logicTableName=t_order, columnName=order_id, value=149948054353477632, values=[], valueRange=null)
###### 02 doEqualSharding(),命中=t_order_0
Generated key2 of order_id:149948054353477632
###### 01 doInSharding(),availableTargetNames=[ds_0, ds_1],shardingValue=ShardingValue(logicTableName=t_order, columnName=user_id, value=null, values=[10, 11], valueRange=null)
###### 01 doInSharding(),命中=[ds_0, ds_1]
通俗地讲,当执行INSERT INTO t_order (user_id, status) VALUES...时:
1、根据01类的doEqualSharding()方法,命中了数据源ds_0。其中,可供选择的数据源名称为:ds_0, ds_1。逻辑表名为t_order。分库列为user_id,其值为10。
2、根据02类的doEqualSharding()方法,命中了表t_order_0。其中,可供选择的表名称为t_order_1, t_order_0。逻辑表名为t_order。分表列为order_id,其值为149948054085042176。
第二个INSERT与第一个类似。
当执行UPDATE t_order SET status = 'UPDATED' WHERE user_id in(...)时:
1、根据01类的doInSharding()方法(注意:不是doEqualSharding()方法了),命中了数据源ds_0, ds_1。其中,可供选择的数据源名称为:ds_0, ds_1。逻辑表名为t_order。分片列为user_id,其值的集合为[10, 11]。
2、因为不涉及分片列为oder_id,未使用到02类。
orderService.select();不涉及上述两个算法。
orderService.fooServiceWithFailure();等同于orderService.fooService();
逻辑表,是相对于物理表而言的。因为Sharding-JDBC的存在,站在MyBatis映射文件的角度来说,它根本不知道有ds_0、ds_1、t_order_0、t_order_1等的存在,只知道t_order,仿佛在操作单数据源似的。
3.3 再回头看看shardingContext.xml