Spring Boot入門教程(四十四): Sharding-JDBC+JPA|MyBatis+Druid分庫分表實現
一:資料庫分片方案
客戶端代理: 分片邏輯在應用端,封裝在jar包中,通過修改或者封裝JDBC層來實現。 噹噹網的 Sharding-JDBC 、阿里的TDDL是兩種比較常用的實現。
中介軟體代理: 在應用和資料中間加了一個代理層。分片邏輯統一維護在中介軟體服務中。 我們現在談的 Mycat、360的Atlas、網易的DDB等等都是這種架構的實現
二:Sharding-JDBC
Sharding-JDBC是一個開源的適用於微服務的分散式資料訪問基礎類庫,它始終以雲原生的基礎開發套件為目標。
Sharding-JDBC定位為輕量級java框架,使用客戶端直連資料庫,以jar包形式提供服務,未使用中間層,無需額外部署,無其他依賴,DBA也無需改變原有的運維方式,可理解為增強版的JDBC驅動,舊程式碼遷移成本幾乎為零。
Sharding-JDBC完整的實現了分庫分表,讀寫分離和分散式主鍵功能,並初步實現了柔性事務。從2016年開源至今,在經歷了整體架構的數次精煉以及穩定性打磨後,如今它已積累了足夠的底蘊,相信可以成為開發者選擇技術元件時的一個參考。
分庫分表
- SQL解析功能完善,支援聚合,分組,排序,LIMIT,TOP等查詢,並且支援級聯表以及笛卡爾積的表查詢
- 支援內、外連線查詢
- 分片策略靈活,可支援=,BETWEEN,IN等多維度分片,也可支援多分片鍵共用,以及自定義分片策略
- 基於Hint的強制分庫分表路由
讀寫分離
- 一主多從的讀寫分離配置,可配合分庫分表使用
- 基於Hint的強制主庫路由
柔性事務
- 最大努力送達型事務
- TCC型事務(TBD)
分散式主鍵
- 統一的分散式基於時間序列的ID生成器
相容性
- 可適用於任何基於java的ORM框架,如:JPA, Hibernate, Mybatis, Spring JDBC Template或直接使用JDBC
- 可基於任何第三方的資料庫連線池,如:DBCP, C3P0, BoneCP, Druid等
- 理論上可支援任意實現JDBC規範的資料庫。目前支援MySQL,Oracle,SQLServer和PostgreSQL
靈活多樣的配置
- Java
- YAML
- Inline表示式
- Spring名稱空間
- Spring boot starter
分散式治理能力 (2.0新功能)
- 配置集中化與動態化,可支援資料來源、表與分片策略的動態切換(2.0.0.M1)
- 客戶端的資料庫治理,資料來源失效自動切換(2.0.0.M2)
- 基於Open Tracing協議的APM資訊輸出(2.0.0.M3)
架構圖
三:sharding-jdbc + jpa + druid整合
0. 資料庫
-- 在db0資料庫上分別建立t_order_0、t_order_1表
USE db0;
DROP TABLE IF EXISTS t_order_0;
CREATE TABLE t_order_0 (
order_id bigint(20) NOT NULL,
user_id bigint(20) NOT NULL,
PRIMARY KEY (order_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
DROP TABLE IF EXISTS t_order_1;
CREATE TABLE t_order_1 (
order_id bigint(20) NOT NULL,
user_id bigint(20) NOT NULL,
PRIMARY KEY (order_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
-- 在db1資料庫上分別建立t_order_0、t_order_1表
USE db1;
DROP TABLE IF EXISTS t_order_0;
CREATE TABLE t_order_0 (
order_id bigint(20) NOT NULL,
user_id bigint(20) NOT NULL,
PRIMARY KEY (order_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
DROP TABLE IF EXISTS t_order_1;
CREATE TABLE t_order_1 (
order_id bigint(20) NOT NULL,
user_id bigint(20) NOT NULL,
PRIMARY KEY (order_id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
1. 引入依賴
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.company</groupId>
<artifactId>sharding-jdbc</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>sharding-jdbc</name>
<description>Demo project for Spring Boot</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.3.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.41</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>druid</artifactId>
<version>1.1.10</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>com.dangdang</groupId>
<artifactId>sharding-jdbc-core</artifactId>
<version>1.5.4</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
注意mysql-connector-java的版本不要太高了
2. application.yml
spring:
jpa:
database: mysql
show-sql: true
hibernate:
ddl-auto: none
注意:hibernate.ddl-auto=none 是因為分表就會有多個表,例如t_order_0、t_order_1等,而ORM只能對映成一個,所以關閉自動的ddl語句。
3. domain
@Entity
@Table(name = "t_order")
@Data
public class Order {
@Id
private Long orderId;
private Long userId;
}
注意:orderId上使用@Id註解並沒有使用@GeneratedValue(strategy = GenerationType.AUTO)的主鍵生成策略,原因是分表必須要保證所有表的主鍵id不重複,如果使用mysql的自動生成,那麼id就會重複,這裡的id一般要使用分散式主鍵id來通過程式碼來生成。
4. Repository
import com.company.shardingjdbc.domain.Order;
import org.springframework.data.repository.CrudRepository;
public interface OrderRepository extends CrudRepository<Order, Long> {
}
5. Controller
import com.company.shardingjdbc.domain.Order;
import com.company.shardingjdbc.repository.OrderRepository;
import com.dangdang.ddframe.rdb.sharding.keygen.KeyGenerator;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
@RestController
@RequestMapping("/order")
public class OrderController {
@Autowired
private OrderRepository orderRepository;
@Autowired
private KeyGenerator keyGenerator;
@RequestMapping("/create")
public Object add() {
for (int i = 0; i < 10; i++) {
Order order = new Order();
order.setUserId((long) i);
order.setOrderId((long) i);
orderRepository.save(order);
}
for (int i = 10; i < 20; i++) {
Order order = new Order();
order.setUserId((long) i + 1);
order.setOrderId((long) i);
orderRepository.save(order);
}
// for (int i = 0; i < 30; i++) {
// Order order = new Order();
// order.setOrderId(keyGenerator.generateKey().longValue());
// order.setUserId(keyGenerator.generateKey().longValue());
// orderRepository.save(order);
// }
return "success";
}
@RequestMapping("query")
private Object queryAll() {
return orderRepository.findAll();
}
}
6. Configuration
package com.company.shardingjdbc.configuration;
import com.alibaba.druid.pool.DruidDataSource;
import com.company.shardingjdbc.common.ModuleDatabaseShardingAlgorithm;
import com.company.shardingjdbc.common.ModuleTableShardingAlgorithm;
import com.dangdang.ddframe.rdb.sharding.api.ShardingDataSourceFactory;
import com.dangdang.ddframe.rdb.sharding.api.rule.DataSourceRule;
import com.dangdang.ddframe.rdb.sharding.api.rule.ShardingRule;
import com.dangdang.ddframe.rdb.sharding.api.rule.TableRule;
import com.dangdang.ddframe.rdb.sharding.api.strategy.database.DatabaseShardingStrategy;
import com.dangdang.ddframe.rdb.sharding.api.strategy.table.TableShardingStrategy;
import com.dangdang.ddframe.rdb.sharding.keygen.DefaultKeyGenerator;
import com.dangdang.ddframe.rdb.sharding.keygen.KeyGenerator;
import com.mysql.jdbc.Driver;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import javax.sql.DataSource;
import java.sql.SQLException;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
@Configuration
public class DataSourceConfiguration {
@Bean
public DataSource getDataSource() throws SQLException {
return buildDataSource();
}
private DataSource buildDataSource() throws SQLException {
// 設定分庫對映
Map<String, DataSource> dataSourceMap = new HashMap<>(2);
// 新增兩個資料庫db0,db1到map裡
dataSourceMap.put("db0", createDataSource("db0"));
dataSourceMap.put("db1", createDataSource("db1"));
// 設定預設db為db0,也就是為那些沒有配置分庫分表策略的指定的預設庫
// 如果只有一個庫,也就是不需要分庫的話,map裡只放一個對映就行了,只有一個庫時不需要指定預設庫,但2個及以上時必須指定預設庫,否則那些沒有配置策略的表將無法操作資料
DataSourceRule dataSourceRule = new DataSourceRule(dataSourceMap, "db0");
// 設定分表對映,將t_order_0和t_order_1兩個實際的表對映到t_order邏輯表
// 0和1兩個表是真實的表,t_order是個虛擬不存在的表,只是供使用。如查詢所有資料就是select * from t_order就能查完0和1表的
TableRule orderTableRule = TableRule.builder("t_order")
.actualTables(Arrays.asList("t_order_0", "t_order_1"))
.dataSourceRule(dataSourceRule)
.build();
// 具體分庫分表策略,按什麼規則來分
ShardingRule shardingRule = ShardingRule.builder()
.dataSourceRule(dataSourceRule)
.tableRules(Arrays.asList(orderTableRule))
.databaseShardingStrategy(new DatabaseShardingStrategy("user_id", new ModuleDatabaseShardingAlgorithm()))
.tableShardingStrategy(new TableShardingStrategy("order_id", new ModuleTableShardingAlgorithm())).build();
DataSource dataSource = ShardingDataSourceFactory.createDataSource(shardingRule);
return dataSource;
}
private static DataSource createDataSource(final String dataSourceName) {
// 使用druid連線資料庫
DruidDataSource result = new DruidDataSource();
result.setDriverClassName(Driver.class.getName());
result.setUrl(String.format("jdbc:mysql://localhost:3306/%s", dataSourceName));
result.setUsername("root");
result.setPassword("root123");
return result;
}
@Bean
public KeyGenerator keyGenerator() {
return new DefaultKeyGenerator();
}
}
ModuleDatabaseShardingAlgorithm
package com.company.shardingjdbc.common;
import com.dangdang.ddframe.rdb.sharding.api.ShardingValue;
import com.dangdang.ddframe.rdb.sharding.api.strategy.database.SingleKeyDatabaseShardingAlgorithm;
import com.google.common.collect.Range;
import java.util.Collection;
import java.util.LinkedHashSet;
/**
* 單鍵資料庫分片演算法.
*
* 支援單鍵和多鍵策略
* <ul>
* <li>單鍵 SingleKeyDatabaseShardingAlgorithm</li>
* <li>多鍵 MultipleKeysDatabaseShardingAlgorithm</li>
* </ul>
*
* 支援的分片策略
* <ul>
* <li> = doEqualSharding 例如 where order_id = 1 </li>
* <li> IN doInSharding 例如 where order_id in (1, 2)</li>
* <li> BETWEEN doBetweenSharding 例如 where order_id between 1 and 2 </li>
* </ul>
*
* @author mengday
*/
public class ModuleDatabaseShardingAlgorithm implements SingleKeyDatabaseShardingAlgorithm<Long> {
/**
* 分片策略 相等=
* @param availableTargetNames 可用的目標名字(這裡指資料名db0、db1)
* @param shardingValue 分片值[logicTableName="t_order" 邏輯表名, columnName="user_id" 分片的列名, value="20" 分片的列名對應的值(user_id=20)]
* @return
*/
@Override
public String doEqualSharding(Collection<String> availableTargetNames, ShardingValue<Long> shardingValue) {
for (String each : availableTargetNames) {
if (each.endsWith(shardingValue.getValue() % 2 + "")) {
return each;
}
}
throw new IllegalArgumentException();
}
@Override
public Collection<String> doInSharding(Collection<String> availableTargetNames, ShardingValue<Long> shardingValue) {
Collection<String> result = new LinkedHashSet<>(availableTargetNames.size());
for (Long value : shardingValue.getValues()) {
for (String tableName : availableTargetNames) {
if (tableName.endsWith(value % 2 + "")) {
result.add(tableName);
}
}
}
return result;
}
@Override
public Collection<String> doBetweenSharding(Collection<String> availableTargetNames,
ShardingValue<Long> shardingValue) {
Collection<String> result = new LinkedHashSet<>(availableTargetNames.size());
Range<Long> range = shardingValue.getValueRange();
for (Long i = range.lowerEndpoint(); i <= range.upperEndpoint(); i++) {
for (String each : availableTargetNames) {
if (each.endsWith(i % 2 + "")) {
result.add(each);
}
}
}
return result;
}
}
package com.company.shardingjdbc.common;
import com.dangdang.ddframe.rdb.sharding.api.ShardingValue;
import com.dangdang.ddframe.rdb.sharding.api.strategy.table.SingleKeyTableShardingAlgorithm;
import com.google.common.collect.Range;
import java.util.Collection;
import java.util.LinkedHashSet;
public final class ModuleTableShardingAlgorithm implements SingleKeyTableShardingAlgorithm<Long> {
/**
* doEqualSharding =
* @param tableNames 實際物理表名
* @param shardingValue [logicTableName="t_order", columnName="order_id", value=20]
*
* select * from t_order from t_order where order_id = 11
* └── SELECT * FROM t_order_1 WHERE order_id = 11
* select * from t_order from t_order where order_id = 44
* └── SELECT * FROM t_order_0 WHERE order_id = 44
*/
* select * from t_order from t_order where order_id = 11
* └── SELECT * FROM t_order_1 WHERE order_id = 11
* select * from t_order from t_order where order_id = 44
* └── SELECT * FROM t_order_0 WHERE order_id = 44
*/
public String doEqualSharding(final Collection<String> tableNames, final ShardingValue<Long> shardingValue) {
for (String each : tableNames) {
if (each.endsWith(shardingValue.getValue() % 2 + "")) {
return each;
}
}
throw new IllegalArgumentException();
}
/**
* select * from t_order from t_order where order_id in (11,44)
* ├── SELECT * FROM t_order_0 WHERE order_id IN (11,44)
* └── SELECT * FROM t_order_1 WHERE order_id IN (11,44)
* select * from t_order from t_order where order_id in (11,13,15)
* └── SELECT * FROM t_order_1 WHERE order_id IN (11,13,15)
* select * from t_order from t_order where order_id in (22,24,26)
* └──SELECT * FROM t_order_0 WHERE order_id IN (22,24,26)
*/
public Collection<String> doInSharding(final Collection<String> tableNames, final ShardingValue<Long> shardingValue) {
Collection<String> result = new LinkedHashSet<>(tableNames.size());
for (Long value : shardingValue.getValues()) {
for (String tableName : tableNames) {
if (tableName.endsWith(value % 2 + "")) {
result.add(tableName);
}
}
}
return result;
}
/**
* select * from t_order from t_order where order_id between 10 and 20
* ├── SELECT * FROM t_order_0 WHERE order_id BETWEEN 10 AND 20
* └── SELECT * FROM t_order_1 WHERE order_id BETWEEN 10 AND 20
*/
public Collection<String> doBetweenSharding(final Collection<String> tableNames, final ShardingValue<Long> shardingValue) {
Collection<String> result = new LinkedHashSet<>(tableNames.size());
Range<Long> range = shardingValue.getValueRange();
for (Long i = range.lowerEndpoint(); i <= range.upperEndpoint(); i++) {
for (String each : tableNames) {
if (each.endsWith(i % 2 + "")) {
result.add(each);
}
}
}
return result;
}
}
7. localhost:8080/order/create
db0
├── t_order_0 user_id為偶數 order_id為偶數
├── t_order_1 user_id為偶數 order_id為奇數
db1
├── t_order_0 user_id為奇數 order_id為偶數
├── t_order_1 user_id為奇數 order_id為奇數
四:sharding-jdbc + mybatis + druid整合
此示例是在jap原有的整合上整合mybatis
1. 引入mybatis依賴
<dependency>
<groupId>org.mybatis.spring.boot</groupId>
<artifactId>mybatis-spring-boot-starter</artifactId>
<version>1.3.2</version>
</dependency>
2. 在Application上添加註解@MapperScan
@MapperScan("com.company.shardingjdbc.mapper")
@SpringBootApplication
public class ShardingJdbcApplication {
public static void main(String[] args) {
SpringApplication.run(ShardingJdbcApplication.class, args);
}
}
3. application.yml
# Mybatis 配置
mybatis:
typeAliasesPackage: com.company.shardingjdbc.domain
mapperLocations: classpath:mapper/*.xml
configuration.map-underscore-to-camel-case: true
# 列印mybatis中的sql語句和結果集
logging:
level.com.company.shardingjdbc.mapper: TRACE
4. OrderMapper
import org.apache.ibatis.annotations.Param;
import java.util.List;
public interface OrderMapper {
void insert(Order order);
List<Order> queryById(@Param("orderIdList") List<Long> orderIdList);
}
5. OrderMapper.xml
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd" >
<mapper namespace="com.company.shardingjdbc.mapper.OrderMapper" >
<select id="queryById" parameterType="Long" resultType="Order">
SELECT * FROM t_order WHERE order_id IN
<foreach collection="orderIdList" item="orderId" open="(" separator="," close=")">
#{orderId}
</foreach>
</select>
<insert id="insert" parameterType="Order">
INSERT INTO t_order (order_id, user_id) VALUES (#{orderId}, #{userId})
</insert>
</mapper>
6. OrderController
import com.dangdang.ddframe.rdb.sharding.keygen.KeyGenerator;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
@RestController
@RequestMapping("/order")
public class OrderController {
@Autowired
private OrderMapper orderMapper;
@RequestMapping("/insert")
public Object insert() {
for (int i = 20; i < 30; i++) {
Order order = new Order();
order.setUserId((long) i);
order.setOrderId((long) i);
orderMapper.insert(order);
}
for (int i = 30; i < 40; i++) {
Order order = new Order();
order.setUserId((long) i + 1);
order.setOrderId((long) i);
orderMapper.insert(order);
}
return "success";
}
@RequestMapping("queryById")
public List<Order> queryById(String orderIds) {
List<String> strings = Arrays.asList(orderIds.split(","));
List<Long> orderIdList = strings.stream().map(item -> Long.parseLong(item)).collect(Collectors.toList());
return orderMapper.queryById(orderIdList);
}
}
7. 插入資料
localhost:8080/order/insert
- ModuleDatabaseShardingAlgorithm: 先根據分片鍵user_id及值來確定要操作的資料庫是db0還是db1
- ModuleTableShardingAlgorithm: 再根據分片鍵order_id及值來確定要操作的資料庫對應的表是t_order_0還是t_order_1
- 當資料庫名和表名都確定了就可以操作資料庫了
localhost:8080/order/queryById?orderIds=20,31,30,21