既有适合小白学习的零基础资料,也有适合3年以上经验的小伙伴深入学习提升的进阶课程,涵盖了95%以上大数据知识点,真正体系化!
由于文件比较多,这里只是将部分目录截图出来,全套包含大厂面经、学习笔记、源码讲义、实战项目、大纲路线、讲解视频,并且后续会持续更新
docker run -e 'ACCEPT\_EULA=Y' -e 'SA\_PASSWORD=${your\_password}' \
-p 1433:1433 --name sqlserver \
-d mcr.microsoft.com/mssql/server:latest
设置代理sqlagent.enabled
,代理设置完成后,需要重启Sqlserver
,因为我们是docker
安装的,直接用docker restart sqlserver
就行了。
[root@hdp-01 ~]# docker exec -it --user root sqlserver bash
root@0274812d0c10:/# /opt/mssql/bin/mssql-conf set sqlagent.enabled true
SQL Server needs to be restarted in order to apply this setting. Please run
'systemctl restart mssql-server.service'.
root@0274812d0c10:/# exit
exit
[root@hdp-01 ~]# docker restart sqlserver
sqlserver
1.3 启用CDC功能
按照如下步骤执行命令,如果看到is_cdc_enabled = 1
,则说明当前数据库
root@0274812d0c10:/# /opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P "${your\_password}"
1> create databases test;
2> go
1> use test;
2> go
Changed database context to 'test'.
1> EXEC sys.sp_cdc_enable_db;
2> go
1> SELECT is_cdc_enabled FROM sys.databases WHERE name = 'test';
2> go
is_cdc_enabled
--------------
1
(1 rows affected)
1> CREATE TABLE t_info (id int,order_date date,purchaser int,quantity int,product_id int,PRIMARY KEY ([id]))
2> go
1>
2>
3> EXEC sys.sp_cdc_enable_table
4> @source_schema = 'dbo',
5> @source_name = 't\_info',
6> @role_name = 'cdc\_role';
7> go
Update mask evaluation will be disabled in net_changes_function because the CLR configuration option is disabled.
Job 'cdc.zeus\_capture' started successfully.
Job 'cdc.zeus\_cleanup' started successfully.
1> select * from t_info;
2> go
id order_date purchaser quantity product_id
----------- ---------------- ----------- ----------- -----------
(0 rows affected)
1.4 检查CDC是否正常开启
用客户端连接Sqlserver
,查看test
库下的INFORMATION_SCHEMA.TABLES
中是否出现TABLE_SCHEMA = cdc
的表,如果出现,说明已经成功安装Sqlserver
并启用了CDC
。
1> use test;
2> go
Changed database context to 'test'.
1> select * from INFORMATION_SCHEMA.TABLES;
2> go
TABLE_CATALOG TABLE_SCHEMA TABLE_NAME TABLE_TYPE
test dbo user_info BASE TABLE
test dbo systranschemas BASE TABLE
test cdc change_tables BASE TABLE
test cdc ddl_history BASE TABLE
test cdc lsn_time_mapping BASE TABLE
test cdc captured_columns BASE TABLE
test cdc index_columns BASE TABLE
test dbo orders BASE TABLE
test cdc dbo_orders_CT BASE TABLE
二、具体实现
2.1 Flik-CDC采集SqlServer主程序
添加依赖包:
<dependency>
<groupId>com.ververica</groupId>
<artifactId>flink-connector-sqlserver-cdc</artifactId>
<version>3.0.0</version>
</dependency>
编写主函数:
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// 设置全局并行度
env.setParallelism(1);
// 设置时间语义为ProcessingTime
env.getConfig().setAutoWatermarkInterval(0);
// 每隔60s启动一个检查点
env.enableCheckpointing(60000, CheckpointingMode.EXACTLY\_ONCE);
// checkpoint最小间隔
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(1000);
// checkpoint超时时间
env.getCheckpointConfig().setCheckpointTimeout(60000);
// 同一时间只允许一个checkpoint
// env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
// Flink处理程序被cancel后,会保留Checkpoint数据
// env.getCheckpointConfig().setExternalizedCheckpointCleanup(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN\_ON\_CANCELLATION);
SourceFunction<String> sqlServerSource = SqlServerSource.<String>builder()
.hostname("localhost")
.port(1433)
.username("SA")
.password("")
.database("test")
.tableList("dbo.t\_info")
.startupOptions(StartupOptions.initial())
.debeziumProperties(getDebeziumProperties())
.deserializer(new CustomerDeserializationSchemaSqlserver())
.build();
DataStreamSource<String> dataStreamSource = env.addSource(sqlServerSource, "\_transaction\_log\_source");
dataStreamSource.print().setParallelism(1);
env.execute("sqlserver-cdc-test");
}
public static Properties getDebeziumProperties() {
Properties properties = new Properties();
properties.put("converters", "sqlserverDebeziumConverter");
properties.put("sqlserverDebeziumConverter.type", "SqlserverDebeziumConverter");
properties.put("sqlserverDebeziumConverter.database.type", "sqlserver");
// 自定义格式,可选
properties.put("sqlserverDebeziumConverter.format.datetime", "yyyy-MM-dd HH:mm:ss");
properties.put("sqlserverDebeziumConverter.format.date", "yyyy-MM-dd");
properties.put("sqlserverDebeziumConverter.format.time", "HH:mm:ss");
return properties;
}
2.2 自定义Sqlserver
反序列化格式:
Flink-CDC
底层技术为debezium
,它捕获到Sqlserver
数据变更(CRUD)的数据格式如下:
#初始化
Struct{after=Struct{id=1,order_date=2024-01-30,purchaser=1,quantity=100,product_id=1},source=Struct{version=1.9.7.Final,connector=sqlserver,name=sqlserver_transaction_log_source,ts_ms=1706574924473,snapshot=true,db=zeus,schema=dbo,table=orders,commit_lsn=0000002b:00002280:0003},op=r,ts_ms=1706603724432}
#新增
Struct{after=Struct{id=12,order_date=2024-01-11,purchaser=6,quantity=233,product_id=63},source=Struct{version=1.9.7.Final,connector=sqlserver,name=sqlserver_transaction_log_source,ts_ms=1706603786187,db=zeus,schema=dbo,table=orders,change_lsn=0000002b:00002480:0002,commit_lsn=0000002b:00002480:0003,event_serial_no=1},op=c,ts_ms=1706603788461}
#更新
Struct{before=Struct{id=12,order_date=2024-01-11,purchaser=6,quantity=233,product_id=63},after=Struct{id=12,order_date=2024-01-11,purchaser=8,quantity=233,product_id=63},source=Struct{version=1.9.7.Final,connector=sqlserver,name=sqlserver_transaction_log_source,ts_ms=1706603845603,db=zeus,schema=dbo,table=orders,change_lsn=0000002b:00002500:0002,commit_lsn=0000002b:00002500:0003,event_serial_no=2},op=u,ts_ms=1706603850134}
#删除
Struct{before=Struct{id=11,order_date=2024-01-11,purchaser=6,quantity=233,product_id=63},source=Struct{version=1.9.7.Final,connector=sqlserver,name=sqlserver_transaction_log_source,ts_ms=1706603973023,db=zeus,schema=dbo,table=orders,change_lsn=0000002b:000025e8:0002,commit_lsn=0000002b:000025e8:0005,event_serial_no=1},op=d,ts_ms=1706603973859}
因此,可以根据自己需要自定义反序列化格式,将数据按照标准统一数据输出,下面是我自定义的格式,供大家参考:
import com.alibaba.fastjson2.JSON;
import com.alibaba.fastjson2.JSONObject;
import com.alibaba.fastjson2.JSONWriter;
import com.ververica.cdc.debezium.DebeziumDeserializationSchema;
import io.debezium.data.Envelope;
import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.util.Collector;
import org.apache.kafka.connect.data.Field;
import org.apache.kafka.connect.data.Schema;
import org.apache.kafka.connect.data.Struct;
import org.apache.kafka.connect.source.SourceRecord;
import java.util.HashMap;
import java.util.Map;
public class CustomerDeserializationSchemaSqlserver implements DebeziumDeserializationSchema<String> {
private static final long serialVersionUID = -1L;
@Override
public void deserialize(SourceRecord sourceRecord, Collector collector) {
Map<String, Object> resultMap = new HashMap<>();
String topic = sourceRecord.topic();
String[] split = topic.split("[.]");
String database = split[1];
String table = split[2];
resultMap.put("db", database);
resultMap.put("tableName", table);
//获取操作类型
Envelope.Operation operation = Envelope.operationFor(sourceRecord);
//获取数据本身
Struct struct = (Struct) sourceRecord.value();
Struct after = struct.getStruct("after");
Struct before = struct.getStruct("before");
String op = operation.name();
resultMap.put("op", op);
//新增,更新或者初始化
if (op.equals(Envelope.Operation.CREATE.name()) || op.equals(Envelope.Operation.READ.name()) || op.equals(Envelope.Operation.UPDATE.name())) {
JSONObject afterJson = new JSONObject();
if (after != null) {
Schema schema = after.schema();
for (Field field : schema.fields()) {
afterJson.put(field.name(), after.get(field.name()));
}
resultMap.put("after", afterJson);
}
}
if (op.equals(Envelope.Operation.DELETE.name())) {
JSONObject beforeJson = new JSONObject();
if (before != null) {
Schema schema = before.schema();
for (Field field : schema.fields()) {
beforeJson.put(field.name(), before.get(field.name()));
}
resultMap.put("before", beforeJson);
}
}
collector.collect(JSON.toJSONString(resultMap, JSONWriter.Feature.FieldBased, JSONWriter.Feature.LargeObject));
}
@Override
public TypeInformation<String> getProducedType() {
return BasicTypeInfo.STRING\_TYPE\_INFO;
}
}
2.3 自定义日期格式转换器
debezium
会将日期转为5位数字,日期时间转为13位的数字,因此我们需要根据Sqlserver
的日期类型转换成标准的时期或者时间格式。Sqlserver
的日期类型主要包含以下几种:
网上学习资料一大堆,但如果学到的知识不成体系,遇到问题时只是浅尝辄止,不再深入研究,那么很难做到真正的技术提升。
一个人可以走的很快,但一群人才能走的更远!不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人,都欢迎加入我们的的圈子(技术交流、学习资源、职场吐槽、大厂内推、面试辅导),让我们一起学习成长!
img-nmrzRjYK-1715794223914)]
[外链图片转存中…(img-oj7uQTtv-1715794223914)]
网上学习资料一大堆,但如果学到的知识不成体系,遇到问题时只是浅尝辄止,不再深入研究,那么很难做到真正的技术提升。
一个人可以走的很快,但一群人才能走的更远!不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人,都欢迎加入我们的的圈子(技术交流、学习资源、职场吐槽、大厂内推、面试辅导),让我们一起学习成长!