flink整合kafka&Debezium(一）项目启动

最新推荐文章于 2024-04-21 09:45:58 发布
最新推荐文章于 2024-04-21 09:45:58 发布
阅读量3.3k 收藏 6
点赞数
分类专栏： 中间件 文章标签： Flink Debezium Kafka Zookeeper Database
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/kamputer/article/details/103764051
版权

Debezium的搭建

参见之前的blog使用docker启动一套
注意kafka的启动参数要加上ADVERTISED_HOST_NAME作为外部访问的地址，因为这个地址会放入zookeeper里，如果不设置就会变成docker内部的地址，外部是无法访问的，并且后面flink也不报错，只是收不到消息。。。
可以使用下面命令

docker run -d --name kafka -p 9092:9092 -e ADVERTISED_HOST_NAME=$(ip a|grep 192.168|awk '{print $2}'|awk -F / '{print $1}') --link zookeeper:zookeeper debezium/kafka:0.10

产生数据

创建表

创建一个有各种类型column的表

create table table_name
(
	id_c varchar(64)
		constraint table_name_pk
			primary key,
	json_c json,
	uuid_c uuid,
	xml_c xml,
	int_c int,
	money_c money,
	jsonb_c jsonb,
	numeric_c numeric,
	boolean_c boolean,
	date_c date,
	timestamp_c timestamp,
	timestamptz_c timestamptz,
	time_c time,
	text_c text,
	char_c char(11),
	varchar_c varchar(12)
);

null数据

在数据库中执行

INSERT INTO public.table_name (id_c) VALUES ('null_value')

监听kafka消息

docker run -it --rm --link zookeeper:zookeeper --link kafka:kafka debezium/kafka:0.10 watch-topic -a -k -m minBytes postgres.public.table_name

会打印以下信息（注意最后一行是刚才的sql）

ARNING: Using default BROKER_ID=1, which is valid only for non-clustered installations.
Using ZOOKEEPER_CONNECT=172.17.0.5:2181
Using KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.17.0.8:9092
Using KAFKA_BROKER=172.17.0.6:9092
Contents of topic postgres.public.table_name:
{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"id_c"}],"optional":false,"name":"postgres.public.table_name.Key"},"payload":{"id_c":"null_value"}}	{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"string","optional":false,"field":"id_c"},{"type":"string","optional":true,"name":"io.debezium.data.Json","version":1,"field":"json_c"},{"type":"string","optional":true,"name":"io.debezium.data.Uuid","version":1,"field":"uuid_c"},{"type":"string","optional":true,"name":"io.debezium.data.Xml","version":1,"field":"xml_c"},{"type":"int32","optional":true,"field":"int_c"},{"type":"bytes","optional":true,"name":"org.apache.kafka.connect.data.Decimal","version":1,"parameters":{"scale":"2"},"field":"money_c"},{"type":"string","optional":true,"name":"io.debezium.data.Json","version":1,"field":"jsonb_c"},{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"numeric_c"},{"type":"boolean","optional":true,"field":"boolean_c"},{"type":"int32","optional":true,"name":"io.debezium.time.Date","version":1,"field":"date_c"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"timestamp_c"},{"type":"string","optional":true,"name":"io.debezium.time.ZonedTimestamp","version":1,"field":"timestamptz_c"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTime","version":1,"field":"time_c"},{"type":"string","optional":true,"field":"text_c"},{"type":"string","optional":true,"field":"char_c"},{"type":"string","optional":true,"field":"varchar_c"}],"optional":true,"name":"postgres.public.table_name.Value","field":"before"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id_c"},{"type":"string","optional":true,"name":"io.debezium.data.Json","version":1,"field":"json_c"},{"type":"string","optional":true,"name":"io.debezium.data.Uuid","version":1,"field":"uuid_c"},{"type":"string","optional":true,"name":"io.debezium.data.Xml","version":1,"field":"xml_c"},{"type":"int32","optional":true,"field":"int_c"},{"type":"bytes","optional":true,"name":"org.apache.kafka.connect.data.Decimal","version":1,"parameters":{"scale":"2"},"field":"money_c"},{"type":"string","optional":true,"name":"io.debezium.data.Json","version":1,"field":"jsonb_c"},{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"numeric_c"},{"type":"boolean","optional":true,"field":"boolean_c"},{"type":"int32","optional":true,"name":"io.debezium.time.Date","version":1,"field":"date_c"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTimestamp","version":1,"field":"timestamp_c"},{"type":"string","optional":true,"name":"io.debezium.time.ZonedTimestamp","version":1,"field":"timestamptz_c"},{"type":"int64","optional":true,"name":"io.debezium.time.MicroTime","version":1,"field":"time_c"},{"type":"string","optional":true,"field":"text_c"},{"type":"string","optional":true,"field":"char_c"},{"type":"string","optional":true,"field":"varchar_c"}],"optional":true,"name":"postgres.public.table_name.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":false,"field":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"int64","optional":true,"field":"txId"},{"type":"int64","optional":true,"field":"lsn"},{"type":"int64","optional":true,"field":"xmin"}],"optional":false,"name":"io.debezium.connector.postgresql.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"}],"optional":false,"name":"postgres.public.table_name.Envelope"},"payload":{"before":null,"after":{"id_c":"null_value","json_c":null,"uuid_c":null,"xml_c":null,"int_c":null,"money_c":null,"jsonb_c":null,"numeric_c":null,"boolean_c":null,"date_c":null,"timestamp_c":null,"timestamptz_c":null,"time_c":null,"text_c":null,"char_c":null,"varchar_c":null},"source":{"version":"0.10.0.Final","connector":"postgresql","name":"postgres","ts_ms":1577691115382,"snapshot":"false","db":"postgres","schema":"public","table":"table_name","txId":576,"lsn":24630784,"xmin":null},"op":"c","ts_ms":1577691115691}}

从}{分成两个json后格式化

{
	"schema": {
		"type": "struct",
		"fields": [{
			"type": "string",
			"optional": false,
			"field": "id_c"
		}],
		"optional": false,
		"name": "postgres.public.table_name.Key"
	},
	"payload": {
		"id_c": "null_value"
	}
}

{
	"schema": {
		"type": "struct",
		"fields": [{
			"type": "struct",
			"fields": [{
				"type": "string",
				"optional": false,
				"field": "id_c"
			}, {
				"type": "string",
				"optional": true,
				"name": "io.debezium.data.Json",
				"version": 1,
				"field": "json_c"
			}, {
				"type": "string",
				"optional": true,
				"name": "io.debezium.data.Uuid",
				"version": 1,
				"field": "uuid_c"
			}, {
				"type": "string",
				"optional": true,
				"name": "io.debezium.data.Xml",
				"version": 1,
				"field": "xml_c"
			}, {
				"type": "int32",
				"optional": true,
				"field": "int_c"
			}, {
				"type": "bytes",
				"optional": true,
				"name": "org.apache.kafka.connect.data.Decimal",
				"version": 1,
				"parameters": {
					"scale": "2"
				},
				"field": "money_c"
			}, {
				"type": "string",
				"optional": true,
				"name": "io.debezium.data.Json",
				"version": 1,
				"field": "jsonb_c"
			}, {
				"type": "struct",
				"fields": [{
					"type": "int32",
					"optional": false,
					"field": "scale"
				}, {
					"type": "bytes",
					"optional": false,
					"field": "value"
				}],
				"optional": true,
				"name": "io.debezium.data.VariableScaleDecimal",
				"version": 1,
				"doc": "Variable scaled decimal",
				"field": "numeric_c"
			}, {
				"type": "boolean",
				"optional": true,
				"field": "boolean_c"
			}, {
				"type": "int32",
				"optional": true,
				"name": "io.debezium.time.Date",
				"version": 1,
				"field": "date_c"
			}, {
				"type": "int64",
				"optional": true,
				"name": "io.debezium.time.MicroTimestamp",
				"version": 1,
				"field": "timestamp_c"
			}, {
				"type": "string",
				"optional": true,
				"name": "io.debezium.time.ZonedTimestamp",
				"version": 1,
				"field": "timestamptz_c"
			}, {
				"type": "int64",
				"optional": true,
				"name": "io.debezium.time.MicroTime",
				"version": 1,
				"field": "time_c"
			}, {
				"type": "string",
				"optional": true,
				"field": "text_c"
			}, {
				"type": "string",
				"optional": true,
				"field": "char_c"
			}, {
				"type": "string",
				"optional": true,
				"field": "varchar_c"
			}],
			"optional": true,
			"name": "postgres.public.table_name.Value",
			"field": "before"
		}, {
			"type": "struct",
			"fields": [{
				"type": "string",
				"optional": false,
				"field": "id_c"
			}, {
				"type": "string",
				"optional": true,
				"name": "io.debezium.data.Json",
				"version": 1,
				"field": "json_c"
			}, {
				"type": "string",
				"optional": true,
				"name": "io.debezium.data.Uuid",
				"version": 1,
				"field": "uuid_c"
			}, {
				"type": "string",
				"optional": true,
				"name": "io.debezium.data.Xml",
				"version": 1,
				"field": "xml_c"
			}, {
				"type": "int32",
				"optional": true,
				"field": "int_c"
			}, {
				"type": "bytes",
				"optional": true,
				"name": "org.apache.kafka.connect.data.Decimal",
				"version": 1,
				"parameters": {
					"scale": "2"
				},
				"field": "money_c"
			}, {
				"type": "string",
				"optional": true,
				"name": "io.debezium.data.Json",
				"version": 1,
				"field": "jsonb_c"
			}, {
				"type": "struct",
				"fields": [{
					"type": "int32",
					"optional": false,
					"field": "scale"
				}, {
					"type": "bytes",
					"optional": false,
					"field": "value"
				}],
				"optional": true,
				"name": "io.debezium.data.VariableScaleDecimal",
				"version": 1,
				"doc": "Variable scaled decimal",
				"field": "numeric_c"
			}, {
				"type": "boolean",
				"optional": true,
				"field": "boolean_c"
			}, {
				"type": "int32",
				"optional": true,
				"name": "io.debezium.time.Date",
				"version": 1,
				"field": "date_c"
			}, {
				"type": "int64",
				"optional": true,
				"name": "io.debezium.time.MicroTimestamp",
				"version": 1,
				"field": "timestamp_c"
			}, {
				"type": "string",
				"optional": true,
				"name": "io.debezium.time.ZonedTimestamp",
				"version": 1,
				"field": "timestamptz_c"
			}, {
				"type": "int64",
				"optional": true,
				"name": "io.debezium.time.MicroTime",
				"version": 1,
				"field": "time_c"
			}, {
				"type": "string",
				"optional": true,
				"field": "text_c"
			}, {
				"type": "string",
				"optional": true,
				"field": "char_c"
			}, {
				"type": "string",
				"optional": true,
				"field": "varchar_c"
			}],
			"optional": true,
			"name": "postgres.public.table_name.Value",
			"field": "after"
		}, {
			"type": "struct",
			"fields": [{
				"type": "string",
				"optional": false,
				"field": "version"
			}, {
				"type": "string",
				"optional": false,
				"field": "connector"
			}, {
				"type": "string",
				"optional": false,
				"field": "name"
			}, {
				"type": "int64",
				"optional": false,
				"field": "ts_ms"
			}, {
				"type": "string",
				"optional": true,
				"name": "io.debezium.data.Enum",
				"version": 1,
				"parameters": {
					"allowed": "true,last,false"
				},
				"default": "false",
				"field": "snapshot"
			}, {
				"type": "string",
				"optional": false,
				"field": "db"
			}, {
				"type": "string",
				"optional": false,
				"field": "schema"
			}, {
				"type": "string",
				"optional": false,
				"field": "table"
			}, {
				"type": "int64",
				"optional": true,
				"field": "txId"
			}, {
				"type": "int64",
				"optional": true,
				"field": "lsn"
			}, {
				"type": "int64",
				"optional": true,
				"field": "xmin"
			}],
			"optional": false,
			"name": "io.debezium.connector.postgresql.Source",
			"field": "source"
		}, {
			"type": "string",
			"optional": false,
			"field": "op"
		}, {
			"type": "int64",
			"optional": true,
			"field": "ts_ms"
		}],
		"optional": false,
		"name": "postgres.public.table_name.Envelope"
	},
	"payload": {
		"before": null,
		"after": {
			"id_c": "null_value",
			"json_c": null,
			"uuid_c": null,
			"xml_c": null,
			"int_c": null,
			"money_c": null,
			"jsonb_c": null,
			"numeric_c": null,
			"boolean_c": null,
			"date_c": null,
			"timestamp_c": null,
			"timestamptz_c": null,
			"time_c": null,
			"text_c": null,
			"char_c": null,
			"varchar_c": null
		},
		"source": {
			"version": "0.10.0.Final",
			"connector": "postgresql",
			"name": "postgres",
			"ts_ms": 1577691115382,
			"snapshot": "false",
			"db": "postgres",
			"schema": "public",
			"table": "table_name",
			"txId": 576,
			"lsn": 24630784,
			"xmin": null
		},
		"op": "c",
		"ts_ms": 1577691115691
	}
}

Flink的搭建

这里参考官方docker方式
创建目录

mkdir flink
vi docker-compose.yml

黏贴以下配置

version: "2.1"
services:
  jobmanager:
    image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

  taskmanager:
    image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

启动集群

docker-compose up

用浏览器访问http://localhost:8081进入flink控制面板

开发flink项目

初始化

使用下面命令用maven生成一个flink项目

mvn archetype:generate -DarchetypeGroupId=org.apache.flink -DarchetypeArtifactId=flink-quickstart-java -DarchetypeVersion=1.9.0 -DgroupId=qbit.flink -DartifactId=flink -Dversion=1.0.0 -Dpackage=qbit.flink -DinteractiveMode=false

对接kafka

maven依赖

去掉pom.xml文件里的相应注释即可加入kafka依赖

<dependency>
	<groupId>org.apache.flink</groupId>
	<artifactId>flink-connector-kafka-0.10_${scala.binary.version}</artifactId>
	<version>${flink.version}</version>
</dependency>

编写java代码

public class StreamingJob {
	public static void main(String[] args) throws Exception {
		// set up the streaming execution environment
		final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
		Properties properties=new Properties();
		properties.setProperty("bootstrap.servers","192.168.1.191:9092");
		properties.setProperty("group.id","flink-test");
		FlinkKafkaConsumer010<String> consumer= new FlinkKafkaConsumer010<>("postgres.public.table_name", new SimpleStringSchema(), properties);
//		consumer.setStartFromEarliest();
		DataStream<String> stream=env.addSource(consumer);
		stream.print();
		// execute program
		env.execute("debezium-test");
	}
}

使用maven package命令打包成jar

上传

进入Submit New Job页面点击Add New上传刚才打包好的jar，然后会看到一个叫debezium的job，状态是RUNNING

运行

如果kafka配置错误会在日志里看到以下内容
INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: Custom Source -> Sink: Print to Std. Out (1/1) (ea5b645e70cf101f4fe5da1fd5a63b80) switched from RUNNING to FAILED.
taskmanager_1  | org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

注意下面的一行打印的ip地址必须是kafka宿主机的地址而不是容器的地址

org.apache.kafka.clients.consumer.internals.AbstractCoordinator  - Discovered coordinator 192.168.1.191:9092 (id: 2147483646 rack: null) for group flink-test.

然后操作数据库，flink就会在CLI界面打印出数据

首席IT民工
关注
  • 0
    点赞
  • 踩
  • 6
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
快速启动flink项目
someInNeed的博客
03-15 1617
按照这个步骤1分钟内创建完成 idea-----File----new---Project------Maven----Create from archetype----Add Archetype 弹出框： GroupId填org.apache.flink ArtifactId填flink-quickstart-java Version填1.14.0 选中刚刚添加的Archetype，点Next 填写你要创建的这个flink demo的GroupId,ArtifactId,Version,点击
Flink CDC基本概念以及MySQL同步到MySQL
最新发布
L
07-29 786
本文基于Flink CDC v2.4.2版本和Flink 1.17.1版本。
Flink整合Kafka&Debezium（三）流转换为表
Qbit编程学习笔记
01-09 1936
提取json 加入maven依赖 <dependency> <groupId>com.jayway.jsonpath</groupId> <artifactId>json-path</artifactId> <version>2.2.0</version> </dependency&...
IDEA上运行Flink任务
热门推荐
程序员欣宸的博客
04-11 1万+
将最简单的flink任务在IDEA上开发和运行，为后面的深入学习和实践打好基础。
一键启动Debezium+Kafka+Flink
Qbit编程学习笔记
01-06 1062
编写配置文件docker-compose.yml version: "3.3" services: database: image: debezium/postgres:11 container_name: database ports: - "5432:5432" environment: - POSTGRES_PASSWORD=mim...
基于Kafka+Debezium+Flink的 SQLServer实时增量同步方案
寒暄的博客
08-31 1485
安装Connector 下载链接 http://client.hub.confluent.io/confluent-hub-client-latest.tar.gz?_ga=2.215682399.93673590.1629776859-1065619546.1629776859 解压 tar confluent-hub-client-latest.tar -C /opt/module/confluent-hub 配置环境变量 export CONN_HOME=/opt/module/confluent
Flink Kafka整合
GavinKai
04-02 817
Apache Kafka 连接器 Flink 提供了Apache Kafka连接器，用于从 Kafka topic 中读取或者向其中写入数据，可提供精确一次的处理语义。 依赖 Kafka Consumer DeserializationSchema 配置 Kafka Consumer 开始消费的位置 Kafka Consumer 和容错 Kafka Consumer Topic 和分区发现 Kafka Consumer 提交 Offset 的行为配置 Kafka Consum..
Kafka集成Flink
weixin_38425605的博客
04-05 1781
集成Flink一、Flink环境准备二、Flink生产者1.在包名下创建java类：FlinkKafkaProducer三、Flink消费者 Flink是一个在大数据开发中非常常用的组件。可以用于Kafka的生产者，亦可以用于Flink的消费者 一、Flink环境准备 创建一个maven项目flink-kafka 添加配置文件 <dependencies> <dependency> <groupId>org
CDC 数据入湖方案：Flink CDC ＞ Kafka ＞ Hudi
Laurence的技术博客
02-20 827
本方案的技术链路为：使用 Flink CDC 将 MySQL 的 CDC 数据 （Json 格式）接入到 Kafka ，然后通过 Flink Hudi Connector 将摄取的 CDC 数据写入到 Hudi 表中。文本是本博客的 CDC 数据入湖系列方案中最为基础的一套堆栈，架构上也比较简单，适合作为 POC 快速搭建 CDC 实时处理链路。如果寻求更加适用于生产环境的解决方案，请参考
Flink CDC 整库 / 多表同步至 Kafka 方案（附源码）
Laurence的技术博客
04-18 2547
我们此前介绍的一些 CDC 开箱即用方案往往都是一张表对应一条独立的链路（作业），需要一个独立的数据库连接，在表数量很大的情况下，会对数据库造成很大压力，同时过多的 Flink 作业会不易于管理和维护，为众多小表创建独立的采集作业也浪费了资源。此外，使用 Flink SQL 针对每张表定义 CDC 作业也是一项繁重的工作，如果能简化或省略编写大量 SQL 的工作也是一项重要的改进。所以，一种更为实用的解决方案是：使用一个 Flink 作业完成整库 / 多表的 CDC 数据接入工作。本文我们会详细介绍一下这一
Hudi 0.11.0 + Flink1.14.4 + Hive + Flink CDC + Kafka 集成
zhangs_123的博客
08-09 3179
Hudi 0.11.0 + Flink1.14.4 + Hive + Flink CDC + Kafka
debezium-datetime-converter:Debezium自定义转换器用于处理mysql datetime类型的问题
03-15
debezium-datetime-converter Debezium自定义转换器用于处理mysql datetime类型的问题 MySQL的 Binlog连接器 德比兹 德比兹图式 日期（2021-01-28） 本地日期（2021-01-28） 整数（18655） 碘化氢时间日期 时间（17:29:04） 期间（PT17H29M4S） 长（62944000000） （（持续时间）数据）.toNanos（）/ 1_000 时间戳记（2021-01-28 17:29:04） ZonedDateTime （2021-01-28T09：29：04Z） 细绳（2021-01-28T09：29：04Z） io.debezium.ZoneedTimestamp 约会时间（2021-01-28 17:29:04） LocalDateTime （2021-01-28T17：29
springboot整合flink cdc监听数据库数据
蔡定努
03-03 1588
【代码】springboot整合flink cdc监听数据库数据。
大数据FLINK实时数仓项目实战
悦分享
10-06 7014
普通的实时计算优先考虑时效性，所以从数据源采集经过实时计算直接得到结果。如此做时效性更好，但是弊端是由于计算过程中的中间结果没有沉淀下来，所以当面对大量实时需求的时候，计算的复用性较差，开发成本随着需求增加直线上升。实时数仓基于一定的数据仓库理念，对数据处理流程进行规划、分层，目的是提高数据的复用性。 原始数据，日志和业务数据 。根据数据对象为单位进行分流，比如订单、页面访问等等。维度数据。对于部分数据对象进行进一步加工，比如独立访问、跳出行为，也可以和维度进行关联，形成宽表，依旧是明细数据。根据某个主题将
探索Flink Connector Debezium：实时数据集成的新篇章
gitblog_00085的博客
04-21 366
探索Flink Connector Debezium：实时数据集成的新篇章 项目地址:https://gitcode.com/quan2005/flink-connector-debezium 在这个大数据时代，实时数据处理和事件驱动的应用已经成为企业决策的关键。Flink Connector Debezium 是一个优秀的开源项目，它将Apache Flink与Debezium相结合，为企业提供...
FlinkCdc--Debezium实现Kafka实时监控mysql binlog日志
atguigu_Jack的博客
11-23 3915
不管是什么大数据组件大部分都分单机和集群模式，这次我配置的是kafka集群监控mysql binlog日志 一.Zookeeper和Kafka集群部署 我的服务器是三台节点aliyun-bigdata-01aliyun-bigdata-02aliyun-bigdata-03 1.第一步实现kafka集群部署和zookeeper集群部署先启动zookeeper再启动kafka,涉及的相关组件如下图所示，去网盘下载 debezium-connector-mysql-1.4.0.Final-p...
Flink系列之：Debezium采集Mysql数据库表数据到Kafka Topic，同步kafka topic数据到StarRocks数据库
zhengzaifeidelushang的博客
01-29 1132
Flink系列之：Debezium采集Mysql数据库表数据到Kafka Topic，同步kafka topic数据到StarRocks数据库
【实战】Kafka Connector用debezium读取mysql的Binlog实时同步Kafka映射成FlinkSql表
遇事不决问春风
05-27 1768
目的：实时同步mysql的数据，之前是通过flink-cdc实现，但是flink-cdc需要编写代码，维护起来相对来说较为麻烦。现在直接采用kafkaconnect来进行，操作简单便于维护 一、前期准备 软件版本 mysql 5.7.30 kafka 2.2.1-cdh6.3.2 Flink 12.2.2 Debezium连接器 debezium 1.5.0 相关介绍 Debezium是一个开源项目，为捕获数据更改(change data capture,CDC)提供了一个低延迟的流式处理平台。你可以安
flink整合Debezium&kafka（二）探索postgresql的数据类型
Qbit编程学习笔记
01-06 1123
在pgsql里执行以下语句插入一条记录 INSERT INTO "public"."table_name" ("id_c", "json_c", "uuid_c", "xml_c", "int_c", "money_c", "jsonb_c", "numeric_c", "boolean_c", "date_c", "timestamp_c", "timestamptz_c", "time_c",...
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值