Debezium 从oracle抓取数据到kafka

最新推荐文章于 2024-05-24 14:23:33 发布
最新推荐文章于 2024-05-24 14:23:33 发布
阅读量3.6k 收藏 27
点赞数 6
文章标签： kafka
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_40548182/article/details/117956341
版权

环境

操作系统版本

[root@localhost kafka_2.13-2.8.0]# cat /etc/redhat-release 
CentOS Linux release 7.5.1804 (Core) 
[root@localhost kafka_2.13-2.8.0]# uname -r
3.10.0-862.el7.x86_64

glibc版本

[root@localhost kafka_2.13-2.8.0]# rpm -qa|grep glibc
glibc-common-2.17-222.el7.x86_64
glibc-2.17-222.el7.x86_64

kafka版本

kafka_2.13-2.8.0

配置zookeeper

10.0.2.18节点配置

[root@localhost kafka_2.13-2.8.0]# cat /opt/kafka_2.13-2.8.0/config/zookeeper.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
# 
#    http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# the directory where the snapshot is stored.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
# Disable the adminserver by default to avoid port conflicts.
# Set the port to something non-conflicting if choosing to enable this
admin.enableServer=false
# admin.serverPort=8080
tickTime=2000
initLimit=5
syncLimit=2
server.1=10.0.2.20:2888:3888
server.2=10.0.2.18:2889:3889
server.3=10.0.2.19:2890:3890

echo "2" > /tmp/zookeeper/myid

10.0.2.19节点配置

[root@localhost kafka_2.13-2.8.0]# cat /opt/kafka_2.13-2.8.0/config/zookeeper.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
# 
#    http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# the directory where the snapshot is stored.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
# Disable the adminserver by default to avoid port conflicts.
# Set the port to something non-conflicting if choosing to enable this
admin.enableServer=false
# admin.serverPort=8080
tickTime=2000
initLimit=5
syncLimit=2
server.1=10.0.2.20:2888:3888
server.2=10.0.2.18:2889:3889
server.3=10.0.2.19:2890:3890

echo "3" > /tmp/zookeeper/myid

10.0.2.20节点配置

[root@localhost kafka_2.13-2.8.0]# cat /opt/kafka_2.13-2.8.0/config/zookeeper.properties
censed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
# 
#    http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# the directory where the snapshot is stored.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
# Disable the adminserver by default to avoid port conflicts.
# Set the port to something non-conflicting if choosing to enable this
admin.enableServer=false
# admin.serverPort=8080
tickTime=2000
initLimit=5
syncLimit=2
server.1=10.0.2.20:2888:3888
server.2=10.0.2.18:2889:3889
server.3=10.0.2.19:2890:3890

echo "1" > /tmp/zookeeper/myid

启动zookeeper集群

每个节点都执行

cd /opt/kafka_2.13-2.8.0
bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

kafka配置

10.0.2.18节点配置

[root@localhost kafka_2.13-2.8.0]# cat /opt/kafka_2.13-2.8.0/config/server.properties
broker.id=2
listeners=PLAINTEXT://10.0.2.18:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

log.retention.hours=168

log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.0.2.18:2181,10.0.2.19:2181,10.0.2.20:2181

zookeeper.connection.timeout.ms=18000

group.initial.rebalance.delay.ms=0
delete.topic.enable=true

10.0.2.19节点配置

[root@localhost kafka_2.13-2.8.0]# cat /opt/kafka_2.13-2.8.0/config/server.properties
broker.id=3
listeners=PLAINTEXT://10.0.2.18:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

log.retention.hours=168

log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.0.2.18:2181,10.0.2.19:2181,10.0.2.20:2181

zookeeper.connection.timeout.ms=18000

group.initial.rebalance.delay.ms=0
delete.topic.enable=true

10.0.2.20节点配置

[root@localhost kafka_2.13-2.8.0]# cat /opt/kafka_2.13-2.8.0/config/server.properties
broker.id=1
listeners=PLAINTEXT://10.0.2.19:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

log.retention.hours=168

log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.0.2.18:2181,10.0.2.19:2181,10.0.2.20:2181

zookeeper.connection.timeout.ms=18000

group.initial.rebalance.delay.ms=0
delete.topic.enable=true

启动kafka集群

所有节点都执行

cd /opt/kafka_2.13-2.8.0
bin/kafka-server-start.sh -daemon config/server.properties

debezium安装包准备

下载安装包

下载安装包如下

debezium-connector-oracle-1.6.0-20210616.001509-60-plugin.tar.gz
instantclient-basic-linux.x64-21.1.0.0.0.zip

下载debezium-connector-oracle
https://debezium.io/documentation/reference/install.html
在这里插入图片描述
下载instantclient
https://debezium.io/documentation/reference/install.html
在这里插入图片描述
跳转到下载页面
https://www.oracle.com/database/technologies/instant-client/downloads.html
在这里插入图片描述
在这里插入图片描述

解压安装包

在这里插入图片描述

复制jar包到对应目录

cp /opt/debezium-connector-oracle/*.jar /opt/kafka_2.13-2.8.0/libs/
cp /opt/instantclient_21_1/*.jar /opt/kafka_2.13-2.8.0/libs/

oracle配置

登录数据库

切换到oracle用户

su - oracle

切换到oralce安装目录
登录oracle数据库

sqlplus / as sysdba

开启归档日志

开启归档日志
需要在mount状态下开始数据库归档，重启至mount

SQL> shutdown immediate
//输出结果
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup mount
ORACLE instance started.

Total System Global Area 1603411968 bytes
Fixed Size            2213776 bytes
Variable Size          989857904 bytes
Database Buffers      603979776 bytes
Redo Buffers            7360512 bytes
Database mounted.

开启数据库归档

SQL> alter database archivelog;
//输出结果
Database altered.

查看归档结果

SQL> archive log list
//输出结果
Database log mode           Archive Mode
Automatic archival           Enabled
Archive destination           /u01/app/oracle/archive_log
Oldest online log sequence     244
Next log sequence to archive   246
Current log sequence           246

开启自动归档

alter system archive log start;

开启强制归档

ALTER DATABASE FORCE LOGGING;

打开数据库

SQL> alter database open;

Database altered.

确认数据库为归档模式

SQL> select log_mode from v$database;

LOG_MODE
------------------------------------
ARCHIVELOG

SQL> select archiver from v$instance;

ARCHIVER
---------------------
STARTED

开启补充日志

开启最小字段补充日志

SQL> alter database add supplemental log data ;
 
Database altered.

开启全体字段补充日志

SQL> alter database add supplemental log data (all) columns; 
 
Database altered.

确认是否开启

select SUPPLEMENTAL_LOG_DATA_MIN min,
       SUPPLEMENTAL_LOG_DATA_PK  pk,
       SUPPLEMENTAL_LOG_DATA_UI  ui,
       SUPPLEMENTAL_LOG_DATA_FK  fk,
       SUPPLEMENTAL_LOG_DATA_ALL "all"
  6  from v$database;

MIN			 PK	   UI	     FK        all
------------------------ --------- --------- --------- ---------
YES			 NO	   NO	     NO        YES

创建debezium相关用户并授权

CREATE USER c DEFAULT TABLESPACE logminer_tbs QUOTA UNLIMITED ON logminer_tbs；
GRANT CREATE SESSION TO c;
GRANT SET CONTAINER TO c;
GRANT SELECT ON V_$DATABASE to c;
GRANT FLASHBACK ANY TABLE TO c;
GRANT SELECT ANY TABLE TO c;
GRANT SELECT_CATALOG_ROLE TO c;
GRANT EXECUTE_CATALOG_ROLE TO c;
GRANT SELECT ANY TRANSACTION TO c;
GRANT LOGMINING TO c;


GRANT CREATE TABLE TO c;
GRANT LOCK ANY TABLE TO c;
GRANT ALTER ANY TABLE TO c;
GRANT CREATE SEQUENCE TO c;

GRANT EXECUTE ON DBMS_LOGMNR TO c;
GRANT EXECUTE ON DBMS_LOGMNR_D TO c;

GRANT SELECT ON V_$LOG TO c;
GRANT SELECT ON V_$LOG_HISTORY TO c;
GRANT SELECT ON V_$LOGMNR_LOGS TO c;
GRANT SELECT ON V_$LOGMNR_CONTENTS TO c;
GRANT SELECT ON V_$LOGMNR_PARAMETERS TO c;
GRANT SELECT ON V_$LOGFILE TO c;
GRANT SELECT ON V_$ARCHIVED_LOG TO c;
GRANT SELECT ON V_$ARCHIVE_DEST_STATUS TO c;

配置kafka-connect

说明：kafka-connect按照分布式方式配置。

cd /opt/kafka_2.13-2.8.0

10.0.2.18节点配置

cat config/connect-distributed.properties
bootstrap.servers=10.0.2.18:9092,10.0.2.19:9092,10.0.2.20:9092
group.id=connect-cluster
#group.id=1
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false

internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false


offset.storage.topic=connect-offsets
offset.storage.replication.factor=3
offset.storage.partitions=3

config.storage.topic=connect-configs
config.storage.replication.factor=3

status.storage.topic=connect-status
status.storage.replication.factor=3

offset.flush.interval.ms=10000
rest.advertised.host.name=10.0.2.18
#rest.advertised.port=8083

offset.storage.file.filename=/tmp/connect.offsets
plugin.path=/opt/debezium-connector-oracle/

10.0.2.19节点配置

cat config/connect-distributed.properties
bootstrap.servers=10.0.2.18:9092,10.0.2.19:9092,10.0.2.20:9092
group.id=connect-cluster
#group.id=1
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false

internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false


offset.storage.topic=connect-offsets
offset.storage.replication.factor=3
offset.storage.partitions=3

config.storage.topic=connect-configs
config.storage.replication.factor=3

status.storage.topic=connect-status
status.storage.replication.factor=3

offset.flush.interval.ms=10000
rest.advertised.host.name=10.0.2.19
#rest.advertised.port=8083

offset.storage.file.filename=/tmp/connect.offsets
plugin.path=/opt/debezium-connector-oracle/

10.0.2.20节点配置

cat config/connect-distributed.properties
bootstrap.servers=10.0.2.18:9092,10.0.2.19:9092,10.0.2.20:9092
group.id=connect-cluster
#group.id=1
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false

internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false


offset.storage.topic=connect-offsets
offset.storage.replication.factor=3
offset.storage.partitions=3

config.storage.topic=connect-configs
config.storage.replication.factor=3

status.storage.topic=connect-status
status.storage.replication.factor=3

offset.flush.interval.ms=10000
rest.advertised.host.name=10.0.2.20
#rest.advertised.port=8083

offset.storage.file.filename=/tmp/connect.offsets
plugin.path=/opt/debezium-connector-oracle/

创建启动必须topic

bin/kafka-topics.sh --create --zookeeper 10.0.2.18:2181 --topic connect-configs --replication-factor 3 --partitions 1 --config cleanup.policy=compact
bin/kafka-topics.sh --create --zookeeper 10.0.2.19:2181 --topic connect-offsets --replication-factor 3 --partitions 50 --config cleanup.policy=compact
bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic connect-status --replication-factor 3 --partitions 10 --config cleanup.policy=compact

启动kafka-connect

在每个节点都执行

cd /opt/kafka_2.13-2.8.0
bin/connect-distributed.sh config/connect-distributed.properties

创建连接器

curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
"name": "inventory-connector",
"config": {
"connector.class" : "io.debezium.connector.oracle.OracleConnector",
"tasks.max" : "1",
"database.server.name" : "server1",
"database.hostname" : "10.0.2.15",
"database.port" : "1521",
"database.user" : "c",
"database.password" : "dbz",
"database.dbname" : "ORCL",
"database.history.kafka.bootstrap.servers" : "10.0.2.20:9092,10.0.2.18:9092,10.0.2.19:9092",
"database.history.kafka.topic": "schema-changes.inventory"
}
}'

查看连接器

[root@localhost kafka_2.13-2.8.0]# curl -s localhost:8083/connectors|jq
[
  "inventory-connector"
]

查看连接器详细信息

[root@localhost kafka_2.13-2.8.0]# curl -s localhost:8083/connectors/inventory-connector|jq
{
  "name": "inventory-connector",
  "config": {
    "connector.class": "io.debezium.connector.oracle.OracleConnector",
    "database.user": "c",
    "database.dbname": "ORCL",
    "tasks.max": "1",
    "database.hostname": "10.0.2.15",
    "database.password": "dbz",
    "database.history.kafka.bootstrap.servers": "10.0.2.20:9092,10.0.2.18:9092,10.0.2.19:9092",
    "database.history.kafka.topic": "schema-changes.inventory",
    "name": "inventory-connector",
    "database.server.name": "server1",
    "database.port": "1521"
  },
  "tasks": [
    {
      "connector": "inventory-connector",
      "task": 0
    }
  ],
  "type": "source"
}

查看连接器状态

[root@localhost kafka_2.13-2.8.0]# curl -s localhost:8083/connectors/inventory-connector/status|jq
{
  "name": "inventory-connector",
  "connector": {
    "state": "RUNNING",
    "worker_id": "127.0.0.1:8083"
  },
  "tasks": [
    {
      "id": 0,
      "state": "RUNNING",
      "worker_id": "127.0.0.1:8083"
    }
  ],
  "type": "source"
}

验证是否生成topic

在这里插入图片描述

验证数据同步

查看oracle表中数据

SQL> conn test/test;
Connected.
SQL> select * from student;
0 rows selected.

查看kafka对应的topic中的数据

bin/kafka-console-consumer.sh --bootstrap-server 10.0.2.20:9092 --topic server1.TEST.STUDENT --from-beginning

在这里插入图片描述

验证乱序插入数据是否可以同步

oracle表中插入数据

SQL> insert into student(sno,sname,ssex,sbirthday,sclass) values(108,'曾华','男',to_date('1977-09-01','yyyy-mm-dd'),95033);

1 row created.

SQL> commit;

Commit complete.
SQL> insert into student(sno,sname,ssex,sbirthday,sclass) values(105,'匡明','男',to_date('1975-10-02','yyyy-mm-dd'),95031);

1 row created.

SQL> commit;

Commit complete.

SQL> insert into student(sno,sname,ssex,sbirthday,sclass) values(107,'王丽','女',to_date('1976-01-23','yyyy-mm-dd'),95033);

1 row created.

SQL> commit;

Commit complete.

SQL> insert into student(sno,sname,ssex,sbirthday,sclass) values(109,'王芳','女',to_date('1975-02-10','yyyy-mm-dd'),95031);

1 row created.

SQL> commit;

Commit complete.

SQL> select * from student; 

       SNO SNAME		       SSEX	 SBIRTHDAY	 SCLASS
---------- --------------------------- --------- --------------- ---------------
       108 曾华                        男        01-SEP-77       95033
       105 匡明                        男        02-OCT-75       95031
       107 王丽                        女        23-JAN-76       95033
       109 王芳                        女        10-FEB-75       95031

验证数据是否同步
在这里插入图片描述

验证update更改数据是否同步

SQL> UPDATE student SET SNAME='UPDATE' WHERE SNO='108';

1 row updated.

SQL> commit;

Commit complete.

SQL> select * from student; 

       SNO SNAME		       SSEX	 SBIRTHDAY	 SCLASS
---------- --------------------------- --------- --------------- ---------------
       108 UPDATE                      男        01-SEP-77       95033
       105 匡明                        男        02-OCT-75       95031
       107 王丽                        女        23-JAN-76       95033
       109 王芳                        女        10-FEB-75       95031

验证更改是否同步
在这里插入图片描述

验证delete更改数据是否同步

SQL> DELETE FROM student WHERE SNO='105';

1 row deleted.

SQL> commit;

Commit complete.

SQL> select * from student; 

       SNO SNAME		       SSEX	 SBIRTHDAY	 SCLASS
---------- --------------------------- --------- --------------- ---------------
       108 UPDATE                      男        01-SEP-77       95033
       107 王丽                        女        23-JAN-76       95033
       109 王芳                        女        10-FEB-75       95031

验证更改是否同步
在这里插入图片描述
验证alter增加字段是否同步

SQL> ALTER TABLE student ADD (age integer default 22 not null);

Table altered.

SQL> commit;

Commit complete.
SQL> select * from student;

       SNO SNAME		       SSEX	 SBIRTHDAY	 SCLASS
---------- --------------------------- --------- --------------- ---------------
       AGE
----------
       108 UPDATE                      男        01-SEP-77       95033
	22

       107 王丽                        女        23-JAN-76       95033
	22

       109 王芳                        女        10-FEB-75       95031
	22

kafka-connect报错
在这里插入图片描述
验证更改是否同步
在这里插入图片描述

报错处理

连接器报错

在这里插入图片描述

解决

1、按照提示，执行命令，打开报错的表的全体字段补充日志

SQL>  ALTER TABLE TEST_OGG.TEST_OGG ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;

Table altered.

2、直接打开全体字段补充日志

SQL> alter database add supplemental log data (all) columns; 

Database altered.

select SUPPLEMENTAL_LOG_DATA_MIN min,
       SUPPLEMENTAL_LOG_DATA_PK  pk,
       SUPPLEMENTAL_LOG_DATA_UI  ui,
       SUPPLEMENTAL_LOG_DATA_FK  fk,
       SUPPLEMENTAL_LOG_DATA_ALL "all"
from v$database;
MIN			 PK	   UI	     FK        all
------------------------ --------- --------- --------- ---------
YES			 NO	   NO	     NO        YES

不能加载插件错误

在这里插入图片描述

解决

debezium-connector-oracle下的jar包复制到kafka的libs目录下
cp /opt/debezium-connector-oracle/* /opt/kafka_2.13-2.8.0/libs/

时空无限
关注
  • 6
    点赞
  • 踩
  • 27
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 11
    评论
Debezium同步之oracle数据到Kafka的同步
Auspicious航的博客
04-12 1275
目录 一、概述 二、设置oracle 2.1 与 Oracle 安装类型的兼容性 2.2 准备数据库 2.3 重做日志大小 2.4 为连接器创建用户 2.5 备用数据库 2.6 故障转移数据库 三、部署 3.1 Debezium Oracle 连接器配置 3.2 可插拔与不可插拔数据库 3.3 添加连接器配置 一、概述 Debezium 的 Oracle 连接器捕获并记录发生在 Oracle 服务器上的数据库中的行级更改，包括在连接器运行...
OGG增量抽取Oracle业务数据到kafka-部署手册.doc
04-26
Kafka 目标端 OGG 安装用于将数据从 Oracle 数据库抽取到 Kafka 中，而 Oracle 目标端 OGG 安装用于将数据从 Oracle 数据库抽取到 Oracle 数据库中。 OGG 源端配置 OGG 源端配置包括源端 Oracle 数据库配置和源端 ...
springboot整合Debezium监控Oracle数据库
weixin_41085114的博客
10-17 1168
debezium
Debezium+Kafka：Oracle 11g 数据实时同步至 DolphinDB 解决方案
最新发布
DolphinDB智臾科技
05-24 907
本篇教程将介绍使用 Debezium 来实时捕获和发布 Oracle 11g 的数据库更改事件，并完成 Oracle 到 DolphinDB 的实时数据同步的完整解决方案。
oracle Debezium
杨晓尚
10-20 2998
Debezium是什么 Debezium是一个开源项目，为捕获数据更改(change data capture,CDC)提供了一个低延迟的流式处理平台。你可以安装并且配置Debezium去监控你的数据库，然后你的应用就可以消费对数据库的每一个行级别(row-level)的更改。只有已提交的更改才是可见的，所以你的应用不用担心事务(transaction)或者更改被回滚(roll back)。 Debezium连接oracle方式 Debezium提供了两种监控数据库的方式，对应了oracle的两种连接方式。
Debezium日常分享系列之：Oracle数据库系列文章之安装和设置
zhengzaifeidelushang的博客
07-09 969
Debezium系列之：Oracle数据库系列文章之安装和设置
Debezium迁移和同步Oracle数据
fayeestone的博客
10-19 4813
Debezium迁移和同步Oracle数据概览特点部署先决条件部署流程快照connector默认流程--创建初始快照即席快照增量快照算法实现kafka topic对事务的支持数据变化事件keyvalue 概览 Debezium是RedHat开源的一个将多种数据源实时变更数据捕获，形成数据流输出的开源工具。当前支持MySQL、PostgreSQL、SQL Server、Oracle、Db2等常用数据库。Debezium Oracle connector是基于kafka conect的一种实现，用于捕获并记录O
基于Debezium 1.6和Oracle 11g 的 Debezium-Oracle实战
qq_35492176的博客
05-16 3646
一、应用场景 1、事件触发 数据库的触发器（trigger）能够一定程度上实现我们需要的事件触发，但通常需要配合定时任务才能实现。 2、数据同步 典型的解决方法有 • 双写 • 变更数据抓取（change data capture, CDC） 其中变更数据抓取既能解决事件触发的问题也可以很好的解决数据同步的问题 二、Debezium Debezium 是 Redhat 开源的数据变更抓取组件，其利用了 Kafka Connect 所以拥有高可用与开箱即用的调度接口，Debezium的Snapshot M
kafka+flume 实时采集oracle数据到hive中.docx
08-11
基于Kafka+Flume实时采集Oracle数据到Hive中 一、Kafka获取Oracle...Kafka能够实时地从Oracle数据库中提取日志信息，而Flume能够实时地将数据写入到HDFS中。最后，Hive可以从HDFS中读取数据，并将其存储到Hive表中。
Flume 抽取MYSQL Oracle数据 JSON格式 推送Kafka
05-28
Flume二次开发，支持抽取MYSQL Oracle数据库数据 以JSON格式推送至Kafka。 demo： sql_json.sources.sql_source.type = com.hbn.rdb.source.SQLSource sql_json.sources.sql_source.connectionurl = jdbc:oracle:...
Oracle Golden Gate 从Oracle同步到Kafka的部署文档
12-12
此文档包含了 Oracle Golden Gate 从Windows下Oracle同步到Linux下的Kafka的部署说明。
Kafka实现ORACLE实时数据处理.docx
06-11
首先，需要安装 Logminer 工具，然后使用 Kafka Connect 的 OracleSourceConnector 将数据从 ORACLE 传输到 Kafka 中。接着，需要启动 Kafka 服务器，然后创建一个 Topic。最后，需要启动 Connector，使用 ...
Debezium 抽取oracle数据
水彩橘子
09-07 2159
1、环境介绍 操作系统：centos 7.9 jdk版本：11.0.12 kafka版本：2.8.0 Debezium版本：1.6（debezium-connector-oracle-1.6.1.Final-plugin.tar.gz） oracle版本：19c 2、安装oracle https://blog.csdn.net/zyj81092211/article/details/120082828 3、设置oracle (1)创建目录 mkdir /u01/app/oracle/oradata/reco
基于debezium实时数据同步（Oracle为主）
吃素的哈士奇de博客
09-03 4034
基于debezium实时数据同步全部需要下载的内容链接1、下载zookeeper-3.4.102、下载kafka_2.13-2.8.03、下载Kafka Connector：建议使用1.6以上版本可以对ddl进行捕获4、安装debezium-connector-oracle4.1下载debezium-connector-oracle-1.6.0.Final-plugin.tar.gz并解压，安装在自己的服务器，我的安装目录是/home/debezium/4.2、将debezium-connector-ora
Debezium实时抽取oracle 19c数据到kafka
m0_59595182的博客
06-25 830
"table.include.list": "BCM.CUST_CORP_INFO,BCM.CR_FAREP_MANAGE_CHILD,BCM.CR_FAREP_MANAGE_MAIN", //要抽取的表。"database.history.kafka.bootstrap.servers" : "xx.xx.xx.xx:9092", //生成的topic放到哪个kakfa中。"database.history.skip.unparseable.ddl":"true", //跳过ddl语句。
使用kafka connect结合debezium的方式采集oracle数据
雾岛与鲸的博客
04-18 4107
环境配置: 1、kafka 2.8.0 2、debezium 1.9.0 3、oracle 11g 1、oracle环境配置 该步骤参考我的上篇文章 https://blog.csdn.net/qq_36039236/article/details/124235751?spm=1001.2014.3001.5502 本章节继续针对FAMILY.STUDENT_INFO表进行操作。 2、kafka connect配置 kafka内部集成了kafka connect, 只需要下载对应的debezium-orac
debezium to oracle 11g 实时同步
weixin_40898246的博客
10-21 3304
第1章 debezium概述 1.1 debezium是什么 Debezium 是一组分布式服务，用于捕获数据库中的更改，以便您的应用程序可以查看这些更改并对其做出响应。 Debezium 在更改事件流中记录每个数据库表中的所有行级更改，应用程序只需读取这些流以查看更改事件发生的相同顺序。 1.2 debezium的依赖 Debezium 构建在 Apache Kafka 之上，并提供与 Kafka Connect 兼容的连接器，用于监控特定的数据库管理系统。 Debezium 在 Kafka 日志中记录数
Debezium日常分享系列之：Debezium2.5稳定版本之Oracle连接器的工作原理
zhengzaifeidelushang的博客
03-24 986
Debezium 的 Oracle 连接器捕获并记录 Oracle 服务器上的数据库中发生的行级更改，包括连接器运行时添加的表。您可以将连接器配置为针对架构和表的特定子集发出更改事件，或者忽略、屏蔽或截断特定列中的值。Debezium 通过使用本机 LogMiner 数据库包、XStream API 或 OpenLogReplicator 从 Oracle 获取更改事件。Debezium技术专栏。
Debezium日常分享系列之：Debezium2.5稳定版本之Oracle数据库的查询模式、事件缓冲、SCN间隙检测、低变化频率偏移管理
zhengzaifeidelushang的博客
03-25 1112
Debezium日常分享系列之：Debezium2.5稳定版本之Oracle数据库的查询模式、事件缓冲、SCN间隙检测、低变化频率偏移管理 Debezium技术专栏

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 11
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

时空无限

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值