debezium-connect-oracle使用

已于 2022-04-01 11:08:30 修改
阅读量4.7k 收藏 9
点赞数 1
文章标签： oracle 数据库 database
于 2022-01-25 14:36:47 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_41600330/article/details/122684748
版权

测试环境：

​ a.数据库：Oracle19c 19.3 已开归档日志

​ b.Kafka：2.8，内置zookeeper

​ c.连接器：debezium-connector-oracle-1.6.2.Final-plugin.tar

​ d.windows10

1.创建用户赋权

SQL> show con_name;

CON_NAME
------------------------------
CDB$ROOT
SQL> CREATE TABLESPACE logminer_tbs DATAFILE 'C:\oracle\app\oradata\ORCL\logminer_tbs.dbf'
  2      SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;

--表空间已创建。

SQL> alter session set container=orclpdb;

--会话已更改。

SQL> show con_name;

CON_NAME
------------------------------
ORCLPDB
SQL> CREATE TABLESPACE logminer_tbs DATAFILE 'C:\oracle\app\oradata\ORCL\orclpdb\logminer_tbs.dbf'
  2      SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED;

--表空间已创建。

SQL> alter session set container =CDB$ROOT;

--会话已更改。
SQL> show con_name;

CON_NAME
------------------------------
CDB$ROOT
SQL> CREATE USER c##dbzuser IDENTIFIED BY dbz
  2      DEFAULT TABLESPACE logminer_tbs
  3      QUOTA UNLIMITED ON logminer_tbs
  4      CONTAINER=ALL;

--用户已创建。
  GRANT CREATE SESSION TO c##dbzuser CONTAINER=ALL;
  GRANT SET CONTAINER TO c##dbzuser CONTAINER=ALL;
  GRANT SELECT ON V_$DATABASE to c##dbzuser CONTAINER=ALL;
  GRANT FLASHBACK ANY TABLE TO c##dbzuser CONTAINER=ALL;
  GRANT SELECT ANY TABLE TO c##dbzuser CONTAINER=ALL;
  GRANT SELECT_CATALOG_ROLE TO c##dbzuser CONTAINER=ALL;
  GRANT EXECUTE_CATALOG_ROLE TO c##dbzuser CONTAINER=ALL;
  GRANT SELECT ANY TRANSACTION TO c##dbzuser CONTAINER=ALL;
  GRANT LOGMINING TO c##dbzuser CONTAINER=ALL;

  GRANT CREATE TABLE TO c##dbzuser CONTAINER=ALL;
  GRANT LOCK ANY TABLE TO c##dbzuser CONTAINER=ALL;
  GRANT ALTER ANY TABLE TO c##dbzuser CONTAINER=ALL;
  GRANT CREATE SEQUENCE TO c##dbzuser CONTAINER=ALL;

  GRANT EXECUTE ON DBMS_LOGMNR TO c##dbzuser CONTAINER=ALL;
  GRANT EXECUTE ON DBMS_LOGMNR_D TO c##dbzuser CONTAINER=ALL;

  GRANT SELECT ON V_$LOG TO c##dbzuser CONTAINER=ALL;
  GRANT SELECT ON V_$LOG_HISTORY TO c##dbzuser CONTAINER=ALL;
  GRANT SELECT ON V_$LOGMNR_LOGS TO c##dbzuser CONTAINER=ALL;
  GRANT SELECT ON V_$LOGMNR_CONTENTS TO c##dbzuser CONTAINER=ALL;
  GRANT SELECT ON V_$LOGMNR_PARAMETERS TO c##dbzuser CONTAINER=ALL;
  GRANT SELECT ON V_$LOGFILE TO c##dbzuser CONTAINER=ALL;
  GRANT SELECT ON V_$ARCHIVED_LOG TO c##dbzuser CONTAINER=ALL;
  GRANT SELECT ON V_$ARCHIVE_DEST_STATUS TO c##dbzuser CONTAINER=ALL;

2.创建测试表，对该表开启归档日志、附加日志

C:\Users\CHEN_he>sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on 星期五 1月 21 13:17:20 2022
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


连接到:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> conn c##dbzuser/dbz;
已连接。
SQL> CREATE TABLE STU ( "s_id" INT PRIMARY KEY, "s_name" VARCHAR ( 255 ) );

表已创建。

SQL> ALTER TABLE C##DBZUSER.STU ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;

表已更改。

3.修改kafka配置文件

vi config/connect-dis...
#添加连接器插件路径，即上边解压的路径，windows下，路径的 \ 要转义为 \\ ,否则启动连接器回报错invalid plugins.path  
plugin.path=debezium-connect-oracle

4.复制所需jar包（Oracle 必须为ojdbc8）

#解压连接器插件，复制jar包到kafka/libs目录下
cp debezium-connect-oracle/*.jar  $kafka_home/libs/

5.启动

./bin/zookeeper-server-start.sh config/zookeeper.con...
./bin/kafka-server-start.sh con.....
./bin/connect-dis.....  con/connect-dis...

6.查看连接器注册情况

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/

查看现有连接器

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d

7.注册连接器

命令行容易出错（建议用postman）

{
    "name": "test_dbz",  // 连接器的名称
    "config": {
        "connector.class" : "io.debezium.connector.oracle.OracleConnector",  
        "database.hostname" : "192.168.211.3",  
        "database.port" : "1521",  
        "database.user" : "c##dbzuser",  
        "database.password" : "dbz",   
        "database.dbname" : "ORCLCDB",  //数据库的名称
        "database.server.name" : "server2",  //服务名
        "tasks.max" : "1",    //连接器创建的最大任务数
        "database.pdb.name" : "ORCLPDB",  //连接器从中捕获更改的 Oracle 可插拔数据库的名称，不是必须的，但是配置上后，pdb需打开
        "table.include.list": "C##DBZUSER.STU",
        "database.history.kafka.bootstrap.servers" : "localhost:9092", //kafka服务地址
        "database.history.kafka.topic": "schema-changes.stu" , //连接器写入和恢复 DDL 语句的数据库历史主题的名称。本主题仅供内部使用，消费者不得使用。
        "decimal.handling.mode": "string", //number类型乱码解决
        "log.mining.strategy": "online_catalog",//读取redolog,redolog写满才会生成归档日志，导致topic接收数据慢
        "snapshot.mode": "schema_only"//取消快照

    }
}

[root@localhost ~]# curl -H "Accept:application/json" localhost:8083/
{"version":"2.8.0","commit":"ebb1d6e21cc92130","kafka_cluster_id":"5m7EcwSpTQ2qQMSGCfZY2w"}
[root@localhost ~]# curl -H "Accept:application/json" localhost:8083/connectors/
[]
[root@localhost ~]# curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '{"name": "stu2","config": {"connector.class": "io.debezium.connector.oracle.OracleConnector","tasks.max": "1","database.server.name": "server2","database.hostname": "192.168.211.3","database.port": "1521","database.user": "c##dbzuser","database.password": "dbz","database.dbname": "ORCL","table.include.list": "C##DBZUSER.STU","database.history.kafka.bootstrap.servers": "localhost:9092","database.history.kafka.topic": "schema-changes.stu"}}'
HTTP/1.1 500 Internal Server Error
Date: Fri, 21 Jan 2022 05:39:16 GMT
Content-Type: application/json
Content-Length: 48
Server: Jetty(9.4.39.v20210325)

{"name": "stu2","config": {"connector.class": "io.debezium.connector.oracle.OracleConnector","tasks.max": "1","database.server.name": "server2","database.hostname": "192.168.211.3","database.port": "1521","database.user": "c##dbzuser","database.password": "dbz","database.dbname": "ORCL","table.include.list": "C##DBZUSER.STU","database.history.kafka.bootstrap.servers": "localhost:9092","database.history.kafka.topic": "schema-changes.stu"}}

8.查看连接器状态

有报错会在下方body中输出

在这里插入图片描述

9.查看topic

C:\Kafka\kafka_2.12-2.8.0>bin\windows\kafka-topics.bat --bootstrap-server localhost:9092 --list
__consumer_offsets
connect-configs
connect-offsets
connect-status
schema-changes.test_dbz
server2
server2.C__DBZUSER.STU  ---创建主题时，## 会被自动转换成 __

10.查看消费者消息

在这里插入图片描述

注意！！！

测试发现，连接器对于这几个用户的变更也会被捕获到

在这里插入图片描述

删除连接器

curl -s http://192.168.211.3:8083/connectors/TEST_MORE -X DELETE

在这里插入图片描述

在这里插入图片描述

{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"before"},{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":true,"field":"sequence"},{"type":"string","optional":false,"field":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"string","optional":true,"field":"txId"},{"type":"string","optional":true,"field":"scn"},{"type":"string","optional":true,"field":"commit_scn"},{"type":"string","optional":true,"field":"lcr_position"}],"optional":false,"name":"io.debezium.connector.oracle.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"optional":false,"name":"server2.PDBUSER.TESTPDB.Envelope"},"payload":{"before":null,"after":{"ID":{"scale":0,"value":"GA=="},"NAME":"kobe"},"source":{"version":"1.6.2.Final","connector":"oracle","name":"server2","ts_ms":1642932301452,"snapshot":"true","db":"ORCLPDB","sequence":null,"schema":"PDBUSER","table":"TESTPDB","txId":null,"scn":"3278144","commit_scn":null,"lcr_position":null},"op":"r","ts_ms":1642932301452,"transaction":null}}
{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"before"},{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":true,"field":"sequence"},{"type":"string","optional":false,"field":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"string","optional":true,"field":"txId"},{"type":"string","optional":true,"field":"scn"},{"type":"string","optional":true,"field":"commit_scn"},{"type":"string","optional":true,"field":"lcr_position"}],"optional":false,"name":"io.debezium.connector.oracle.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"optional":false,"name":"server2.PDBUSER.TESTPDB.Envelope"},"payload":{"before":null,"after":{"ID":{"scale":0,"value":"Fw=="},"NAME":"jodan"},"source":{"version":"1.6.2.Final","connector":"oracle","name":"server2","ts_ms":1642932301453,"snapshot":"true","db":"ORCLPDB","sequence":null,"schema":"PDBUSER","table":"TESTPDB","txId":null,"scn":"3278144","commit_scn":null,"lcr_position":null},"op":"r","ts_ms":1642932301453,"transaction":null}}
{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"before"},{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":true,"field":"sequence"},{"type":"string","optional":false,"field":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"string","optional":true,"field":"txId"},{"type":"string","optional":true,"field":"scn"},{"type":"string","optional":true,"field":"commit_scn"},{"type":"string","optional":true,"field":"lcr_position"}],"optional":false,"name":"io.debezium.connector.oracle.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"optional":false,"name":"server2.PDBUSER.TESTPDB.Envelope"},"payload":{"before":null,"after":{"ID":{"scale":0,"value":"FA=="},"NAME":"ccc"},"source":{"version":"1.6.2.Final","connector":"oracle","name":"server2","ts_ms":1642932301453,"snapshot":"true","db":"ORCLPDB","sequence":null,"schema":"PDBUSER","table":"TESTPDB","txId":null,"scn":"3278144","commit_scn":null,"lcr_position":null},"op":"r","ts_ms":1642932301453,"transaction":null}}
{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"before"},{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":true,"field":"sequence"},{"type":"string","optional":false,"field":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"string","optional":true,"field":"txId"},{"type":"string","optional":true,"field":"scn"},{"type":"string","optional":true,"field":"commit_scn"},{"type":"string","optional":true,"field":"lcr_position"}],"optional":false,"name":"io.debezium.connector.oracle.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"optional":false,"name":"server2.PDBUSER.TESTPDB.Envelope"},"payload":{"before":null,"after":{"ID":{"scale":0,"value":"FQ=="},"NAME":"ccc"},"source":{"version":"1.6.2.Final","connector":"oracle","name":"server2","ts_ms":1642932301453,"snapshot":"true","db":"ORCLPDB","sequence":null,"schema":"PDBUSER","table":"TESTPDB","txId":null,"scn":"3278144","commit_scn":null,"lcr_position":null},"op":"r","ts_ms":1642932301453,"transaction":null}}
{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"before"},{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":true,"field":"sequence"},{"type":"string","optional":false,"field":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"string","optional":true,"field":"txId"},{"type":"string","optional":true,"field":"scn"},{"type":"string","optional":true,"field":"commit_scn"},{"type":"string","optional":true,"field":"lcr_position"}],"optional":false,"name":"io.debezium.connector.oracle.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"optional":false,"name":"server2.PDBUSER.TESTPDB.Envelope"},"payload":{"before":null,"after":{"ID":{"scale":0,"value":"AQ=="},"NAME":"CH"},"source":{"version":"1.6.2.Final","connector":"oracle","name":"server2","ts_ms":1642932301453,"snapshot":"true","db":"ORCLPDB","sequence":null,"schema":"PDBUSER","table":"TESTPDB","txId":null,"scn":"3278144","commit_scn":null,"lcr_position":null},"op":"r","ts_ms":1642932301453,"transaction":null}}
{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"before"},{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":true,"field":"sequence"},{"type":"string","optional":false,"field":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"string","optional":true,"field":"txId"},{"type":"string","optional":true,"field":"scn"},{"type":"string","optional":true,"field":"commit_scn"},{"type":"string","optional":true,"field":"lcr_position"}],"optional":false,"name":"io.debezium.connector.oracle.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"optional":false,"name":"server2.PDBUSER.TESTPDB.Envelope"},"payload":{"before":null,"after":{"ID":{"scale":0,"value":"Ag=="},"NAME":"CHh"},"source":{"version":"1.6.2.Final","connector":"oracle","name":"server2","ts_ms":1642932301453,"snapshot":"true","db":"ORCLPDB","sequence":null,"schema":"PDBUSER","table":"TESTPDB","txId":null,"scn":"3278144","commit_scn":null,"lcr_position":null},"op":"r","ts_ms":1642932301454,"transaction":null}}
{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"before"},{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":true,"field":"sequence"},{"type":"string","optional":false,"field":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"string","optional":true,"field":"txId"},{"type":"string","optional":true,"field":"scn"},{"type":"string","optional":true,"field":"commit_scn"},{"type":"string","optional":true,"field":"lcr_position"}],"optional":false,"name":"io.debezium.connector.oracle.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"optional":false,"name":"server2.PDBUSER.TESTPDB.Envelope"},"payload":{"before":null,"after":{"ID":{"scale":0,"value":"Cg=="},"NAME":"qqq"},"source":{"version":"1.6.2.Final","connector":"oracle","name":"server2","ts_ms":1642932301454,"snapshot":"true","db":"ORCLPDB","sequence":null,"schema":"PDBUSER","table":"TESTPDB","txId":null,"scn":"3278144","commit_scn":null,"lcr_position":null},"op":"r","ts_ms":1642932301454,"transaction":null}}
{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"before"},{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":true,"field":"sequence"},{"type":"string","optional":false,"field":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"string","optional":true,"field":"txId"},{"type":"string","optional":true,"field":"scn"},{"type":"string","optional":true,"field":"commit_scn"},{"type":"string","optional":true,"field":"lcr_position"}],"optional":false,"name":"io.debezium.connector.oracle.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"optional":false,"name":"server2.PDBUSER.TESTPDB.Envelope"},"payload":{"before":null,"after":{"ID":{"scale":0,"value":"Cg=="},"NAME":"qqq"},"source":{"version":"1.6.2.Final","connector":"oracle","name":"server2","ts_ms":1642932301454,"snapshot":"true","db":"ORCLPDB","sequence":null,"schema":"PDBUSER","table":"TESTPDB","txId":null,"scn":"3278144","commit_scn":null,"lcr_position":null},"op":"r","ts_ms":1642932301454,"transaction":null}}
{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"before"},{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":true,"field":"sequence"},{"type":"string","optional":false,"field":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"string","optional":true,"field":"txId"},{"type":"string","optional":true,"field":"scn"},{"type":"string","optional":true,"field":"commit_scn"},{"type":"string","optional":true,"field":"lcr_position"}],"optional":false,"name":"io.debezium.connector.oracle.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"optional":false,"name":"server2.PDBUSER.TESTPDB.Envelope"},"payload":{"before":null,"after":{"ID":{"scale":0,"value":"Cg=="},"NAME":"qqq"},"source":{"version":"1.6.2.Final","connector":"oracle","name":"server2","ts_ms":1642932301454,"snapshot":"true","db":"ORCLPDB","sequence":null,"schema":"PDBUSER","table":"TESTPDB","txId":null,"scn":"3278144","commit_scn":null,"lcr_position":null},"op":"r","ts_ms":1642932301454,"transaction":null}}
{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"before"},{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":true,"field":"sequence"},{"type":"string","optional":false,"field":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"string","optional":true,"field":"txId"},{"type":"string","optional":true,"field":"scn"},{"type":"string","optional":true,"field":"commit_scn"},{"type":"string","optional":true,"field":"lcr_position"}],"optional":false,"name":"io.debezium.connector.oracle.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"optional":false,"name":"server2.PDBUSER.TESTPDB.Envelope"},"payload":{"before":null,"after":{"ID":{"scale":0,"value":"Cg=="},"NAME":"qqq"},"source":{"version":"1.6.2.Final","connector":"oracle","name":"server2","ts_ms":1642932301454,"snapshot":"last","db":"ORCLPDB","sequence":null,"schema":"PDBUSER","table":"TESTPDB","txId":null,"scn":"3278144","commit_scn":null,"lcr_position":null},"op":"r","ts_ms":1642932301454,"transaction":null}}
{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"before"},{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"scale"},{"type":"bytes","optional":false,"field":"value"}],"optional":true,"name":"io.debezium.data.VariableScaleDecimal","version":1,"doc":"Variable scaled decimal","field":"ID"},{"type":"string","optional":true,"field":"NAME"}],"optional":true,"name":"server2.PDBUSER.TESTPDB.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":true,"field":"sequence"},{"type":"string","optional":false,"field":"schema"},{"type":"string","optional":false,"field":"table"},{"type":"string","optional":true,"field":"txId"},{"type":"string","optional":true,"field":"scn"},{"type":"string","optional":true,"field":"commit_scn"},{"type":"string","optional":true,"field":"lcr_position"}],"optional":false,"name":"io.debezium.connector.oracle.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"optional":false,"name":"server2.PDBUSER.TESTPDB.Envelope"},"payload":{"before":null,"after":{"ID":{"scale":0,"value":"Hg=="},"NAME":"CCC"},"source":{"version":"1.6.2.Final","connector":"oracle","name":"server2","ts_ms":1642961344000,"snapshot":"false","db":"ORCLPDB","sequence":null,"schema":"PDBUSER","table":"TESTPDB","txId":"09001800a1020000","scn":"3630790","commit_scn":"3635039","lcr_position":null},"op":"c","ts_ms":1642933132429,"transaction":null}}

报错：

未开启表的附加日志

{
    "name": "TEST_MORE",
    "connector": {
        "state": "RUNNING",
        "worker_id": "192.168.211.3:8083"
    },
    "tasks": [
        {
            "id": 0,
            "state": "FAILED",
            "worker_id": "192.168.211.3:8083",
            "trace": "org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.\r\n\tat io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)\r\n\tat io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:211)\r\n\tat io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:63)\r\n\tat io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:159)\r\n\tat io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:122)\r\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\r\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\tat java.lang.Thread.run(Thread.java:748)\r\nCaused by: io.debezium.DebeziumException:
************************************************* 
Supplemental logging not configured for table ORCL.C##DBZUSER.TEST2.  Use command: ALTER TABLE C##DBZUSER.TEST2 ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS\r\n\tat 
**************************************************
io.debezium.connector.oracle.logminer.LogMinerHelper.checkSupplementalLogging(LogMinerHelper.java:408)\r\n\tat io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:130)\r\n\t... 8 more\r\n"
        }
    ],
    "type": "source"
}

未开启pdb会报可插拔数据库未打开错误

alter pluggable database orclpdb open;
洛日飞车
关注
  • 1
    点赞
  • 踩
  • 9
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
Debezium-oracle-connector 1.9.7 源码分析
雾岛与鲸的博客
03-10 1222
由于生产上需要用到debezium采集oracle的数据，所以本着用一个工具就需要了解其运行原理的初衷，对debezium相关的源码进行分析。 好了，直接开始吧。 首先，我们需要通过一段代码找到debezium执行的入口。 相关的pom依赖:
springboot-kafka-connect-debezium-ksqldb:该项目的目标是与Kafka，Debezium和ksqlDB一起使用。 为此，我们有
04-04
springboot-kafka-连接-debezium-ksqldb 该项目的目标是与 ， 和 。 为此，我们有： research-service ，用于在中插入/更新/删除记录； Source Connectors ，用于监视MySQL中记录的更改并将与这些更改相关的消息推送...
Debezium实战-第4章 连接器-Oracle
Kettle实战
04-29 643
4.6 Oracle 适用于 Oracle 的 Debezium 连接器 概述 Debezium 的 Oracle 连接器捕获并记录发生在 Oracle 服务器上的数据库中的行级更改，包括在连接器运行时添加的表。您可以将连接器配置为针对特定的schemas和表的子集发出更改事件，或者忽略、屏蔽或截断特定列中的值。 有关与此连接器兼容的 Oracle 数据库版本的信息，请参阅Debezium 版本概述。 Debezium 通过使用本机 LogMiner 数据库包或XStream API从 Oracle 获取更
Debezium分享系列之：Debezium2.6稳定版本Oracle数据库Debezium Connector的属性参数详解
最新发布
zhengzaifeidelushang的博客
04-18 207
可以通过将值分配给一组以 schema.history.internal. Producer.* 和 schema.history.internal.consumer.* 前缀开头的传递配置属性来定义 Kafka 生产者和消费者客户端的配置。Debezium 提供了一组 schema.history.internal.* 属性，用于控制连接器如何与架构历史主题交互。与数据库模式历史客户端的传递属性的情况一样，Debezium 在将属性传递到数据库驱动程序之前会从属性中去除前缀​​。下表描述了通知属性。
meethigher-配置Debezium Connector for Oracle
meethigher的博客
10-13 148
配置Debezium Connector for Oracle 一、踩坑收藏 环境 操作系统：centos7.9 oracle版本：oracle-database-ee-19c-1.0-1.x86_64.rpm zookeeper版本：apache-zookeeper-3.7.0-bin.tar.gz kafka版本：kafka_2.12-2.7.0.tgz 参考文章 Debezium Connector for Oracle :: Debezium Documentation Apache Kafk
Debezium系列之：Debezium2.X之Oracle数据库的Debezium连接器
zhengzaifeidelushang的博客
05-26 2000
Debezium 的 Oracle 连接器捕获并记录在 Oracle 服务器上的数据库中发生的行级更改，包括在连接器运行时添加的表。可以将连接器配置为针对模式和表的特定子集发出更改事件，或者忽略、屏蔽或截断特定列中的值。Debezium 通过使用本机 LogMiner 数据库包或XStream API从 Oracle 获取更改事件。虽然该连接器可能适用于各种 Oracle 版本，但只有 Oracle EE 12 和 19 已经过测试。Oracle数据库实战链接后续会补上。
使用Pulsar+Debezium Connector获取oracle数据变更
zheng20090514的博客
12-14 1235
所需软件及版本信息： apache-pulsar-2.9.0-bin.tar.gz pulsar-io-debezium-oracle-2.9.0.nar instantclient-basic-linux.x64-19.13.0.0.0dbru.zip oracle 11gR2 下载：apache-pulsar-2.9.0-bin.tar.gz　pulsar-io-debezium-oracle-2.9.0.nar oracle软件安装略。 配置Oracle CDC（XStream） ==.
kafka-connect-oracle:Kafka Source Connector for Oracle
05-28
Kafka ConnectOraclekafka-connect-oracle是一个Kafka源连接器，用于从Oracle数据库捕获所有基于行的DML更改并将这些更改流式传输到Kafka。 变更数据捕获逻辑基于Oracle LogMiner解决方案。 仅从Oracle中提取已提交...
debezium-connector-oracle-1.6.0.Alpha1-plugin.tar.gz
06-22
debezium-connector-oracle
kafka-connect-oracle-1.0.71.jar
06-29
2021年6.29号打包，最新版本：kafka-connect-oracle-1.0.71.jar，用于根据oracle binglog日志同步数据到kafka
cp-server-connect-operator-debezium-mysql:带有debezium的cp-server-connect-operator
03-30
cp服务器连接操作符-debezium-mysql ConfluentInc的cp-server-connect-operator现在增加了105％的mysql debezium！
ORACLE LOGMINER DEBEZIUM
大蚊子
04-07 737
debezium 查询v$LOGMNR_CONTENTS时，添加了一些过滤条件，例如表、SCN、OPERATION_CODE等。debezium 源码相关logminer。4、操作数据库blablabla.....1、构建数据字典,（可跳过）logminer 捕获流程。附录： debezium。3、启动logminer。3、启动logminer。
使用debezium采集oracle数据报错: NonRelationalTableException: Table xxx is not a relational table
雾岛与鲸的博客
03-09 686
使用debezium采集oracle日志数据的时候报错: Table xxx is not a relational table 报错日志如下: ```java traceorg.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
Debezium迁移和同步Oracle数据
fayeestone的博客
10-19 4588
Debezium迁移和同步Oracle数据概览特点部署先决条件部署流程快照connector默认流程--创建初始快照即席快照增量快照算法实现kafka topic对事务的支持数据变化事件keyvalue 概览 Debezium是RedHat开源的一个将多种数据源实时变更数据捕获，形成数据流输出的开源工具。当前支持MySQL、PostgreSQL、SQL Server、Oracle、Db2等常用数据库。Debezium Oracle connector是基于kafka conect的一种实现，用于捕获并记录O
Debezium系列之：优化集群参数和支持debezium connector个性化设置
zhengzaifeidelushang的博客
07-18 851
Debezium系列之：优化集群参数和支持debezium connector个性化设置
基于debezium实时数据同步（Oracle为主）
吃素的哈士奇de博客
09-03 3860
基于debezium实时数据同步全部需要下载的内容链接1、下载zookeeper-3.4.102、下载kafka_2.13-2.8.03、下载Kafka Connector：建议使用1.6以上版本可以对ddl进行捕获4、安装debezium-connector-oracle4.1下载debezium-connector-oracle-1.6.0.Final-plugin.tar.gz并解压，安装在自己的服务器，我的安装目录是/home/debezium/4.2、将debezium-connector-ora
Debezium结合Kafka Connect实时捕获MySQL变更事件写入Elasticsearch实现搜索流程
weixin_33998125的博客
12-15 1509
前段时间写了MySql实时数据变更事件捕获kafka confluent之debezium,使用的是confluent整套的，接下来这篇将会介绍完整实战。 首先明确需求，公司订单数据越来越大，商户端和E端各种业务需求也越来越多查询越发复杂，我们想引进elasticsearch来实现查询和搜索。那么问题来了，实时更新的订单数据如何同步到es中，业务代码中insert或者update es中的i...
Debezium offset源码解析及回调代码
qq_42382336的博客
11-04 1693
对debezium offset存储流程源码简单分析，分享调整offset代码
六：Debezium本地环境搭建（1）-代码编译
al229的博客
04-20 442
这里需要手动导入两个jar包，oracle的驱动包和xstreams包，因为非开源，这里就不附了，请到oracle官网下载。debezium-build-parent 这个工程，可以注释掉该工程下pom不相关的工程，可以参考注释以下工程。编译成功即完成代码编译部分的工作，下一节讲下怎样启动本地代码，以便于debug跟踪。这里如果全部编译，可能编译不成功，因此，本次只编译oracle和基础包。切换到对应的代码分支，本次使用的是 v 1.8.1.final。待编译成功后，单独编译。
kafka写入hive debezium-json
08-19
首先，Kafka是一个分布式流处理平台，用于在高吞吐量的情况下处理大量的实时数据流。Hive是一个基于Hadoop的数据仓库工具，用于存储和查询大规模结构化和半结构化数据。 要将Kafka写入Hive，需要使用Debezium它是一个开源的、基于事件的可信变更数据捕获（CDC）平台。Debezium可以将数据库的变更事件转换为Kafka主题中的实时流，并且可以实时监控数据库中的变动。 要实现Kafka写入Hive，首先需要配置Debezium将数据库的变更事件连接到Kafka中。Debezium会以JSON格式将变更事件转换为Kafka消息，并将其写入指定的主题中。 然后，可以使用Kafka Connect来读取Kafka主题中的消息，并将其写入Hive中。Kafka Connect是Kafka提供的一组工具，用于将Kafka主题与外部数据系统进行连接。 在Kafka Connect中，可以使用HDFS Connector来将Kafka消息写入Hive。HDFS Connector会将Kafka消息转换为Hive支持的格式，并将其写入Hive表中。 需要配置HDFS Connector的连接器以指定Kafka主题、Hive表和目标位置。连接器会自动将Kafka消息转换为Hive表的列，并将其写入Hive表中的对应位置。 一旦连接器配置完成并启动，Kafka中的变更事件就会实时地写入Hive表中。可以通过查询Hive表来获取Kafka中的数据，并根据需要进行分析和处理。 总而言之，要将Kafka写入Hive，可以使用Debezium将数据库变更事件转换为Kafka消息，并使用Kafka Connect的HDFS Connector将Kafka消息写入Hive表中。这样就能实现将实时数据流从Kafka写入Hive的目的。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值