goldengate java adapter_GoldenGate File Adapter

可以配置ETL工具(OWB/DATASTAGE等)实现实时E-LT功能。

组件：​

Data file：记录变更；ffwriter.properties控制 

Control file：表示datafile已经ready，可以触发ETL的批处理了。可以控制ETL以更小间隔进行处

理(micro batch) ​

Demo配置​

1. 环境准备

源数据库对应正常的OGG安装，另外需要一个DI服务器用于接收从日志抽取过来的数据并使用File

Adapter写成文件。(这个DI 服务器实际上是可以和源在同一台物理主机上的，建议是放在ETL系统中。)

介质准备：

1、 OGG ：正常的 OGG，下载对应数据库和操作系统的版本

2、 OGG File Adapter：对应这里DI服务器的操作系统，和源数据库无关。

2. 源端OGG配置

EXTRACT ext_adaSETENV (ORACLE_SID = "tianbaob") SETENV (NLS_LANG

= "AMERICAN_AMERICA.AL32UTF8") USERID ogg, PASSWORD oggEXTTRAIL

./dirdat/ta dynamicresolution table

tianbaobao.*;

ADD EXTRACT ext_ada, TRANLOG, BEGIN NOW ADD EXTTRAIL

./dirdat/ta, EXTRACT ext_ada, MEGABYTES 10

extract pmp_adapassthru rmthost localhost, mgrport 8809 rmttrail

./dirdat/ra dynamicresolution table tianbaobao.*;

add extract pmp_ada, EXTTRAILSOURCE ./dirdat/ta add rmttrail

./dirdat/ra, ext pmp_ada, MEGABYTES 10

同时需要使用defgen生成源端的定义文件

ggsci>edit param flatfile DEFSFILE

./dirdef/srcdef.def USERID ogg, PASSWORD oggTABLE

tianbaobao.*;

[oracle@tianbaobao fileadapter]$  ./defgen

paramfile dirprm/flatfile.prm

生成srcdef.def文件，ftp传输到目标DI

server的dirdef文件目录下。

3. 安装DI server上的OGG File Adapter

配置修改User Exit

properties文件属性这里我只修改了一条，Datafile记录格式：DSV、LDV。goldengate.flatfilewriter.writers=dsvwriter,ldvwriter

配置fileadapter进程参数

Extract ffwriterCUserExit ./flatfilewriter.so CUSEREXIT PassThru

IncludeUpdateBefores, PARAMS "dirprm/ffwriter.properties"sourcedefs

./dirdef/srcdef.defTable tianbaobao.*;

ADD EXTRACT ffwriter, EXTTRAILSOURCE dirdat/ra

4. 启动所有OGG进程

GGSCI (tianbaobao.ht8888.ht3) 10> info all

Program

Status

Group

Lag at Chkpt  Time Since Chkpt

MANAGER

RUNNING

JAGENT

STOPPED

EXTRACT

RUNNING

EXT_ADA

00:00:00

00:00:05

EXTRACT

RUNNING

PMP_ADA

00:00:00

00:00:09

GGSCI (tianbaobao.ht8888.ht3) 2> start

ffwriter

Sending START request to MANAGER ...EXTRACT FFWRITER

starting

GGSCI (tianbaobao.ht8888.ht3) 3> info all

Program

Status

Group

Lag at Chkpt  Time Since Chkpt

MANAGER

RUNNING

EXTRACT

RUNNING

FFWRITER

00:00:00

00:00:01

5. 源端DB做DML操作

TIANBAOBAO@tianbaob>insert into tb10 values

(12345);TIANBAOBAO@tianbaob>commit;TIANBAOBAO@tianbaob>update

tb10 set id=54321 where

id=12345;TIANBAOBAO@tianbaob>commit;TIANBAOBAO@tianbaob>delete

tb10 where id=54321;TIANBAOBAO@tianbaob>commit;

6. 查看flatfile的dirout目录下的文件

[oracle@tianbaobao fileadapter]$ cd dirout[oracle@tianbaobao

dirout]$ ls -tlr总用量 32-rw-rw-rw- 1 oracle

oinstall  82

5月   7 17:12

pump_TIANBAOBAO_TB10_2015-05-07_17-12-35_00000_data.dsv-rw-rw-rw- 1

oracle oinstall  84

5月   7 17:12

output_2015-05-07_17-12-35.data-rw-rw-rw- 1 oracle oinstall 255

5月   7 17:12

output_2015-05-07_17-12-35.stats-rw-rw-rw- 1 oracle oinstall 130

5月   7 17:13

TIANBAOBAO.TB10_data.control-rw-rw-rw- 1 oracle oinstall 164

5月   7 17:13

pump_TIANBAOBAO_TB10_2015-05-07_17-12-55_00001_data.dsv-rw-rw-rw- 1

oracle oinstall  82

5月   7 17:13

output.ctrl-rw-rw-rw- 1 oracle oinstall 255

5月   7 17:13

output_2015-05-07_17-12-55.stats-rw-rw-rw- 1 oracle oinstall 168

5月   7 17:13

output_2015-05-07_17-12-55.data

查看DSV格式：

[oracle@tianbaobao dirout]$ more

TIANBAOBAO.TB10_data.control./dirout/pump_TIANBAOBAO_TB10_2015-05-07_17-12-35_00000_data.dsv,./dirout/pump_TIANBAOBAO_TB10_2015-05-07_17-12-55_00001_data.dsv[oracle@tianbaobao

dirout]$ more

pump_TIANBAOBAO_TB10_2015-05-07_17-12-35_00000_data.dsv"000000000000044c"|"3"|"I"|"2015-05-07

17:12:29.000373"|"TIANBAOBAO"|"TB10"|12345[oracle@tianbaobao

dirout]$ more

pump_TIANBAOBAO_TB10_2015-05-07_17-12-55_00001_data.dsv"00000000000004cc"|"3"|"K"|"2015-05-07

17:12:49.000481"|"TIANBAOBAO"|"TB10"|54321"000000000000055b"|"3"|"D"|"2015-05-07

17:12:57.000169"|"TIANBAOBAO"|"TB10"|54321[oracle@tianbaobao

dirout]$

查看LDV格式：

[oracle@tianbaobao dirout]$ more

output.ctrl./dirout/output_2015-05-07_17-12-35.data,./dirout/output_2015-05-07_17-12-55.data[oracle@tianbaobao

dirout]$ more output_2015-05-07_17-12-55.data

TIANBAOBAOTIANBAOBAO[oracle@tianbaobao dirout]$ more

output_2015-05-07_17-12-55.stats|||2|2015-05-07

17:12:29.000373|2015-05-07 17:12:57.000169|2015-05-07

17:12:49.000481|2015-05-07

17:12:57.000169TIANBAOBAO|TB10|TIANBAOBAO.TB10|2|2015-05-07

17:12:29.000373|2015-05-07 17:12:57.000169|2015-05-07

17:12:49.000481|2015-05-07 17:12:57.000169[oracle@tianbaobao

dirout]$ more output_2015-05-07_17-12-35.data

TIANBAOBAO[oracle@tianbaobao dirout]$ more

output_2015-05-07_17-12-35.stats|||1|2015-05-07

17:12:29.000373|2015-05-07 17:12:29.000373|2015-05-07

17:12:29.000373|2015-05-07

17:12:29.000373TIANBAOBAO|TB10|TIANBAOBAO.TB10|1|2015-05-07

17:12:29.000373|2015-05-07 17:12:29.000373|2015-05-07

17:12:29.000373|2015-05-07 17:12:29.000373[oracle@tianbaobao

dirout]$

更多细节配置详见手册！

​