OGG TO HDFS,及Kerberose验证方法

最新推荐文章于 2021-04-03 08:22:21 发布

weixin_34161083

最新推荐文章于 2021-04-03 08:22:21 发布

阅读量612

点赞数

文章标签： python 操作系统大数据

原文链接：https://my.oschina.net/u/2503743/blog/761863

版权

2019独角兽企业重金招聘Python工程师标准>>>

一、准备工作

1、源端ORACLE RDBMS 12c,及Oracle GoldenGate 12c for Oracle。安装需要有图形界面，需要准备好VNC或其他。

2、目的端Oracle GoldenGate 12 for Big Data 12 ,HADOOP 2.6

二、安装步骤。

1、源端ORACLE数据库安装及GoldenGate安装都要用到./runInstaller。ORACLE安装忽略，本文安装在/data1/oracle/。OGG FOR ORACLE安装在/data/oracle/oggora。

2、源端 ORACLE需要做的配置。

$sqlplus / as sysdba

SQL>SELECT log_mode FROM v$database;

看看是否是归档模式，若不是要做如下操作。

SQL> alter database mount;

Database altered.

SQL> alter database archivelog;

Database altered.

SQL> alter database open;

SQL> alter database add supplemental log data;

Database altered.

SQL> alter database force logging;

Database altered.

SQL>SELECT log_mode FROM v$database;

LOG_MODE

------------

ARCHIVELOG

开启ORACLE中相关OGG的参数

SQL> alter system set enable_goldengate_replication=true sid='*' scope=both;

System altered.

创建ogg表空间，用户，并赋予DBA权值，不然，后面可能无法执行

SQL> CREATE TABLESPACE OGG DATAFILE '/data1/oracle/app/oracle/oradata/orcl/ogg.dbf' size 10M autoextend on;

SQL> create user ogg identified by ogg default tablespace ogg temporary tablespace temp;

SQL> EXEC DBMS_GOLDENGATE_AUTH.GRANT_ADMIN_PRIVILEGE('OGGUSER');

SQL> grant connect, dba to ogg;

Grant succeeded.

3、源端GOLDENGATE要做配置

安装后OGG FOR ORACLE后，默认已经create subdirs了，并且已经配置启动mgr。

运行 oggora$./ggsci进行入ogg命令行

GGSCI (TEST-BDD-057) 1>edit params mgr

PORT 7809

userid ogg,password ogg

purgeoldextracts /data1/oracle/oggora/*,usecheckpoints

accessrule,prog server,allow

重启MGR

GGSCI (TEST-BDD-057) 3> stop mgr

Manager process is required by other GGS processes.

Are you sure you want to stop it (y/n)?y

Sending STOP request to MANAGER ...

Request processed.

Manager stopped.

GGSCI (TEST-BDD-057) 4> start mgr

Manager started.

GGSCI (TEST-BDD-057) 5> info mgr detail

Manager is running (IP port host01.example.com.7809, Process ID 52948).

GGSCI (TEST-BDD-057) 6>

4、需要配置目标端MGR，才能回过来进行源端的EXTRACT的下一步操作。因此，开始安装目标端的OGG FOR BIGDATA。

$mkdir ogg

$tar xvpf ../ggs_Adapters_Linux_x64.tar

$./ggsci

GGSCI (NM-ITC-NF8460M3-303-016) 1> create subdirs

GGSCI (NM-ITC-NF8460M3-303-016) 2> edit params mgr

PORT 17809

DYNAMICPORTLIST 17810-17820

AUTORESTART REPLICAT *, RETRIES 5, WAITMINUTES 3, RESETMINUTES 60

PURGEOLDEXTRACTS ./dirdat/*, usecheckpoints, minkeepdays 1

LAGREPORTHOURS 1

LAGINFOMINUTES 30

GGSCI (NM-ITC-NF8460M3-303-016) 3> start manager

Manager started.

5、回到源端进行初始化EXTRACT，并在目标端生成文件。

$vi dirprm/ini_ext.prm

SOURCEISTABLE

userid ogg,password ogg

RMTHOST 10.142.78.36, MGRPORT 17809

RMTFILE /usr/bdusr01/ogg/dirdat/initld,megabytes 2,purge

table ggtest.*;

-bash-4.1$ ./extract paramfile dirprm/ini_ext.prm reportfile dirrpt/ini_ext.rpt

6、切到目标端，查看是否生成文件

[bdusr01@NM-ITC-NF8460M3-303-016 ogg]$ ls dirdat

initld

7、在源端配置extract。

$./ggsci

Oracle GoldenGate Command Interpreter for Oracle

Version 12.2.0.1.1 OGGCORE_12.2.0.1.0_PLATFORMS_151211.1401_FBO

Linux, x64, 64bit (optimized), Oracle 12c on Dec 12 2015 02:56:48

Operating system character set identified as UTF-8.

GGSCI (TEST-BDD-057) 1> edit params ggext

extract ggext

userid ogg,password ogg

--RMTHOSTOPTIONS

RMTHOST sandbox, MGRPORT 7839

RMTFILE /usr/bdusr01/ogg/dirdat/or, MEGABYTES 2, PURGE

DDL include objname ogg.*

TABLE ogg.*;

GGSCI (TEST-BDD-057) 2> dblogin userid ogg,password ogg

Successfully logged into database.

GGSCI (TEST-BDD-057) 3> register extract GGEXT database

2016-10-20 15:37:21 INFO OGG-02003 Extract GGEXT successfully registered with database at SCN 17151616.

GGSCI (TEST-BDD-057) 4> add extract ggext, INTEGRATED TRANLOG, BEGIN NOW

EXTRACT (Integrated) added.

8、在目标端，编辑hdfs.props。

[bdusr01@NM-ITC-NF8460M3-303-016 ogg]$ vi dirprm/hdfs.props

gg.handlerlist=hdfs

gg.handler.hdfs.type=hdfs

gg.handler.hdfs.includeTokens=true

gg.handler.hdfs.maxFileSize=256m

gg.handler.hdfs.rootFilePath=/data/hjpt/test/jl

gg.handler.hdfs.fileRollInterval=0

gg.handler.hdfs.inactivityRollInterval=0

gg.handler.hdfs.fileSuffix=.txt

gg.handler.hdfs.partitionByTable=true

gg.handler.hdfs.rollOnMetadataChange=true

#gg.handler.hdfs.authType=none

gg.handler.hdfs.authType=kerberos

gg.handler.hdfs.kerberosKeytabFile=/usr/bdusr01/bdusr01.keytab

gg.handler.hdfs.kerberosPrincipal=krbtgt/HADOOP.CHINATELECOM.CN@HADOOP.CHINATELECOM.CN

gg.handler.hdfs.format=delimitedtext

gg.handler.hdfs.format.includeColumnNames=false

gg.handler.hdfs.mode=tx

gg.handler.hdfs.format.includeOpTimestamp=true

gg.handler.hdfs.format.includeCurrentTimestamp=true

gg.handler.hdfs.format.insertOpKey=I

gg.handler.hdfs.format.updateOpKey=U

gg.handler.hdfs.format.deleteOpKey=D

gg.handler.hdfs.format.truncateOpKey=T

gg.handler.hdfs.format.encoding=UTF-8

gg.handler.hdfs.format.fieldDelimiter=CDATA[\u0001]

#gg.handler.hdfs.format.fieldDelimiter='|'

gg.handler.hdfs.format.lineDelimiter=CDATA[\n]

gg.handler.hdfs.format.includeTableName=true

gg.handler.hdfs.format.keyValueDelimiter=CDATA[=]

#gg.handler.hdfs.format.kevValuePairDelimiter=CDATA[,]

gg.handler.hdfs.format.pkUpdateHandling=update

gg.handler.hdfs.format.nullValueRepresentation=

gg.handler.hdfs.format.missingValueRepresentation=CDATA[]

gg.handler.hdfs.format.includePosition=false

goldengate.userexit.timestamp=utc

goldengate.userexit.writers=javawriter

javawriter.stats.display=TRUE

javawriter.stats.full=TRUE

gg.log=log4j

gg.log.level=INFO

gg.report.time=30sec

gg.classpath=/etc/hadoop/conf.empty:/usr/lib/hadoop/*:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop/lib/*

javawriter.bootoptions=-Xmx8200m –Xms8192m -Djava.class.path=ggjava/ggjava.jar

9、在目标端配置JAVA_HOME,LD_LIBRARY_PATH，必需要JAVA 1.7，不然后面运行可能出错。

export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64

export LD_LIBRARY_PATH=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre/lib/amd64:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre/lib/amd64/server:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre/lib/amd64/libjsig.so:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre/lib/amd64/server/libjvm.so

10、在目标端初始化hdfs

[bdusr01@NM-ITC-NF8460M3-303-016 ogg]$./replicat paramfile dirprm/irhdfs.prm reportfile dirrpt/ini_rhdfs.rpt

[bdusr01@NM-ITC-NF8460M3-303-016 ogg]$ hadoop dfs -ls /data/hjpt/test/jl

Found 2 items

...

10、在源端启动extract

GGSCI (TEST-BDD-057) 6>start extract ggext

Sending START request to MANAGER ...

EXTRACT GGEXT starting

11、在目标端配置REPLICAT。

[bdusr01@NM-ITC-NF8460M3-303-016 ogg]$ vi dirprm/rhdfs.prm

REPLICAT rhdfs

setenv HADOOP_COMMON_LIB_NATIVE_DIR=/usr/lib/hadoop/lib/native

DDL include all

TARGETDB LIBFILE libggjava.so SET property=dirprm/hdfs.props

REPORTCOUNT EVERY 1 MINUTES, RATE

GROUPTRANSOPS 10000

MAP ogg.*, TARGET bdtest.*;

[bdusr01@NM-ITC-NF8460M3-303-016 ogg]$./ggsci

GGSCI (NM-ITC-NF8460M3-303-016) 1> add replicat rhdfs, exttrail dirdat/or

REPLICAT added.

GGSCI (NM-ITC-NF8460M3-303-016) 2> start replicat rhdfs

Sending START request to MANAGER ...

REPLICAT RHDFS starting

GGSCI (NM-ITC-NF8460M3-303-016) 3>

12、目标端HIVE建表

hive> CREATE EXTERNAL TABLE BDTEST.TEST_TAB_1 (tran_flag string, tab_name string, tran_time_utc timestamp, tran_time_loc string,something string, something1 string,

> PK_ID INT, RND_STR VARCHAR(10),USE_DATE string,RND_STR_1 string, ACC_DATE string)

> stored as textfile location '/data/hjpt/test/jl';

OK

Time taken: 0.327 seconds

hive> select * from BDTEST.TEST_TAB_1;

...

13、在源端插入数据，再到目标端查看变化，即可。

参考文档：

http://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/goldengate/12c/OGG12c_Integrated_Replicat/index.html

http://www.tuicool.com/articles/IzqAZnI

转载于:https://my.oschina.net/u/2503743/blog/761863