ogg mysql to kafka_OGG 从 MYSQL5.7同步到BigData(Kafka)

1 说明

1.1 软件版本

Mysql:5.7.26 下载地址

Mysql Ogg:12.2.0.2 下载地址

BigData Ogg:12.3.2.1.1 下载地址

2 安装

2.1 Mysql

(1) 解压并重命名

tar -zxvf mysql-5.7.26-1.el6.x86_64.rpm-bundle.tar.gz

mv mysql-5.7.26-linux-glibc2.12-x86_64 /usr/local/mysql5.7

(2) Mysql配置文件(/etc/my.cnf)

[mysqld]

datadir=/usr/local/mysql5.7/data

socket=/var/lib/mysql/mysql.sock

user=mysql

symbolic-links=0

#######数据库超时时间

connect_timeout=388000

#######wait_timeout=388000

interactive_timeout=388000

##########默认存储引擎

default-storage-engine=INNODB

innodb_buffer_pool_size=1073741824

########ogg配置

log-bin=mysql-bin #添加这一行就ok

binlog-format=ROW #选择row模式

server_id=1 #配置mysql replaction需要定义，不能和canal的slaveId重复

[mysqld_safe]

log-error=/var/log/mysqld5.7.log

pid-file=/var/run/mysqld/mysqld5.7.pid

[client]

default-character-set=utf8

(3) Mysql.server(/usr/local/mysql5.7/support-files/mysql.server

basedir=/usr/local/mysql5.7

datadir=/usr/local/mysql5.7/data

(4) 初始化

cd /usr/local/mysql5.7/bin

./mysqld --initialize -> 这一步会输出root初始密码，请复制下来

(5) 启动文件配置

cp /usr/local/mysql5.7/support-files/mysql.server /etc/init.d/mysql5.7

(6) 启动数据库

services mysqld5.7 start

(7) 可能出现sock异常

解决

ln -s /var/lib/mysql/mysql.sock /tmp/mysql.sock

(8) Mysql的path

先将旧版本的备份

cd /usr/bin

mkdir mysql_bak

mv mysql* mysql_bak;

建立5.7版本的PATH

vim /etc/profile

export MYSQL_HOME=/usr/local/mysql5.7

export PATH=

MYSQL_HOME/bin

(9) Mysql用root登录

mysql -uroot

输入之前复制的密码

alter user root identified by ‘root’; -> 修改root密码

(10) 创建测试库

Mysql -uroot -proot <

Create user gg1 identified by gg1;

Create database gg1db default character set =utf8;

Grant all privileges on gg1db.* to gg1@localhost identified by ‘gg1’;

Flush privileges;

!

2.2 源端

(1) 建立ggs用户,并拷贝ggs包到ggs用户下

user add ggs

Passwd ggs -> ggs

chown -R ggs:ggs /home/ggs

(2) 解压到ogg目录下

su - ggs

mkdir ogg

uuzip 122022_ggs_Linux_x64_MySQL_64bit.zip

tar -xvf *tar -C ogg

(3) env.sh

su - ggs

touch env.sh

export OGG_HOME=$HOME/ogg

export PATH=

OGG_HOME

export LD_LIBRARY_PATH=

OGG_HOME

(4) 初始化ogg

su - ggs

source env.sh

cd ogg

./ggsci

===>执行：create subdirs

(5) 配置进程

su - ggs

source env.sh

cd ogg

./ggsci

===>执行：dblogin sourcedb gg1db@localhost:3306,userid gg1,password gg1;

===>执行：edit param mgr

===>输入:

port 7015

ACCESSRULE, PROG *, IPADDR *, ALLOW

===>保存退出

===>执行:edit param ext1

===>输入:

EXTRACT ext1

setenv (MYSQL_HOME="/usr/local/mysql5.7")

dboptions host 192.168.102.3, connectionport 3306

tranlogoptions altlogdest /usr/local/mysql5.7/data/mysql-bin.index

SOURCEDB gg1db@localhost:3306,userid gg1,PASSWORD gg1

EXTTRAIL ./dirdat/et

table gg1db.*;

===>保存退出

===>执行:edit parm pump1

===>输入:

EXTRACT pump1

SOURCEDB gg1db@localhost:3306,userid gg1,PASSWORD gg1

RMTHOST 192.168.102.3, MGRPORT 8015

RMTTRAIL ./dirdat/rt

table gg1db.*;

===>保存退出;

===>执行:add extract ext1, tranlog,begin now

===>执行:add exttrail ./dirdat/et, extract ext1

===>执行:add extract pump1, exttrailsource ./dirdat/et

===>执行:add rmttrail ./dirdat/rt,extract pump1

(6) 配置def

su - ggs

source env.sh

cd ogg

./ggsci

===>执行:edit param defgen

===>输入:

defsfile ./dirdef/defgen.def

sourcedb gg1db@localhost:3306,userid gg1,password gg1

table gg1db.*;

===>保存退出

生成defgen.def

退出ggsci；

执行：./defgen paramfile ./dirprm/defgen.prm

===》在dirdef目录下会生成一个defgen.def文件，需要将文件拷贝到目标端(ogg)下对应的dirdef目录下

2.3 目标端

(1) 建立ogg用户,并拷贝ogg包到ogg用户下

user add ogg

Passwd ogg -> ogg

chown -R ogg:ogg /home/ogg

(2) 解压到ogg目录下

su - ogg

mkdir ogg

uuzip OGG_BigData_Linux_x64_12.3.2.1.1.zip

tar -xvf *tar -C ogg

(3) env.sh

su - ogg

touch env.sh

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/java/jdk1.8.0_131/jre/lib/amd64/server/

export LD_LIBRARY_PATH=

HOME/ogg

export JAVA_LIBDIR=/usr/share/java

(4) 初始化ogg

su - ogg

source env.sh

cd ogg

./ggsci

===>执行：create subdirs

(5) 配置进程

su - ogg

source env.sh

cd ogg

./ggsci

===>执行：edit param mgr

===>输入:

PORT 8015

ACCESSRULE, PROG *, IPADDR *, ALLOW

===>保存退出

===>执行:edit param rep1

===>输入:

replicat rep1

sourcedefs ./dirdef/defgen.def

TARGETDB LIBFILE libggjava.so SET property=./dirprm/kafka.props

MAP gg1db., TARGET gg1db.;

===>保存退出

===>执行:add replicat rep1, exttrail ./dirdat/rt (注意，这里的exttrail必须和源端的dump配置一致)

(6) kafka.props配置(dirprm目录下)

gg.handlerlist = kafkahandler

gg.handler.kafkahandler.type=kafka

gg.handler.kafkahandler.KafkaProducerConfigFile=custom_kafka_producer.properties

#######The following resolves the topic name using the short table name

gg.handler.kafkahandler.topicMappingTemplate=tggtest #这个是定义的主题

############The following selects the message key using the concatenated primary keys

############gg.handler.kafkahandler.keyMappingTemplate=${primaryKeys}

###########gg.handler.kafkahandler.format=avro_op

gg.handler.kafkahandler.SchemaTopicName=tggtest ########定义的主题

gg.handler.kafkahandler.BlockingSend =false

gg.handler.kafkahandler.includeTokens=false

gg.handler.kafkahandler.mode=op

gg.handler.kafkahandler.format=json

#########gg.handler.kafkahandler.format.insertOpKey=I

#######gg.handler.kafkahandler.format.updateOpKey=U

#########gg.handler.kafkahandler.format.deleteOpKey=D

#######gg.handler.kafkahandler.format.truncateOpKey=T

goldengate.userexit.writers=javawriter

javawriter.stats.display=TRUE

javawriter.stats.full=TRUE

gg.log=log4j

gg.log.level=INFO

gg.report.time=30sec

##########Sample gg.classpath for Apache Kafka

gg.classpath=dirprm/:/usr/local/kafka/libs/*

##########Sample gg.classpath for HDP

#########gg.classpath=/etc/kafka/conf:/usr/hdp/current/kafka-broker/libs/*

javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar

(7) custom_kafka_producer.properties配置(dirprm目录下)

bootstrap.servers=localhost:9092

acks=1

reconnect.backoff.ms=1000

value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer

key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer

######## 100KB per partition

batch.size=16384

linger.ms=0

key.converter=org.apache.kafka.connect.json.JsonConverter

value.converter=org.apache.kafka.connect.json.JsonConverter

key.converter.schemas.enable=false

value.converter.schemas.enable=false

3 启动

3.1 非ogg程序

Mysql启动； 3306端口

Kafka启动； 9092端口

Zookeeper启动；2181端口

3.2 源端

Su - ggs

Source env.sh

Cd ogg

./ggsci

===>执行: start mgr

===>执行:start ext1

===>执行:start pump1

日志 ggserr.log

3.3 目标端

Su - ogg

Source env.sh

Cd ogg

./ggsci

===>执行: start mgr

===>执行:start rep1

TBD：如果源端频繁请求一个未知的端口，可能是dirdat的目录配置错误，亦或者可以在目标端下执行如下操作：

cd ogg

./servier -p 7819 (7819是源端ggserr中的端口)

日志 ggserr.log

3.4 检查

(1) Kafka 主题是否自动创建

Cd /usr/local/kakfa

./bin/kafka-topics.sh --list --zookeeper localhost:2181

如果出现tggtest->说明创建成果

(2) 在mysql源端插入或者更新数据，观察相关日志

源端：/home/ggs/ogg/dirdat/et00000*

目标端:/home/ogg/ogg/dirdat/rt0000*

(3) Kafka程序

-- coding:utf-8 --

import kafka

from kafka import KafkaConsumer

consumer = KafkaConsumer('tggtest', bootstrap_servers=['localhost:9092'])

for message in consumer:

print "%s:%d:%d: key=%s value=%s" %(message.topic, message.partition,message.offset, message.key,message.value)

接收到插入的消息

tggtest:0:65: key=gg1db.t1 value={"table":"gg1db.t1","op_type":"I","op_ts":"2019-05-20 22:26:24.000108","current_ts":"2019-05-20T22:26:30.252000","pos":"00000000000000008226","after":{"id":2,"name":"g"}}