1 说明
1.1 软件版本
Mysql:5.7.26 下载地址
Mysql Ogg:12.2.0.2 下载地址
BigData Ogg:12.3.2.1.1 下载地址
2 安装
2.1 Mysql
(1) 解压并重命名
tar -zxvf mysql-5.7.26-1.el6.x86_64.rpm-bundle.tar.gz
mv mysql-5.7.26-linux-glibc2.12-x86_64 /usr/local/mysql5.7
(2) Mysql配置文件(/etc/my.cnf)
[mysqld]
datadir=/usr/local/mysql5.7/data
socket=/var/lib/mysql/mysql.sock
user=mysql
symbolic-links=0
#######数据库超时时间
connect_timeout=388000
#######wait_timeout=388000
interactive_timeout=388000
##########默认存储引擎
default-storage-engine=INNODB
innodb_buffer_pool_size=1073741824
########ogg配置
log-bin=mysql-bin #添加这一行就ok
binlog-format=ROW #选择row模式
server_id=1 #配置mysql replaction需要定义,不能和canal的slaveId重复
[mysqld_safe]
log-error=/var/log/mysqld5.7.log
pid-file=/var/run/mysqld/mysqld5.7.pid
[client]
default-character-set=utf8
(3) Mysql.server(/usr/local/mysql5.7/support-files/mysql.server
basedir=/usr/local/mysql5.7
datadir=/usr/local/mysql5.7/data
(4) 初始化
cd /usr/local/mysql5.7/bin
./mysqld --initialize -> 这一步会输出root初始密码,请复制下来
(5) 启动文件配置
cp /usr/local/mysql5.7/support-files/mysql.server /etc/init.d/mysql5.7
(6) 启动数据库
services mysqld5.7 start
(7) 可能出现sock异常
解决
ln -s /var/lib/mysql/mysql.sock /tmp/mysql.sock
(8) Mysql的path
先将旧版本的备份
cd /usr/bin
mkdir mysql_bak
mv mysql* mysql_bak;
建立5.7版本的PATH
vim /etc/profile
export MYSQL_HOME=/usr/local/mysql5.7
export PATH=
MYSQL_HOME/bin
(9) Mysql用root登录
mysql -uroot
输入之前复制的密码
alter user root identified by ‘root’; -> 修改root密码
(10) 创建测试库
Mysql -uroot -proot <
Create user gg1 identified by gg1;
Create database gg1db default character set =utf8;
Grant all privileges on gg1db.* to gg1@localhost identified by ‘gg1’;
Flush privileges;
!
2.2 源端
(1) 建立ggs用户,并拷贝ggs包到ggs用户下
user add ggs
Passwd ggs -> ggs
chown -R ggs:ggs /home/ggs
(2) 解压到ogg目录下
su - ggs
mkdir ogg
uuzip 122022_ggs_Linux_x64_MySQL_64bit.zip
tar -xvf *tar -C ogg
(3) env.sh
su - ggs
touch env.sh
export OGG_HOME=$HOME/ogg
export PATH=
OGG_HOME
export LD_LIBRARY_PATH=
OGG_HOME
(4) 初始化ogg
su - ggs
source env.sh
cd ogg
./ggsci
===>执行:create subdirs
(5) 配置进程
su - ggs
source env.sh
cd ogg
./ggsci
===>执行:dblogin sourcedb gg1db@localhost:3306,userid gg1,password gg1;
===>执行:edit param mgr
===>输入:
port 7015
ACCESSRULE, PROG *, IPADDR *, ALLOW
===>保存退出
===>执行:edit param ext1
===>输入:
EXTRACT ext1
setenv (MYSQL_HOME="/usr/local/mysql5.7")
dboptions host 192.168.102.3, connectionport 3306
tranlogoptions altlogdest /usr/local/mysql5.7/data/mysql-bin.index
SOURCEDB gg1db@localhost:3306,userid gg1,PASSWORD gg1
EXTTRAIL ./dirdat/et
table gg1db.*;
===>保存退出
===>执行:edit parm pump1
===>输入:
EXTRACT pump1
SOURCEDB gg1db@localhost:3306,userid gg1,PASSWORD gg1
RMTHOST 192.168.102.3, MGRPORT 8015
RMTTRAIL ./dirdat/rt
table gg1db.*;
===>保存退出;
===>执行:add extract ext1, tranlog,begin now
===>执行:add exttrail ./dirdat/et, extract ext1
===>执行:add extract pump1, exttrailsource ./dirdat/et
===>执行:add rmttrail ./dirdat/rt,extract pump1
(6) 配置def
su - ggs
source env.sh
cd ogg
./ggsci
===>执行:edit param defgen
===>输入:
defsfile ./dirdef/defgen.def
sourcedb gg1db@localhost:3306,userid gg1,password gg1
table gg1db.*;
===>保存退出
生成defgen.def
退出ggsci;
执行:./defgen paramfile ./dirprm/defgen.prm
===》在dirdef目录下会生成一个defgen.def文件,需要将文件拷贝到目标端(ogg)下对应的dirdef目录下
2.3 目标端
(1) 建立ogg用户,并拷贝ogg包到ogg用户下
user add ogg
Passwd ogg -> ogg
chown -R ogg:ogg /home/ogg
(2) 解压到ogg目录下
su - ogg
mkdir ogg
uuzip OGG_BigData_Linux_x64_12.3.2.1.1.zip
tar -xvf *tar -C ogg
(3) env.sh
su - ogg
touch env.sh
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/java/jdk1.8.0_131/jre/lib/amd64/server/
export LD_LIBRARY_PATH=
HOME/ogg
export JAVA_LIBDIR=/usr/share/java
(4) 初始化ogg
su - ogg
source env.sh
cd ogg
./ggsci
===>执行:create subdirs
(5) 配置进程
su - ogg
source env.sh
cd ogg
./ggsci
===>执行:edit param mgr
===>输入:
PORT 8015
ACCESSRULE, PROG *, IPADDR *, ALLOW
===>保存退出
===>执行:edit param rep1
===>输入:
replicat rep1
sourcedefs ./dirdef/defgen.def
TARGETDB LIBFILE libggjava.so SET property=./dirprm/kafka.props
MAP gg1db., TARGET gg1db.;
===>保存退出
===>执行:add replicat rep1, exttrail ./dirdat/rt (注意,这里的exttrail必须和源端的dump配置一致)
(6) kafka.props配置(dirprm目录下)
gg.handlerlist = kafkahandler
gg.handler.kafkahandler.type=kafka
gg.handler.kafkahandler.KafkaProducerConfigFile=custom_kafka_producer.properties
#######The following resolves the topic name using the short table name
gg.handler.kafkahandler.topicMappingTemplate=tggtest #这个是定义的主题
############The following selects the message key using the concatenated primary keys
############gg.handler.kafkahandler.keyMappingTemplate=${primaryKeys}
###########gg.handler.kafkahandler.format=avro_op
gg.handler.kafkahandler.SchemaTopicName=tggtest ########定义的主题
gg.handler.kafkahandler.BlockingSend =false
gg.handler.kafkahandler.includeTokens=false
gg.handler.kafkahandler.mode=op
gg.handler.kafkahandler.format=json
#########gg.handler.kafkahandler.format.insertOpKey=I
#######gg.handler.kafkahandler.format.updateOpKey=U
#########gg.handler.kafkahandler.format.deleteOpKey=D
#######gg.handler.kafkahandler.format.truncateOpKey=T
goldengate.userexit.writers=javawriter
javawriter.stats.display=TRUE
javawriter.stats.full=TRUE
gg.log=log4j
gg.log.level=INFO
gg.report.time=30sec
##########Sample gg.classpath for Apache Kafka
gg.classpath=dirprm/:/usr/local/kafka/libs/*
##########Sample gg.classpath for HDP
#########gg.classpath=/etc/kafka/conf:/usr/hdp/current/kafka-broker/libs/*
javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar
(7) custom_kafka_producer.properties配置(dirprm目录下)
bootstrap.servers=localhost:9092
acks=1
reconnect.backoff.ms=1000
value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
######## 100KB per partition
batch.size=16384
linger.ms=0
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
3 启动
3.1 非ogg程序
Mysql启动; 3306端口
Kafka启动; 9092端口
Zookeeper启动;2181端口
3.2 源端
Su - ggs
Source env.sh
Cd ogg
./ggsci
===>执行: start mgr
===>执行:start ext1
===>执行:start pump1
日志 ggserr.log
3.3 目标端
Su - ogg
Source env.sh
Cd ogg
./ggsci
===>执行: start mgr
===>执行:start rep1
TBD:如果源端频繁请求一个未知的端口,可能是dirdat的目录配置错误,亦或者可以在目标端下执行如下操作:
cd ogg
./servier -p 7819 (7819是源端ggserr中的端口)
日志 ggserr.log
3.4 检查
(1) Kafka 主题是否自动创建
Cd /usr/local/kakfa
./bin/kafka-topics.sh --list --zookeeper localhost:2181
如果出现tggtest->说明创建成果
(2) 在mysql源端插入或者更新数据,观察相关日志
源端:/home/ggs/ogg/dirdat/et00000*
目标端:/home/ogg/ogg/dirdat/rt0000*
(3) Kafka程序
-- coding:utf-8 --
import kafka
from kafka import KafkaConsumer
consumer = KafkaConsumer('tggtest', bootstrap_servers=['localhost:9092'])
for message in consumer:
print "%s:%d:%d: key=%s value=%s" %(message.topic, message.partition,message.offset, message.key,message.value)
接收到插入的消息
tggtest:0:65: key=gg1db.t1 value={"table":"gg1db.t1","op_type":"I","op_ts":"2019-05-20 22:26:24.000108","current_ts":"2019-05-20T22:26:30.252000","pos":"00000000000000008226","after":{"id":2,"name":"g"}}