一、资源信息
这边就不赘述
二、配置相关数据源、目标数据源和java环境
mysql数据源
1、数据库,create database testdb1;
2、用户权限,需要拥有select权限和binlog拉取权限,此处使用root权限
3、同步的表(切换到syncdb1数据库),create table tb1(a int, b char(10), primary key(a));
pgsql目的数据库
1、用户,create user testdb with password 'testdb';
2、数据库,create database testdb with owner 'testdb';
3、同步的表(使用testdb用户切换到testdb数据库),create table tb1(a int, b char(10), primary key(a));
java环境的安装
1、下载二进制安装包:jdk-8u101-linux-x64.tar.gz
2、解压二进制包并做软链接:tar xf jdk-8u101-linux-x64.tar.gz && ln -s /data/jdk1.8.0_101 /usr/java
3、配置路径和java环境变量:vim /etc/profile.d/java.sh
export JAVA_HOME=/usr/java
export JRE_HOME=$JAVA_HOME/jre
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
4、source生效:source /etc/profile.d/java.sh
5、安装jsvc,yum install jsvc
三、kafka的安装和启动配置
1、下载地址:https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/
2、kafka官方文档:http://kafka.apache.org/
3、解压缩:tar xf kafka_2.11-2.0.0.tgz && cd kafka_2.11-2.0.0
4、ZooKeeper
启动,bin/zookeeper-server-start.sh config/zookeeper.properties
关闭,bin/zookeeper-server-stop.sh config/zookeeper.properties
5、Kafka server
启动,bin/kafka-server-start.sh config/server.properties
启动,bin/kafka-server-stop.sh config/server.properties
6、Topic
创建,bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic green
查询,bin/kafka-topics.sh --list --zookeeper localhost:2181
删除,bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic green
7、Producer(不是本实验必须的,作为学习使用)
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic green
>aaa
>123
>
8、Consumer(不是本实验必须的,作为学习使用)
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic green --from-beginning
aaa
123
四、maxwell的安装和启动配置
1、下载地址:https://github.com/zendesk/maxwell/releases
2、maxwell官方文档:https://github.com/zendesk/maxwell
3、解压缩:tar xf maxwell-1.17.1.tar.gz && cd maxwell-1.17.1
4、修改配置文件,cp config.properties.example config.properties && vim config.properties
log_level=info
# kafka info
producer=kafka
kafka.bootstrap.servers=localhost:9092
kafka_topic=green
ddl_kafka_topic=green
# mysql login info
host=xx.xx.xx.xx
port=3306
user=root
password=123456
5、启动maxwell,bin/maxwell --config config.properties
6、maxwell默认在源数据库生成库maxwell记录相关信息
五、bireme的安装和启动配置
1、下载地址:https://github.com/HashDataInc/bireme/releases
2、bireme官方文档:https://github.com/HashDataInc/bireme/blob/master/README_zh-cn.md
3、解压缩:tar xf bireme-1.0.0.tar.gz && cd bireme-1.0.0
4、修改配置文件,vim etc/config.properties
# target database where the data will sync into.
target.url = jdbc:postgresql://xxx.xxx.xxx.xxx:5432/testdb
target.user = testdb
target.passwd = testdb
# data source name list, separated by comma.
data_source = maxwell1
# data source "mysql1" type
maxwell1.type = maxwell
# kafka server which maxwell write binlog into.
maxwell1.kafka.server = 127.0.0.1:9092
# kafka topic which maxwell write binlog into.
maxwell1.kafka.topic = green
# kafka groupid used for consumer.
maxwell1.kafka.groupid = bireme
# set the IP address for bireme state server.
state.server.addr = 0.0.0.0
# set the port for bireme state server.
state.server.port = 8080
5、修改配置文件,vim etc/maxwell1.properties(表映射配置)
note:maxwell1.properties的maxwell1一定要和bireme的data_source保持一致
testdb1.tb1 = public.tb1
testdb2.tb1 = public.tb1
6、启动bireme,bin/bireme start
六、测试
1、mysql数据源
insert into tb1 select 1,'a';
insert into tb1 select 2,'b';
2、pgsql目标数据库
testdb=# select * from tb1;
a | b
---+------------
1 | a
2 | b
(2 rows)