canal & maxwell

canal & maxwell

1、maxwell

1.1 在Linux上下载文件包

在Linux上下载：
curl -sLo - https://github.com/zendesk/maxwell/releases/download/v1.37.0/maxwell-1.25.0.tar.gz \
       | tar zxvf -
       
在Linux上下载：
curl -sLo - https://github.com/zendesk/maxwell/releases/download/v1.25.0/maxwell-1.25.0.tar.gz \
| tar zxvf -
cd maxwell-1.25.0

1.2 mysql 配置

1、在开启binlog要同步数据的mysql上创建maxwell数据库，用于存放maxwell元数据
CREATE DATABASE maxwell ;
2、maxwell在mysql的账号密码，并赋予权限。
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'maxwell'@'%' IDENTIFIED BY '123456kssl';

1.3 maxwell配置文件

log_level=info

producer=kafka
kafka.bootstrap.servers= 119.103.106.63:9092,114.119.127.82:9092,119.18.164.41:9092
#需要添加
kafka_topic=OPGDIdhisOrders
# mysql login info
host=192.168.122.78
user=maxwell
password=123456kssl
#需要添加 后续初始化会用
client_id=maxwell_1
producer_partition_by=table
#认证
kafka.security.protocol= SASL_PLAINTEXT
kafka.sasl.mechanism= GSSAPI
kafka.sasl.kerberos.service.name= kafka
kafka.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule  required serviceName=kafka useKeyTab=true storeKey=true keyTab="/opt/module/maxwell-1.25.0/kafka.keytab" principal="kafka@HADOOP.COM";

#binlog数据过滤
exclude_dbs=*
include_dbs=*
include_tables=customer,customer_order,basic_information,extra_info,customer_bank,bank_information,contact_information,customer_bill_pay_log,order_zhanqi

1.4 启动命令

/opt/module/maxwell-1.25.0/bin/maxwell --config  /opt/module/maxwell-1.25.0/config.properties >/dev/null 2>&1 &
备注：会根据配置文件中的过滤条件过滤数据

nohup /opt/module/maxwell-1.25.0/bin/maxwell --config  /opt/module/maxwell-1.25.0/config.properties  --filter="exclude:*.*, include:*.customer,include:*.customer_order,include:*.basic_information, include:*.extra_info, include:*.customer_bank, include:*.bank_information, include:*.contact_information,include:*.customer_bill_pay_log, include:*.order_zhanqi" 

1.5 bootstrap功能开启

bin/maxwell-bootstrap --user maxwell  --password 123456kssl --host Linux1  --database bantuandana --table customer --client_id maxwell_1


--client_id
maxwell-bootstrap不具备将数据直接导入kafka或者hbase的能力，通过--client_id指定将数据交给哪个maxwell进程处理，在