配置mysql
创建用户并授权
创建用户
CREATE USER 'debezium'@'%' IDENTIFIED BY 'password';
授予权限
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'debezium' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
启用二进制日志
1.检查是否支持二进制日志
SELECT variable_value as "BINARY LOGGING STATUS (log-bin) ::" FROM information_schema.global_variables WHERE variable_name='log_bin';
2.如果是OFF
,请使用以下属性配置您的 MySQL 服务器配置文件并重启服务器,如下表所述:
log_bin = mysql-bin
binlog_format = ROW
binlog_row_image = FULL
expire_logs_days = 10
3.检查二进制文件
SELECT variable_value as "BINARY LOGGING STATUS (log-bin) ::" FROM information_schema.global_variables WHERE variable_name='log_bin';
启用gdit
因为现在使用的单节点暂时未开启
安装kafka
启动kafka-connect
修改config/connect-distributed.properties的plugin.path和bootstrap.servers的值
修改日志文件 config/connect-log4j.properties
log4j.rootLogger=INFO, kafkaConnectAppender
log4j.appender.kafkaConnectAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.kafkaConnectAppender.DatePattern='.'yyyy-MM-dd
log4j.appender.kafkaConnectAppender.File=/hadoop/kafka_2.11-0.11.0.2/logs/kafka-connect.log
log4j.appender.kafkaConnectAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaConnectAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.I0Itec.zkclient=ERROR
log4j.logger.org.reflections=ERROR
启动kafka-connnector bin/connect-distributed.sh -daemon config/connect-distributed.properties
api查看kafka信息http://ip:8083
启动mysql-connector
1.配置mysql-connector文件
{
"name": "inventory-connector",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"database.hostname": "mysqlip",
"database.port": "3306",
"database.user": "debezium",
"database.password": "password",
"database.server.id": "184054",
"database.server.name": "fullfillment",
"database.include.list": "数据库1,数据库2",
"database.history.kafka.bootstrap.servers": "kafka:9092",
"database.history.kafka.topic": "dbhistory.fullfillment",
"include.schema.changes": "true"
}
}
其他详细配置参数官方文档https://debezium.io/documentation/reference/1.6/connectors/mysql.html#mysql-connector-properties
2.通过api启动mysql-connect
post方式提交http://ip:8083/connectors,参数为json文件
3.查看connect信息:get http://ip:8083/connectors 查看所有的connnectors
get http://ip:8083/connectors/inventory-connector/status 获取当前的状态
详细api请看:https://kafka.apache.org/documentation.html#connect_rest
查看kafka
1.dbhistory.fullfillment为表结构修改数据
2..dbhistory.fullfillment.[table] 为数据修改数据
可能遇到的问题:
ClassNotFoundException: org.apache.kafka.connect.header.ConnectHeaders
kafka版本过低,使用1.1及以上的版本即可