利用Kafka-connect分布式模式实现mysql数据库存储到topic，并从topic取数据到mysql数据库

一、搭建kafka

二、下载connect jar和mysql-connect包到kafka目录下的lib文件夹并配置connect-distributed.properties

记得先关防火墙

#建表
CREATE TABLE `users_input` (
  `uid` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(20) NOT NULL,
  `age` int(11) NOT NULL,
  PRIMARY KEY (`uid`)
) ENGINE=InnoDB AUTO_INCREMENT=30 DEFAULT CHARSET=utf8;
CREATE TABLE `users_output` (
  `uid` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(20) NOT NULL,
  `age` int(11) NOT NULL,
  PRIMARY KEY (`uid`)
) ENGINE=InnoDB AUTO_INCREMENT=29 DEFAULT CHARSET=utf8;

cd /usr/local/kafka/config
vi connect-distributed.properties

修改如下

# Broker Server的访问ip和端口号
bootstrap.servers={ip}:9092
# 指定集群id
group.id=connect-cluster
# 指定rest服务的端口号
rest.port=8083

三、启动connect

connect-distributed.sh -daemon /usr/local/kafka/config/connect-distributed.properties

启动成功后访问

http://{ip}:8083/connector-plugins
可以看到

四、使用POST方式请求/connectors接口来新增一个source的connector

curl -X POST -H 'Content-Type: application/json' -i 'http://{ip}:8083/connectors' \
--data \
'{"name":"test-upload-mysql","config":{
"connector.class":"io.confluent.connect.jdbc.JdbcSourceConnector",
"connection.url":"jdbc:mysql://{ip}:3306/test?user=root&password=root&serverTimezone=GMT%2B8",
"table.whitelist":"users_input",
"incrementing.column.name": "uid",
"mode":"incrementing",
"topic.prefix": "test-mysql-"}}'

新增成功后访问

http://{ip}:8083/connectors
会出现

五、使用POST方式请求/connectors接口来新增一个sink的connector

curl -X POST -H 'Content-Type: application/json' -i 'http://{ip}:8083/connectors' \
--data \
'{"name":"test-download-mysql","config":{
"connector.class":"io.confluent.connect.jdbc.JdbcSinkConnector",
"connection.url":"jdbc:mysql://{ip}:3306/test?user=root&password=root&serverTimezone=GMT%2B8",
"topics":"test-mysql-users_input",
"auto.create":"false",
"insert.mode": "upsert",
"pk.mode":"record_value",
"pk.fields":"uid",
"table.name.format": "users_output"}}'

六、测试

connect-distributed.sh -daemon /usr/local/kafka/config/connect-distributed.properties

 kafka-console-consumer.sh --bootstrap-server {ip}:9092 --topic test-mysql-users_input --from-beginning

往users_input表添加数据后users_output表会同步增加