flume 从MySQL导入到kafka

从MySQL导入到kafka

环境

• centos 6
• JDK 1.8
• flume 1.9
• flume-ng-sql-source 1.5
• zookeeper 3.5
• kafka 2.3

启动zk,kafka

//# 启动zookeeper
$ zkServer.sh start

//# 启动kafka
$ kafka-server-start.sh -daemon server.properties

Flume插件flume-ng-sql-source安装, MYSQL添加数据库和数据,参见:flume 从mysql读数据,写入hdfs

Flume配置

$ cd /var/tmp
$ vi example.conf
# Define
a1.sources = mysqlSource
a1.sinks = kafkaSink
a1.channels = memChannel

# Describe/configure the source
a1.sources.mysqlSource.type = org.keedio.flume.source.SQLSource
a1.sources.mysqlSource.hibernate.connection.url = jdbc:mysql://vps126:63751/chenzl
a1.sources.mysqlSource.hibernate.connection.user = chenzl
a1.sources.mysqlSource.hibernate.connection.password = chenzl
a1.sources.mysqlSource.table = users
a1.sources.mysqlSource.columns.to.select = *
a1.sources.mysqlSource.run.query.delay=5000
a1.sources.mysqlSource.status.file.path = /tmp/flume
a1.sources.mysqlSource.status.file.name = mysqlSource.status

# Describe the sink
a1.sinks.kafkaSink.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.kafkaSink.kafka.topic = mysqlUsers
a1.sinks.kafkaSink.kafka.bootstrap.servers = localhost:9092
a1.sinks.kafkaSink.kafka.producer.acks = 1
a1.sinks.kafkaSink.flumeBatchSize = 20


# Use a channel which buffers events in memory
a1.channels.memChannel.type = memory
a1.channels.memChannel.capacity = 1000
a1.channels.memChannel.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.mysqlSource.channels = memChannel
a1.sinks.kafkaSink.channel = memChannel

运行命令

$ cd /var/tmp
$ flume-ng agent --conf /usr/local/flume/conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console

查看kakfa

//# 查看topic
$ kafka-topics.sh --list --bootstrap-server localhost:9092
mysqlUsers

//# 查看主题
$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mysqlUsers --from-beginning
"1","alice","alice@abc.com","engineering","2019-09-19 14:04:29.0"
"2","bob","bob@abc.com","sales","2019-09-19 14:04:30.0"

查看状态文件

$ cat /tmp/flume/mysqlSource.status
{"SourceName":"mysqlSource","URL":"jdbc:mysql:\/\/vps126:63751\/chenzl","LastIndex":"2","ColumnsToSelect":"*","Table":"users"}

MySQL插入数据

> INSERT INTO users (name, email, department) VALUES ('chenzl', 'chenzl@abc.com', 'technology');

查看kakfa

//# 查看主题
$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mysqlUsers --from-beginning
"1","alice","alice@abc.com","engineering","2019-09-19 14:04:29.0"
"2","bob","bob@abc.com","sales","2019-09-19 14:04:30.0"
"3","chenzl","chenzl@abc.com","technology","2019-09-19 15:05:48.0"

MySQL删除数据

> DELETE FROM users WHERE id=3;

flume不会同步MySQL更新,删除的变化;