数据采集flume-ng-sql-source 到kafka（window）

最新推荐文章于 2024-06-08 01:09:18 发布

xiaoping117

最新推荐文章于 2024-06-08 01:09:18 发布

阅读量177

点赞数

文章标签： kafka flume window mysql

本文链接：https://blog.csdn.net/xiaoping117/article/details/109719129

版权

本文详细介绍了如何配置Apache Flume 1.9.0，包括设置JAVA_HOME和FLUME_HOME，创建SQLSource从MySQL读取数据并发送到Kafka。重点在于配置SQL查询、通道和Kafka sink的参数。

摘要由CSDN通过智能技术生成

下载 apache-flume-1.9.0-bin

配置：JAVA_HOME，FLUME_HOME

新增配置文件

F:\apache-flume-1.9.0-bin\test\mysql2kafka.conf

a1.channels = ch-1
a1.sources = src-1
a1.sinks = k1
###########sql source#################
# For each one of the sources, the type is defined
a1.sources.src-1.type = org.keedio.flume.source.SQLSource
a1.sources.src-1.hibernate.connection.url = jdbc:mysql://xxxxx:19860/sms
# Hibernate Database connection properties
a1.sources.src-1.hibernate.connection.user = root
a1.sources.src-1.hibernate.connection.password = *
a1.sources.src-1.hibernate.connection.autocommit = true
a1.sources.src-1.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect
a1.sources.src-1.hibernate.connection.driver_class = com.mysql.jdbc.Driver
a1.sources.src-1.run.query.delay=5000
a1.sources.src-1.status.file.path = ./
a1.sources.src-1.status.file.name = sqlSource.status
a1.sources.src-1.delimiter.entry = |
# Custom query
a1.sources.src-1.start.from = 0
a1.sources.src-1.custom.query = SELECT putintime,"园区A" as yuanqu,"设备B" as shebei,msgcont FROM 201801cont where putintime> $@$ ORDER BY putintime DESC
a1.sources.src-1.batch.size = 100
a1.sources.src-1.max.rows = 200
a1.sources.src-1.hibernate.connection.provider_class = org.hibernate.connection.C3P0ConnectionProvider
a1.sources.src-1.hibernate.c3p0.min_size=1
a1.sources.src-1.hibernate.c3p0.max_size=10
a1.sources.src-1.delimiter.entry = ;
a1.sources.src-1.enclose.by.quotes = false

################################################################
a1.channels.ch-1.type = memory
a1.channels.ch-1.capacity = 10000
a1.channels.ch-1.transactionCapacity = 10000
a1.channels.ch-1.byteCapacityBufferPercentage = 20
a1.channels.ch-1.byteCapacity = 800000

################################################################
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.topic = flumetestmsg
a1.sinks.k1.brokerList = 10.255.30.132:9092
a1.sinks.k1.requiredAcks = 1
a1.sinks.k1.batchSize = 20
a1.sinks.k1.channel = c1

a1.sinks.k1.channel = ch-1
a1.sources.src-1.channels=ch-1