flume监听mysql_Flume监听oracle表增量的步骤详解

需求：获取oracle表增量信息，发送至udp514端口，支持ip配置

步骤：

(1)需要的jar  oracle的 odbc5.jar(oracle安装目录  /jdbc/lib下查找)

(2)flume的开源包flume-ng-sql-source-1.4.3.jar  最新的好像是1.5的 小版本记不住了 这个下载地址直接csdn上就有

这两个jar 都拷贝到flume的lib下

(3)flume配置文件

a1.sources = r1

a1.sinks = k1

a1.channels = c1

#接收syslog配置

#a1.sources.r1.type = syslogudp

#a1.sources.r1.port = 514

#a1.sources.r1.host = 0.0.0.0

###########sql source#################

a1.sources.r1.type = org.keedio.flume.source.SQLSource

a1.sources.r1.hibernate.connection.url = jdbc:oracle:thin:@10.9.224.36:1521/soc

# Hibernate Database connection properties

a1.sources.r1.hibernate.connection.user = soc_app

a1.sources.r1.hibernate.connection.password = soc_app

a1.sources.r1.hibernate.connection.autocommit = true

a1.sources.r1.hibernate.dialect = org.hibernate.dialect.Oracle10gDialect

a1.sources.r1.hibernate.connection.driver_class = oracle.jdbc.driver.OracleDriver

a1.sources.r1.run.query.delay=1

a1.sources.r1.status.file.path = /usr/local/flume

a1.sources.r1.status.file.name = a1.sqlSource.status

# Custom query

a1.sources.r1.start.from = '2017-07-31 07:06:20'

a1.sources.r1.custom.query = SELECT CHR(39)||TO_CHAR(CREATETIME,'YYYY-MM-DD HH24:MI:SS')||CHR(39),MSG FROM FLUME_NG_SQL_SOURCE WHERE CREATETIME > TO_DATE($@$,'YYYY-MM-DD HH24:MI:SS') ORDER BY CREATETIME ASC

a1.sources.r1.batch.size = 6000

a1.sources.r1.max.rows = 1000

a1.sources.r1.hibernate.connection.provider_class = org.hibernate.connection.C3P0ConnectionProvider

a1.sources.r1.hibernate.c3p0.min_size=1

a1.sources.r1.hibernate.c3p0.max_size=10

###########sql source#################

#自定义类型

a1.sinks.k1.type = com.neusoft.utils.SyslogSink

#配置发送ip

a1.sinks.k1.destination = 10.9.224.36

a1.channels.c1.type = memory

a1.channels.c1.capacity = 10000

a1.channels.c1.transactionCapacity = 10000

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

#启动命令 ./flume-ng agent -c ../conf -f ../conf/flume-sql.conf -n a1 -Dflume.root.logger=INFO,console &

由于是自己定义的sink 在我的博客中也有提到，所以配置了自定义的sink。

总结

以上所述是小编给大家介绍的Flume监听oracle表增量的步骤详解，希望对大家有所帮助，如果大家有任何疑问请给我留言，小编会及时回复大家的。在此也非常感谢大家对我们网站的支持！

本文标题: Flume监听oracle表增量的步骤详解

本文地址: http://www.cppcns.com/shujuku/oracle/234846.html