flume拦截mysql_Flume-自定义 Source 读取 MySQL 数据

这里记录的是自己手动实现。

测试中要读取的表

CREATE TABLE`student` (

`id`int(11) NOT NULLAUTO_INCREMENT,

`name`varchar(255) COLLATE utf8_bin DEFAULT NULL,

`age`int(11) DEFAULT NULL,PRIMARY KEY(`id`)

) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;

记录表(必须)，告诉 Flume 每次从哪开始读取

CREATE TABLE`flume_meta` (

`source_tab`varchar(255) COLLATE utf8_bin NOT NULL,

`current_index`bigint(255) DEFAULT NULL,PRIMARY KEY(`source_tab`)

) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;

一、编写自定义 Source

1.添加 pom 依赖

4.0.0

com

flume

1.0-SNAPSHOT

org.apache.flume

flume-ng-core

1.9.0

mysql

mysql-connector-java

5.1.27

org.apache.maven.plugins

maven-compiler-plugin

1.8

1.8

UTF-8

2.编写类

MySQLSourceHelper，JDBC 工具类，主要是读取数据表和更新读取记录

packagesource;importorg.apache.flume.Context;importorg.slf4j.Logger;importorg.slf4j.LoggerFactory;importjava.math.BigInteger;import java.sql.*;importjava.util.ArrayList;importjava.util.List;public classMySQLSourceHelper {private static final Logger LOG = LoggerFactory.getLogger(MySQLSourceHelper.class);//开始 id

privateString startFrom;private static final String DEFAULT_START_VALUE = "0";//表名

privateString table;//用户传入的查询的列

privateString columnsToSelect;private static final String DEFAULT_Columns_To_Select = "*";private staticString dbUrl, dbUser, dbPassword, dbDriver;private static Connection conn = null;private static PreparedStatement ps = null;//获取 JDBC 连接

private staticConnection getConnection() {try{

Class.forName(dbDriver);returnDriverManager.ge