Connector扩展说明
实现一个新的connector用户需要做的事情在上图中Planning部分,其中根据source还是sink来决定实现DynamicTableSourceFactory还是DynamicTableSinkFactory,也可以同时实现这两个接口。
以下针对JdbcDynamicTableFactory源码进行走读:
JdbcDynamicTableFactory类主体结构
public static final String IDENTIFIER = "jdbc";
其他参数见类:JdbcConnectorOptions
public class JdbcConnectorOptions { public static final ConfigOption<String> URL = ConfigOptions.key("url") .stringType() .noDefaultValue() .withDescription("The JDBC database URL.");
创建source动态表 :createDynamicTableSource
JdbcDynamicTableSource
Lookup与scan接口对比请查看官网:
User-defined Sources & Sinks | Apache Flink
lookup 简单理解为根据提交查找,不必读取整个表,并且可以在需要时从(可能不断变化的)外部表中延迟获取各个值。
scan 在运行时扫描来自外部存储系统的所有行
接下来就是构造如何读取数据库数据的select语句了
1、设置驱动、url,数据库用户名密码等信息
2、设置其他参数,fetchsize,limit,分区条件
3、根据方言构建select,设置数据库中数据类型与flink数据类型映射关系转换器
等
RuntimeProvider
build返回的为JdbcRowDataInputFormat
public JdbcRowDataInputFormat build() { if (this.queryTemplate == null) { throw new NullPointerException("No query supplied"); } if (this.rowConverter == null) { throw new NullPointerException("No row converter supplied"); } if (this.parameterValues == null) { LOG.debug("No input splitting configured (data will be read with parallelism 1)."); } return new JdbcRowDataInputFormat( new SimpleJdbcConnectionProvider(connOptionsBuilder.build()), this.fetchSize, this.autoCommit, this.parameterValues, this.queryTemplate, this.resultSetType, this.resultSetConcurrency, this.rowConverter, this.rowDataTypeInfo); }
JdbcRowDataInputFormat
Connection获取
Connection dbConn = connectionProvider.getOrEstablishConnection();
SimpleJdbcConnectionProvider
@Override public Connection getOrEstablishConnection() throws SQLException, ClassNotFoundException { if (connection != null) { return connection; } if (jdbcOptions.getDriverName() == null) { connection = DriverManager.getConnection( jdbcOptions.getDbURL(), jdbcOptions.getUsername().orElse(null), jdbcOptions.getPassword().orElse(null)); } else { Driver driver = getLoadedDriver(); Properties info = new Properties(); jdbcOptions.getUsername().ifPresent(user -> info.setProperty("user", user)); jdbcOptions.getPassword().ifPresent(password -> info.setProperty("password", password)); connection = driver.connect(jdbcOptions.getDbURL(), info); if (connection == null) { // Throw same exception as DriverManager.getConnection when no driver found to match // caller expectation. throw new SQLException( "No suitable driver found for " + jdbcOptions.getDbURL(), "08001"); } } return connection; }
读取一行数据
/** * Stores the next resultSet row in a tuple. * * @param reuse row to be reused. * @return row containing next {@link RowData} * @throws IOException */ @Override public RowData nextRecord(RowData reuse) throws IOException { try { if (!hasNext) { return null; } RowData row = rowConverter.toInternal(resultSet); // update hasNext after we've read the record hasNext = resultSet.next(); return row; } catch (SQLException se) { throw new IOException("Couldn't read data - " + se.getMessage(), se); } catch (NullPointerException npe) { throw new IOException("Couldn't access resultSet", npe); } }
把resultSet转成rowData
AbstractJdbcRowConverter
@Override public RowData toInternal(ResultSet resultSet) throws SQLException { GenericRowData genericRowData = new GenericRowData(rowType.getFieldCount()); for (int pos = 0; pos < rowType.getFieldCount(); pos++) { Object field = resultSet.getObject(pos + 1); genericRowData.setField(pos, toInternalConverters[pos].deserialize(field)); } return genericRowData; }
Dialect方言选择
根据URL 通过DialectFactory选择方言和driver,并通过继承AbstractDialect实现对应的方言
JdbcDialectFactory识别不同的URL,对应不同的数据库类型
dbcDialect
方言的接口,提供类型验证和类型转换,和insert,update,delete等语句的构造
扩展FLink jdbc connector 支持其它类型的DB数据库
扩展jdbc connector支持其它数据库需要做如下工作
1、DialectFactory实现
2、如下截图中添加上述新的实现类
JdbcDynamicTableFactory如何生效呢?
FactoryUtil中类discoverFactories方法(java SPI机制)
static List<Factory> discoverFactories(ClassLoader classLoader) { final List<Factory> result = new LinkedList<>(); ServiceLoaderUtil.load(Factory.class, classLoader) .forEach( loadResult -> { if (loadResult.hasFailed()) { if (loadResult.getError() instanceof NoClassDefFoundError) { LOG.debug( "NoClassDefFoundError when loading a " + Factory.class + ". This is expected when trying to load a format dependency but no flink-connector-files is loaded.", loadResult.getError()); // After logging, we just ignore this failure return; } throw new TableException( "Unexpected error when trying to load service provider for factories.", loadResult.getError()); } result.add(loadResult.getService()); }); return result; }