TableSource
TableSource提供对存储在外部系统(数据库,键值存储,消息队列)或文件中的数据的访问的通用接口。在TableEnvironment中注册TableSource后,可以通过Table API或SQL查询对其进行访问。它提供了表的schema 以及与该表的schema 映射到行的记录。
TableSource[T] {
def getTableSchema: TableSchema
def getReturnType: TypeInformation[T]
def explainSource: String
}
- getTableSchema() - 返回产生的表的Schema,即表的字段的名称和类型。字段类型是使用Flink的DataType定义的。
- getReturnType()- 返回DataStream(StreamTableSource)或DataSet(BatchTableSource)的物理类型以及TableSource生成的记录。
- explainSource() - 返回描述TableSource的字符串。此方法是可选的,仅用于显示目的。
TableSource接口将逻辑表Schema与返回的DataStream或DataSet的物理类型分开。因此,表Schema的所有字段(getTableSchema())必须映射到具有相应物理返回类型(getReturnType())类型的字段。默认情况下,此映射是基于字段名称完成的。例如,一个TableSource用两个字段[name:String,size:Integer]定义一个表模式,它需要一个TypeInformation,该字段至少包含两个名为name和size的字段,类型分别为String和Integer,这可以是PojoTypeInfo或RowTypeInfo,它们具有两个名为name和size且具有匹配类型的字段。
但是,某些类型(例如Tuple或CaseClass类型)确实支持自定义字段名称。如果TableSource返回具有固定字段名称类型的DataStream或DataSet,则它可以实现DefinedFieldMapping接口,以将表模式中的字段名称映射到物理返回类型的字段名称。
BatchTableSource
BatchTableSource接口扩展了TableSource接口并定义了另一种方法:
BatchTableSource[T] extends TableSource[T] {
def getDataSet(execEnv: ExecutionEnvironment): DataSet[T]
}
- getDataSet(execEnv):返回带有表数据的DataSet。 DataSet的类型必须与TableSource.getReturnType()方法定义的返回类型相同。通过使用DataSet API的data source可以创建DataSet。通常,BatchTableSource是通过包装InputFormat或batch connector来实现的。
案例实战
目前从测试的角度上讲,Flink目前还不支持Scala的Tuple和CaseClass,所以下面以Pojo和java Tuple为例,笔者可能认为这是一个bug,期待后续版本更新。
import org.apache.flink.api.scala.ExecutionEnvironment
import org.apache.flink.table.api.scala.BatchTableEnvironment
import org.apache.flink.streaming.api.scala._
object FlinkUserDefineBatchTableSource {
def main(args: Array[String]): Unit = {
val fbEnv = ExecutionEnvironment.getExecutionEnvironment
val fbtEnv = BatchTableEnvironment.create(fbEnv)
fbtEnv.registerTableSource("t_employee",`自定义的TableSource`)
val resultTable = fbtEnv.sqlQuery("select * from t_employee")
fbtEnv.toDataSet[(Int,String,Boolean,Double,String)](resultTable)
.print()
}
}
pojo案例
class Employee {
var id:Int=_
var name:String=_
var sex:Boolean=_
var salary:Double=_
var dept:String=_
def this(id: Int, name: String, sex: Boolean, salary: Double, dept: String)={
this()
this.id=id
this.name=name
this.sex=sex
this.salary=salary
this.dept=dept
}
}
object Employee{
def apply(id: Int, name: String, sex: Boolean, salary: Double, dept: String): Employee = new Employee(id, name, sex, salary, dept)
}
import org.apache.flink.api.common.typeinfo.TypeInformation
import org.apache.flink.api.java.{DataSet, ExecutionEnvironment}
import org.apache.flink.table.api.{DataTypes, TableSchema}
import org.apache.flink.table.sources.BatchTableSource
class UserDefineBatchTableSourcePojo extends BatchTableSource[Employee] {
override def getDataSet(execEnv: ExecutionEnvironment): DataSet[Employee] = {
var dataSet=execEnv.fromElements(Employee(1,"zhangsan",true,10000.0,"研发"),Employee(2,"lisi",false,15000.0,"市场"))
dataSet
}
override def getReturnType: TypeInformation[Employee] = {
TypeInformation.of(classOf[Employee])
}
override def getTableSchema: TableSchema = {
new TableSchema.Builder()
.field("id",DataTypes.INT())
.field("name",DataTypes.STRING())
.field("sex",DataTypes.BOOLEAN())
.field("salary",DataTypes.DOUBLE())
.field("dept",DataTypes.STRING())
.build()
}
}
Tupe案例
当实现Tuple的时候,需要实现DefinedFieldMapping接口,同时这里需要注意,目前Flink对Scala的支持还不够完善,这里的Tupe只能使用org.apache.flink.api.java.tuple
包下的Tuple
import java.util
import org.apache.flink.api.common.typeinfo.{TypeHint, TypeInformation}
import org.apache.flink.api.java.{DataSet, ExecutionEnvironment}
import org.apache.flink.table.api.{DataTypes, TableSchema}
import org.apache.flink.table.sources.{BatchTableSource, DefinedFieldMapping}
import org.apache.flink.api.java.tuple.Tuple5
import org.apache.flink.streaming.api.scala._
class UserDefineBatchTableSource_Tuple extends BatchTableSource[Tuple5[Int,String,Boolean,Double,String]] with DefinedFieldMapping {
override def getDataSet(execEnv: ExecutionEnvironment): DataSet[Tuple5[Int,String,Boolean,Double,String]] = {
execEnv.fromElements(new Tuple5(1,"zhangsan",true,10000.0,"研发"),new Tuple5(2,"lisi",false,15000.0,"市场"))
}
override def getReturnType: TypeInformation[Tuple5[Int,String,Boolean,Double,String]] = {
createTypeInformation[Tuple5[Int, String, Boolean, Double, String]]
}
override def getTableSchema: TableSchema = {
new TableSchema.Builder()
.field("id",DataTypes.INT())
.field("name",DataTypes.STRING())
.field("sex",DataTypes.BOOLEAN())
.field("salary",DataTypes.DOUBLE())
.field("dept",DataTypes.STRING())
.build()
}
override def getFieldMapping: util.Map[String, String] = {
val fieldMap = new util.HashMap[String, String]()
fieldMap.put("id","f0")
fieldMap.put("name","f1")
fieldMap.put("sex","f2")
fieldMap.put("salary","f3")
fieldMap.put("dept","f4")
fieldMap
}
}
InputFormatTableSource
上述案例是通过集成BatchTableSource实现,目前Flink-1.9之后BatchTableSource
已经被废弃,推荐集成InputFormatTableSource
接口
import java.util
import org.apache.flink.api.common.io.InputFormat
import org.apache.flink.api.common.typeinfo.TypeInformation
import org.apache.flink.api.java.io.TupleCsvInputFormat
import org.apache.flink.api.java.typeutils.TupleTypeInfo
import org.apache.flink.api.scala.typeutils.Types
import org.apache.flink.core.fs.Path
import org.apache.flink.table.api.{DataTypes, TableSchema}
import org.apache.flink.table.sources.{DefinedFieldMapping, InputFormatTableSource}
import org.apache.flink.api.java.tuple.Tuple5
import org.apache.flink.api.scala._
class UserDefineInputFormatTableSource extends InputFormatTableSource[Tuple5[Int,String,Boolean,Double,String]] with DefinedFieldMapping{
override def getInputFormat: InputFormat[Tuple5[Int,String,Boolean,Double,String], _] = {
new TupleCsvInputFormat[Tuple5[Int, String, Boolean, Double, String]](
new Path("D:\\Users\\Administrator\\IdeaProjects\\20200716\\FlinkTableAPI\\src\\main\\resources\\data.txt"),
"\n",
",",
new TupleTypeInfo[Tuple5[Int, String, Boolean, Double, String]](Types.INT,Types.STRING,Types.BOOLEAN,Types.DOUBLE,Types.STRING))
}
override def getReturnType: TypeInformation[Tuple5[Int, String, Boolean, Double, String]] = {
createTypeInformation[Tuple5[Int, String, Boolean, Double, String]]
}
override def getTableSchema: TableSchema = {
new TableSchema.Builder()
.field("id",DataTypes.INT())
.field("name",DataTypes.STRING())
.field("sex",DataTypes.BOOLEAN())
.field("salary",DataTypes.DOUBLE())
.field("dept",DataTypes.STRING())
.build()
}
override def getFieldMapping: util.Map[String, String] = {
val fieldMap = new util.HashMap[String, String]()
fieldMap.put("id","f0")