flink中通过jdbc查询结果集使用 flink table api 创建临时视图

最新推荐文章于 2024-04-12 15:54:35 发布

Vicky_Tang

最新推荐文章于 2024-04-12 15:54:35 发布

阅读量502

点赞数 1

文章标签： flink 大数据 scala

本文链接：https://blog.csdn.net/sweet19920711/article/details/129582670

版权

1.maven依赖

<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-table-planner-blink_${scala.version}</artifactId>
  <version>${flink.version}</version>
  <scope>provided</scope>
</dependency>

<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-table-api-scala-bridge_${scala.version}</artifactId>
  <version>${flink.version}</version>
  <scope>provided</scope>
</dependency>

<!--Table的公共依赖-->
<!-- https://mvnrepository.com/artifact/org.apache.flink/flink-table-common -->
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-table-common</artifactId>
  <scope>provided</scope>
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-json</artifactId>
  <version>${flink.version}</version>
</dependency>

<dependency>
  <groupId>org.apache.hive</groupId>
  <artifactId>hive-jdbc</artifactId>
  <version>1.1.0</version>
  <exclusions>
    <exclusion>
      <artifactId>jetty-all</artifactId>
      <groupId>org.eclipse.jetty.aggregate</groupId>
    </exclusion>
    <exclusion>
      <artifactId>jetty</artifactId>
      <groupId>org.mortbay.jetty</groupId>
    </exclusion>
    <exclusion>
      <artifactId>jetty-util</artifactId>
      <groupId>org.mortbay.jetty</groupId>
    </exclusion>
    <exclusion>
      <artifactId>protobuf-java</artifactId>
      <groupId>com.google.protobuf</groupId>
    </exclusion>
  </exclusions>
</dependency>

2.创建TableEnventment环境

val settings = EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build()
val env = StreamExecutionEnvironment.getExecutionEnvironment
val tableEnvironment = StreamTableEnvironment.create(env,settings)

3.使用jdbc获取结果集

3.1创建样例类

private case class dict(dict_type:String, dict_name:String)

3.2创建jdbc连接获取结果集

val userName = "*******"
val password = "*******"
val driver = "org.apache.hive.jdbc.HiveDriver"
val hiveProperties = new Properties()
hiveProperties.setProperty("driver",driver)
hiveProperties.setProperty("userName",userName)
hiveProperties.setProperty("password",password)
Class.forName(driver)
val conn = DriverManager.getConnection("jdbc:hive2://*************",userName,password)

//查询为表数据，并将数据写入ArrayBuff
val res = conn.prepareStatement(
  """
|select dict_type1,dict_name1
|from dwd_order.t_dimersion_ed
|where dict_type = '003'
|""".stripMargin).executeQuery()
var arrayBuff:ArrayBuffer[dict] = ArrayBuffer()
while (res.next()) {
  arrayBuff.append(dict(res.getString("dict_type1"),
                        res.getString("dict_name1")))
}

4.将ArrayBuff转化为DataStream后生成临时视图

    //将数据转化成DataStream
    val dimStream = env.fromCollection[dict](arrayBuff.toSeq)

    //创建维表临时视图
    tableEnvironment.createTemporaryView("temp_dict",dimStream)