FLink学习笔记：04-Flink DataStream的TransForm操作

最新推荐文章于 2023-04-25 11:16:05 发布
最新推荐文章于 2023-04-25 11:16:05 发布
阅读量273 收藏
点赞数 1
分类专栏： FLink 大数据 文章标签： scala flink
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wangzhongyudie/article/details/124441023
版权

文章目录

map

说明: 一个输入对应一个输出

示例代码:

package com.hjt.yxh.apitest

import org.apache.flink.api.common.functions.{FilterFunction, FlatMapFunction, MapFunction, ReduceFunction}
import org.apache.flink.streaming.api.functions.co.{CoFlatMapFunction, CoMapFunction}
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.scala._
import org.apache.flink.util.Collector

case class Student(id: String,name: String,age: Int)

class KeyWordFiter(keyWord: String) extends FilterFunction[Student]{
  override def filter(t: Student): Boolean = {
    t.name.contains(keyWord)
  }
}

class StudentMapper extends MapFunction[String,Student]{
  override def map(t: String): Student = {
    val arry = t.split(",")
    Student(arry(0),arry(1),arry(2).toInt)
  }
}

object TransFormTest {

  def main(args: Array[String]): Unit = {

    //1.创建环境
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    env.setParallelism(1)

    //2.创建文件流
    val inputPath = "D:\\java_workspace\\hadoop\\FlinkDemo\\src\\main\\resources\\Data\\student.txt"
    val dataSteam = env.readTextFile(inputPath)

    //map 匿名函数
    val mapStream: DataStream[Student] = dataSteam.map(data=>{
      val arry = data.split(",")
      Student(arry(0),arry(1),arry(2).toInt)
    })
    mapStream.print("mapFunction:匿名函数")

    //map 匿名类
    val mapStream2: DataStream[Student] = dataSteam.map(new MapFunction[String,Student]{
      override def map(t: String): Student = {
        val arry = t.split(",")
        Student(arry(0),arry(1),arry(2).toInt)
      }
    })
    mapStream2.print("mapFunction:匿名类")

    //map 自定义类
    val mapStream3:DataStream[Student] = dataSteam.map(new StudentMapper)
    mapStream3.print("mapFunction:自定义函数")

    env.execute("transform test")
  }
}

flatMap

对输入的对象做操作，输出可能是一个，或者多个，或者是空

代码:

package com.hjt.yxh.apitest

import org.apache.flink.api.common.functions.{FilterFunction, FlatMapFunction, MapFunction, ReduceFunction}
import org.apache.flink.streaming.api.functions.co.{CoFlatMapFunction, CoMapFunction}
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.scala._
import org.apache.flink.util.Collector

case class Student(id: String,name: String,age: Int)

class StudentFlatMapper extends FlatMapFunction[String,Student]{
  override def flatMap(t: String, collector: Collector[Student]): Unit = {
    val arry = t.split(",")
    collector.collect(Student(arry(0),arry(1),arry(2).toInt))
  }
}

object TransFormTest {

  def main(args: Array[String]): Unit = {

    //1.创建环境
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    env.setParallelism(1)

    //2.创建文件流
    val inputPath = "D:\\java_workspace\\hadoop\\FlinkDemo\\src\\main\\resources\\Data\\student.txt"
    val dataSteam = env.readTextFile(inputPath)

    //flatMap 匿名函数
    val flatMapStream2 = dataSteam.flatMap(data=>{
      val arry = data.split(",")
      List(Student(arry(0),arry(1),arry(2).toInt))
    })
    flatMapStream2.print("匿名函数:flatMapFunction")

    //flatMap 匿名类实现
    val flatMapStream = dataSteam.flatMap(new FlatMapFunction[String,Student] {
      override def flatMap(t: String, collector: Collector[Student]): Unit = {
        val arry = t.split(",")
        collector.collect(Student(arry(0),arry(1),arry(2).toInt))
      }
    })
    flatMapStream.print("自定义匿名类flatMapFunction:")

    //flatMap 自定义类
    val flatMapStream3 = dataSteam.flatMap(new StudentFlatMapper)
    flatMapStream3.print("自定义类:flatMapFunction")
//    mapStream.print()
    env.execute("transform test")
  }
}

filter

对元素做过滤操作，只返回符合条件的元素

package com.hjt.yxh.apitest

import org.apache.flink.api.common.functions.{FilterFunction, FlatMapFunction, MapFunction, ReduceFunction}
import org.apache.flink.streaming.api.functions.co.{CoFlatMapFunction, CoMapFunction}
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.scala._
import org.apache.flink.util.Collector

case class Student(id: String,name: String,age: Int)

class KeyWordFiter(keyWord: String) extends FilterFunction[Student]{
  override def filter(t: Student): Boolean = {
    t.name.contains(keyWord)
  }
}

object TransFormTest {

  def main(args: Array[String]): Unit = {

    //1.创建环境
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    env.setParallelism(1)

    //2.创建文件流
    val inputPath = "D:\\java_workspace\\hadoop\\FlinkDemo\\src\\main\\resources\\Data\\student.txt"
    val dataSteam = env.readTextFile(inputPath)

    //map 匿名函数
    val mapStream: DataStream[Student] = dataSteam.map(data=>{
      val arry = data.split(",")
      Student(arry(0),arry(1),arry(2).toInt)
    })
    
    //filter 匿名函数lambda 表达式
    val filterStream = mapStream.filter(student=>student.name.contains("wang"))
    filterStream.print("filterFunction:匿名函数")

    //自定义filter类
    val filterStream2 = mapStream.filter(new KeyWordFiter("wang"))
    filterStream2.print("filterFunction:自定义类")

    //自定义匿名类
    val filterStream3 = mapStream.filter(new FilterFunction[Student] {
      override def filter(t: Student): Boolean = {
        t.name.contains("wang")
      }
    })
    filterStream3.print("filterFunction:匿名类")
    
    env.execute("transform test")

  }

}

keyBy与滚动聚合算子

简单的聚合算子有

  • min() 分组后求最小值，如果是样例，最有最小的值是最小记录，其他的成员的值是第一条记录的值
  • max() 求最大值
  • sum() 求和
  • minBy() 最小值所在的记录
  • maxBy() 最大值所在的记录

keyBy与reduce

KeyedStream → DataStream：一个分组数据流的聚合操作，合并当前的元素 和上次聚合的结果，产生一个新的值，返回的流中包含每一次聚合的结果，而不是 只返回最后一次聚合的最终结果。

package com.hjt.yxh.apitest

import org.apache.flink.api.common.functions.{FilterFunction, FlatMapFunction, MapFunction, ReduceFunction}
import org.apache.flink.streaming.api.functions.co.{CoFlatMapFunction, CoMapFunction}
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.scala._
import org.apache.flink.util.Collector

case class Student(id: String,name: String,age: Int)


class StudentReducer extends ReduceFunction[Student]{
  override def reduce(t: Student, t1: Student): Student = {
     if(t.age<t1.age) t else t1
  }
}

object TransFormTest {

  def main(args: Array[String]): Unit = {

    //1.创建环境
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    env.setParallelism(1)

    //2.创建文件流
    val inputPath = "D:\\java_workspace\\hadoop\\FlinkDemo\\src\\main\\resources\\Data\\student.txt"
    val dataSteam = env.readTextFile(inputPath)

    //简单聚合算子
    val aggrateStream = mapStream.keyBy("id").minBy("age")
    aggrateStream.print("滚动聚合算子")

    //reduce 匿名函数
    val reduceStream = mapStream.keyBy("id").reduce((curStudent,newStudent)=>{
      if(curStudent.age<newStudent.age) Student(curStudent.id,curStudent.name,curStudent.age)
      else Student(newStudent.id,newStudent.name,newStudent.age)
    })
    reduceStream.print("reduceFunction:匿名函数")

    //reduce 匿名类
    val reduceStream2 = mapStream.keyBy("id").reduce(new ReduceFunction[Student]{
      override def reduce(t: Student, t1: Student): Student = {
        if(t.age<t1.age) t else t1
      }
    })
    reduceStream2.print("reduceFunction:匿名类")
    //reduce 自定义函数
    val reduceStream3 = mapStream.keyBy("id").reduce(new StudentReducer)
    reduceStream3.print("reduceFunction:自定义类")

    env.execute("transform test")
  }

}

分流 split和Select

  • split
    DataStream → SplitStream：根据某些特征把一个 DataStream 拆分成两个或者 多个 DataStream
  • select
    SplitStream→DataStream：从一个 SplitStream 中获取一个或者多个 DataStream

代码:

package com.hjt.yxh.apitest

import org.apache.flink.api.common.functions.{FilterFunction, FlatMapFunction, MapFunction, ReduceFunction}
import org.apache.flink.streaming.api.functions.co.{CoFlatMapFunction, CoMapFunction}
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.scala._
import org.apache.flink.util.Collector

case class Student(id: String,name: String,age: Int)


object TransFormTest {

  def main(args: Array[String]): Unit = {

    //1.创建环境
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    env.setParallelism(1)

    //2.创建文件流
    val inputPath = "D:\\java_workspace\\hadoop\\FlinkDemo\\src\\main\\resources\\Data\\student.txt"
    val dataSteam = env.readTextFile(inputPath)

    //map 匿名函数
    val mapStream: DataStream[Student] = dataSteam.map(data=>{
      val arry = data.split(",")
      Student(arry(0),arry(1),arry(2).toInt)
    })

    //分流
    val splitStream = mapStream.split(data=>{
      if (data.age>=18 && data.age <=55) List("青年人") else if (data.age > 55) List("老年人") else List("未成年人")
    })
    val manStream = splitStream.select("青年人")
    val childrenStream = splitStream.select("未成年人")
    val allManStream = splitStream.select("成年人","未成年人","老年人")
    manStream.print("青年人")    childrenStream.print("未成年人")
    allManStream.print("All")
    
    env.execute("transform test")
  }
}

合流

Union

DataStream → DataStream：对两个或者两个以上的 DataStream 进行 union 操 作，产生一个包含所有 DataStream 元素的新 DataStream。

package com.hjt.yxh.apitest

import org.apache.flink.api.common.functions.{FilterFunction, FlatMapFunction, MapFunction, ReduceFunction}
import org.apache.flink.streaming.api.functions.co.{CoFlatMapFunction, CoMapFunction}
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.scala._
import org.apache.flink.util.Collector

case class Student(id: String,name: String,age: Int)

class KeyWordFiter(keyWord: String) extends FilterFunction[Student]{
  override def filter(t: Student): Boolean = {
    t.name.contains(keyWord)
  }
}

class StudentMapper extends MapFunction[String,Student]{
  override def map(t: String): Student = {
    val arry = t.split(",")
    Student(arry(0),arry(1),arry(2).toInt)
  }
}

class StudentFlatMapper extends FlatMapFunction[String,Student]{
  override def flatMap(t: String, collector: Collector[Student]): Unit = {
    val arry = t.split(",")
    collector.collect(Student(arry(0),arry(1),arry(2).toInt))
  }
}

class StudentReducer extends ReduceFunction[Student]{
  override def reduce(t: Student, t1: Student): Student = {
     if(t.age<t1.age) t else t1
  }
}

class StudentCoMapper extends CoMapFunction[(String,String),Student,(String,String,String)]{
  override def map1(in1: (String, String)): (String,String,String) = {
    (in1._1,in1._2,"正常人")
  }

  override def map2(in2: Student): (String,String,String) = {
    (in2.id, in2.name, "病人")
  }
}

class StudentCoFlatMapper extends CoFlatMapFunction[(String,String),Student,Tuple3[String,String,String]]{
  override def flatMap1(in1: (String, String), collector: Collector[Tuple3[String,String,String]]): Unit = {
    collector.collect(Tuple3(in1._1,in1._2,"正常人"))
  }

  override def flatMap2(in2: Student, collector: Collector[Tuple3[String,String,String]]): Unit = {
    collector.collect(Tuple3(in2.id,in2.name,"病人"))
  }
}

object TransFormTest {

  def main(args: Array[String]): Unit = {

    //1.创建环境
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    env.setParallelism(1)

    //2.创建文件流
    val inputPath = "D:\\java_workspace\\hadoop\\FlinkDemo\\src\\main\\resources\\Data\\student.txt"
    val dataSteam = env.readTextFile(inputPath)

    //map 匿名函数
    val mapStream: DataStream[Student] = dataSteam.map(data=>{
      val arry = data.split(",")
      Student(arry(0),arry(1),arry(2).toInt)
    })

    //分流
    val splitStream = mapStream.split(data=>{
      if (data.age>=18 && data.age <=55) List("青年人") else if (data.age > 55) List("老年人") else List("未成年人")
    })
    val manStream = splitStream.select("青年人")
    val childrenStream = splitStream.select("未成年人")
    val allManStream = splitStream.select("成年人","未成年人","老年人")
//    manStream.print("青年人")
//    childrenStream.print("未成年人")
//    allManStream.print("All")

    //合流
    val unionStream:DataStream[Student] = manStream.union(childrenStream)
    unionStream.print()

    env.execute("transform test")

  }

}

Connect

DataStream,DataStream → ConnectedStreams：连接两个保持他们类型的数 据流，两个数据流被 Connect 之后，只是被放在了一个同一个流中，内部依然保持 各自的数据和形式不发生任何变化，两个流相互独立。

CoMap

类似于map，只是操作的对象变成了合流后的对象

package com.hjt.yxh.apitest

import org.apache.flink.api.common.functions.{FilterFunction, FlatMapFunction, MapFunction, ReduceFunction}
import org.apache.flink.streaming.api.functions.co.{CoFlatMapFunction, CoMapFunction}
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.scala._
import org.apache.flink.util.Collector

case class Student(id: String,name: String,age: Int)

class StudentCoMapper extends CoMapFunction[(String,String),Student,(String,String,String)]{
  override def map1(in1: (String, String)): (String,String,String) = {
    (in1._1,in1._2,"正常人")
  }

  override def map2(in2: Student): (String,String,String) = {
    (in2.id, in2.name, "病人")
  }
}

object TransFormTest {

  def main(args: Array[String]): Unit = {

    //1.创建环境
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    env.setParallelism(1)

    //2.创建文件流
    val inputPath = "D:\\java_workspace\\hadoop\\FlinkDemo\\src\\main\\resources\\Data\\student.txt"
    val dataSteam = env.readTextFile(inputPath)


    //分流
    val splitStream = mapStream.split(data=>{
      if (data.age>=18 && data.age <=55) List("青年人") else if (data.age > 55) List("老年人") else List("未成年人")
    })
    val manStream = splitStream.select("青年人")
    val childrenStream = splitStream.select("未成年人")
    val allManStream = splitStream.select("成年人","未成年人","老年人")
//    manStream.print("青年人")
//    childrenStream.print("未成年人")
//    allManStream.print("All")

    //合流
    val unionStream:DataStream[Student] = manStream.union(childrenStream)
//    unionStream.print()
    val newStream:DataStream[(String,String)] = childrenStream.map(data=>{
      (data.id,data.name)
    })
    val conncetStream = newStream.connect(manStream)

    //CoMap
    val coMapStream = conncetStream.map(data1=>(data1._1,data1._2,"正常人"),data2=>(data2.id,data2.name,"不正常"))
    coMapStream.print("CoMapFunction:匿名函数")

    //CoMap
    val coMapStream2 = conncetStream.map(new CoMapFunction[(String,String),Student,Tuple3[String,String,String]] {
      override def map1(in1: (String, String)): (String,String,String) = {
        (in1._1,in1._2,"正常人")
      }
      override def map2(in2: Student): (String,String,String) = {
        (in2.id,in2.name,"病人")
      }
    })
    coMapStream2.print("CoMapFunction:匿名类")

    //CoMap
    val coMapStream3 = conncetStream.map(new StudentCoMapper)
    coMapStream3.print("CoMapFunction:自定义类")

    env.execute("transform test")

  }

}

CoFlatMap

package com.hjt.yxh.apitest

import org.apache.flink.api.common.functions.{FilterFunction, FlatMapFunction, MapFunction, ReduceFunction}
import org.apache.flink.streaming.api.functions.co.{CoFlatMapFunction, CoMapFunction}
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.scala._
import org.apache.flink.util.Collector

case class Student(id: String,name: String,age: Int)

class StudentCoFlatMapper extends CoFlatMapFunction[(String,String),Student,Tuple3[String,String,String]]{
  override def flatMap1(in1: (String, String), collector: Collector[Tuple3[String,String,String]]): Unit = {
    collector.collect(Tuple3(in1._1,in1._2,"正常人"))
  }

  override def flatMap2(in2: Student, collector: Collector[Tuple3[String,String,String]]): Unit = {
    collector.collect(Tuple3(in2.id,in2.name,"病人"))
  }
}

object TransFormTest {

  def main(args: Array[String]): Unit = {

    //1.创建环境
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    env.setParallelism(1)

    //2.创建文件流
    val inputPath = "D:\\java_workspace\\hadoop\\FlinkDemo\\src\\main\\resources\\Data\\student.txt"
    val dataSteam = env.readTextFile(inputPath)

    //map 匿名函数
    val mapStream: DataStream[Student] = dataSteam.map(data=>{
      val arry = data.split(",")
      Student(arry(0),arry(1),arry(2).toInt)
    })

    //分流
    val splitStream = mapStream.split(data=>{
      if (data.age>=18 && data.age <=55) List("青年人") else if (data.age > 55) List("老年人") else List("未成年人")
    })
    val manStream = splitStream.select("青年人")
    val childrenStream = splitStream.select("未成年人")
    val allManStream = splitStream.select("成年人","未成年人","老年人")
//    manStream.print("青年人")
//    childrenStream.print("未成年人")
//    allManStream.print("All")

    //合流
    val unionStream:DataStream[Student] = manStream.union(childrenStream)
//    unionStream.print()
    val newStream:DataStream[(String,String)] = childrenStream.map(data=>{
      (data.id,data.name)
    })
    val conncetStream = newStream.connect(manStream)

    //CoFlatMap
    val coFlatMapStream2 = conncetStream.flatMap(new CoFlatMapFunction[(String,String),Student,Tuple3[String,String,String]] {
      override def flatMap1(in1: (String, String), collector: Collector[(String,String,String)]): Unit = {
        collector.collect((in1._1,in1._2,"正常人"))
      }

      override def flatMap2(in2: Student, collector: Collector[Tuple3[String,String,String]]): Unit = {
        collector.collect((in2.id,in2.name,"病人"))
      }
    })
    coFlatMapStream2.print("CoFlatMapFunction:匿名类")

    val coFlatMapStream3 = conncetStream.flatMap(new StudentCoFlatMapper)
    coFlatMapStream3.print("CoFlatMapFunction:自定义类")

    env.execute("transform test")

  }

}
wangzhongyudie
关注
  • 1
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Flink学习笔记 --- DataStream Transformations
杨鑫newlife的专栏
07-03 1119
Flink学习笔记 --- DataStream Transformations
Flink系列：DataStream主要操作
NIO4444
07-18 283
Flink DataStream API——Transform
Fenggms的博客
10-30 401
map val streamMap = stream.map { x => x * 2 } flatMap val streamFlatMap = stream.flatMap{ x => x.split(" ") } Filter val streamFilter = stream.filter{ x => x == 1 } KeyBy DataStream → KeyedSt...
快速入门Flink (8) —— DataStream 的 Transformation常用操作
大数据梦想家
08-10 3507
写在前面： 博主是一名大数据的初学者，昵称来源于《爱丽丝梦游仙境》中的Alice和自己的昵称。作为一名互联网小白，写博客一方面是为了记录自己的学习历程，一方面是希望能够帮助到很多和自己一样处于起步阶段的萌新。由于水平有限，博客中难免会有一些错误，有纰漏之处恳请各位大佬不吝赐教！个人小站:http://alices.ibilibili.xyz/ , 博客主页:https://alice.blog.csdn.net/ 尽管当前水平可能不及各位大佬，但我还是希望自己能够做得更好，因为一天的生活就是一生的缩影。.
flink笔记6 DataStream API（二）Transform、sink介绍和使用
weixin_43631997的博客
04-12 373
3.Transform (1) 简单转换算子 ① Map：输入一个元素，然后返回一个元素，中间可以做一些清洗转换等操作 object Transform1 { def main(args: Array[String]): Unit = { val inputdata = List(1,2,3,4,5) val resultdata = inputdata.map(_+10) print(resultdata) } } // 结果: List(11, 1.
flink-table-runtime-blink_2.11-1.10.0-API文档-中文版.zip
04-23
赠送jar包：flink-table-runtime-blink_2.11-1.10.0.jar； 赠送原API文档：flink-table-runtime-blink_2.11-1.10.0-javadoc.jar； 赠送源代码：flink-table-runtime-blink_2.11-1.10.0-sources.jar； 赠送Maven依赖...
flink-connector-kafka-0.10-2.11-1.10.0-API文档-中文版.zip
04-23
赠送jar包：flink-connector-kafka-0.10_2.11-1.10.0.jar； 赠送原API文档：flink-connector-kafka-0.10_2.11-1.10.0-javadoc.jar； 赠送源代码：flink-connector-kafka-0.10_2.11-1.10.0-sources.jar； 赠送Maven...
flink-table-planner-blink_2.11-1.13.2-API文档-中文版.zip
06-05
赠送jar包：flink-table-planner-blink_2.11-1.13.2.jar； 赠送原API文档：flink-table-planner-blink_2.11-1.13.2-javadoc.jar； 赠送源代码：flink-table-planner-blink_2.11-1.13.2-sources.jar； 赠送Maven依赖...
flink-table-planner-blink_2.11-1.13.2-API文档-中英对照版.zip
06-05
赠送jar包：flink-table-planner-blink_2.11-1.13.2.jar； 赠送原API文档：flink-table-planner-blink_2.11-1.13.2-javadoc.jar； 赠送源代码：flink-table-planner-blink_2.11-1.13.2-sources.jar； 赠送Maven依赖...
flink-connector-base-1.13.2-API文档-中英对照版.zip
06-05
赠送jar包：flink-connector-base-1.13.2.jar； 赠送原API文档：flink-connector-base-1.13.2-javadoc.jar； 赠送源代码：flink-connector-base-1.13.2-sources.jar； 赠送Maven依赖信息文件：flink-connector-base-...
Flink_DataStream 的 Transformation
qq_44509920的博客
07-19 147
文章目录DataStream 的 Transformation1. KeyBy2. Connect3. Split 和 select数据输出 Data Sinks1.sink 到 kafka2. sink 到 mysql DataStream 的 Transformation 1. KeyBy 逻辑上将一个流分成不相交的分区，每个分区包含相同键的元素。在内部，这是通过散 列分区来实现的 package com.czxy.flink.stream.transformation import org.apac
Flink学习8---DataStream之Transformation算子简介
zhuzuwei的博客
07-05 642
一、Flink针对DataStream提供了大量的已经实现的算子。 1. map: 输入一个元素，返回一个元素，中间可以进行清洗转换等操作。 2.FlatMap: 压平，即将嵌套集合转换并平铺成非嵌套集合，可以根据业务需求返回0个、一个或者多个元素。 3. Filter: 过滤函数，对传入的数据进行判断，符合条件的数据才会被留下。 4.KeyBy: 根据指定的Key进行分组，Key相同的数据会进入同一个分区。用法： （1）DataStream.keyBy("key")指定对象中的具体key字段.
Flink系列-10、Flink DataStream的Transformation
最新发布
不积跬步无以至千里，不积小流无以成江河
04-25 324
Flink系列-10、Flink DataStream的Transformation
Flink DataStream 编程入门
weixin_33842304的博客
06-19 188
流处理是 Flink 的核心，流处理的数据集用 DataStream 表示。数据流从可以从各种各样的数据源中创建（消息队列、Socket 和 文件等），经过 DataStream 的各种 transform 操作，最终输出文件或者标准输出。这个过程跟之前文章中介绍的 Flink 程序基本骨架一样。本篇介绍 DataStream 相关的入门知识。Flink 101为了学习 Flink 的朋友能查看到...
Flink DataStream 常用 Transformations 算子、测流、自定义分区器
qq_41301707的博客
03-08 352
目录1. DataStream 常用 Transformations 算子1.1 Map1.2 FlatMap1.3 Filter1.4 KeyBy1.5 Reduce1.6 Fold1.7 Aggregations1.8 Union1.9 Connect1.10 Split + Select2. 测流2.1 Filter 分流2.2 Split + Select 分流2.3 测流输出3. 自定义分区器 1. DataStream 常用 Transformations 算子 数据 hadoop,spar
Flink流处理——Transform多流转换
m0_57126456的博客
11-12 1340
Split 和 Select split: DataStream → SplitStream：根据某些特征把一个 DataStream 拆分成两个或者多个 DataStream。 Select: SplitStream→DataStream：从一个 SplitStream 中获取一个或者多个DataStream package com.atguigu.transfrom; import com.atguigu.bean.SensorReading; import org.a
Flink / Scala - 5.DataStream Transformations 常用转换函数详解
BITDDD小栈
04-06 2155
本文介绍 Flink 的主要数据形式: DataStream，即流式数据的常用转换函数，通过 Transformation 可以将一个 DataStream 转换为新的 DataStream。
4.1.11 Flink-流处理框架-Flink流处理API之Transform转换算子
敲代码的乔帮主
02-21 483
1.基本转换算子 1.1 map map算子属于基本转换算子，来一个数据处理一个。 DataStream<Integer> mapStram = dataStream.map(new MapFunction<String, Integer>() { public Integer map(String value) throws Exception { return value.length(); } }); 1.2 flatMap扁平化算子 DataS...
Flink-DataStream快速上手
qq_44002865的博客
10-11 897
Flink的详细使用 1. 安装部署 安装 第一步：将 flink-1.10.1-bin-scala_2.12.tgz 上传到服务器中并解压缩 第二步：修改 conf/flink-conf.yaml 文件 # 修改 jobmanager.rpc.address 参数，修改为 jobmanager 的机器 jobmanager.rpc.address: hadoop151 第三步：修改 conf/slaves 文件 # slave 机器 hadoop152 hadoop153 第四步：将
flink通过flink-connector-tidb-cdc使用streamapi方式连接tidb
04-04
Flink通过Flink-connector-tidb-cdc连接TiDB的过程如下： 1. 准备工作 在开始使用Flink-connector-tidb-cdc之前，需要准备以下内容： - TiDB实例 - TiCDC组件 - Flink-connector-tidb-cdc依赖的jar包 2. 创建...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值