sink 写入本地文件
2.4. Sink
Flink 没有类似于spark中 foreach 方法,让用户进行迭代的操作。虽有对外的输出操作都要利用Sink完成。最后通过类似如下方式完成整个任务最终输出操作。
stream. addsink(new MySink (xxxx) )
2.4.1 将数据写入文件(File Sink)
package com.flink.sink.study
import com.flink.streamapi.study.SensorReading
import org.apache.flink.streaming.api.scala._
object SinkStudy01 {
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
// 需设置并行度为 1 否则会写入多个文件
env.setParallelism(1)
val inputPath = "/home/ied/myFlinkStudy/resources/sensor.txt"
val StreamData = env.readTextFile(inputPath)
val resStream = StreamData
.map(x => {
val arr = x.split(", ")
SensorReading(arr(0), arr(1).toLong, arr(2).toDouble)
})
resStream.print()
// 写入文件操作
resStream.writeAsCsv("/home/ied/myFlinkStudy/resources/out.txt")
// 现在推荐的方法
resStream.addSink(StreamingFileSink.forRowFormat(
new Path("/home/ied/myFlinkStudy/resources/out2.txt"),
new SimpleStringEncoder[SensorReading]()
).build()
)
env.execute("file sink test")
}
}
resStream.writeAsCsv("/home/ied/myFlinkStudy/resources/out.txt")
resStream.addSink(StreamingFileSink.forRowFormat(
new Path("/home/ied/myFlinkStudy/resources/out2.txt"),
new SimpleStringEncoder[SensorReading]()
).build()
)