需求
从kafka读取数据,用Struct Streaming处理,然后保存到hdfs.
案例
启动kafka生产者
kafka-console-producer.sh \
--broker-list mypc01:9092,mypc02:9092,mypc03:9092 \
--topic pet
scala代码
import org.apache.spark.sql.{DataFrame, SparkSession}
object hdfsSink extends App {
//构建spark Session
private val session: SparkSession = SparkSession.builder().master("local[2]").appName("test")
.getOrCreate()
//读取kafka上的数据
private val df: DataFrame = session.readStream.format("kafka")
.option("kafka.bootstrap.servers", "mypc01:9092,mypc02:9092,mypc03:9092")
.option("subscribe", "pet")
.load()
//处理数据,转成非二进制,因为下面要用csv存储
private val df1: DataFrame = df.selectExpr("cast(value as String)","cast(key as String)","cast(partition as Int)")
df1.writeStream
.format("csv")
.option("path", "hdfs://mypc01/sparkstreaming")
.option("checkpointLocation", "out2")
.start()
.awaitTermination()
}
注意事项
- 如果保存为
text
格式,则会报错,因为从kafka读取的数据默认有很多字段,比如key,value,Partition,等等.
Text data source supports only a single column, and you have 7 columns.
- 保存为csv格式时,注意要处理成非二进制的,否则会报错
CSV data source does not support binary data type.
- 保存为
parquet
个数比较省空间 - 一个
micro batch
会生成一个文件,(micro batch的大小暂时没找到在哪定义的)