kafka--Struct Streaming--hdfs案例

需求

从kafka读取数据,用Struct Streaming处理,然后保存到hdfs.

案例

启动kafka生产者

kafka-console-producer.sh \
--broker-list mypc01:9092,mypc02:9092,mypc03:9092 \
--topic pet

scala代码

import org.apache.spark.sql.{DataFrame, SparkSession}

object hdfsSink extends App {
//构建spark Session
  private val session: SparkSession = SparkSession.builder().master("local[2]").appName("test")
    .getOrCreate()
//读取kafka上的数据
  private val df: DataFrame = session.readStream.format("kafka")
    .option("kafka.bootstrap.servers", "mypc01:9092,mypc02:9092,mypc03:9092")
    .option("subscribe", "pet")
    .load()

  //处理数据,转成非二进制,因为下面要用csv存储
  private val df1: DataFrame = df.selectExpr("cast(value as String)","cast(key as String)","cast(partition as Int)")

  df1.writeStream
    .format("csv")
    .option("path", "hdfs://mypc01/sparkstreaming")
    .option("checkpointLocation", "out2")
    .start()
    .awaitTermination()
}

注意事项

• 如果保存为text格式,则会报错,因为从kafka读取的数据默认有很多字段,比如key,value,Partition,等等.

Text data source supports only a single column, and you have 7 columns.

• 保存为csv格式时,注意要处理成非二进制的,否则会报错

 CSV data source does not support binary data type.

• 保存为parquet个数比较省空间
• 一个micro batch会生成一个文件,(micro batch的大小暂时没找到在哪定义的)