SparkStreaming Demo

配置

官方提供依赖添加方式：

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.12</artifactId>
    <version>2.4.6</version>
    <scope>provided</scope>
</dependency>

idea本地调试报错：

Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf

解决方案：

删掉provided

Demo

import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}

object StreamAPP {

  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local[3]").setAppName("streamTest")
    val ssc = new StreamingContext(conf, Seconds(5))

    val lines = ssc.socketTextStream("hadoop001", 9999)

    val value = lines.flatMap(_.split(",")).map(x => (x, 1)).reduceByKey(_ + _)

    value.print()
    ssc.start()
    ssc.awaitTermination()


  }
}

启动9999端口发送数据

[wzj@hadoop001 ~]$ nc -lk 9999
123
123,234,456
123,234,456
123,234,456
123,234,456
123,234,456
123,234,456
123,234,456
123,234,456
123,234,456

idea输出结果：

-------------------------------------------
Time: 1594392930000 ms
-------------------------------------------
(234,4)
(456,4)
(123,4)

-------------------------------------------
Time: 1594392935000 ms
-------------------------------------------
(234,2)
(456,2)
(123,2)

-------------------------------------------
Time: 1594392940000 ms
-------------------------------------------

---

欢迎关注公众号，一起愉快的交流