配置
官方提供依赖添加方式:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.12</artifactId>
<version>2.4.6</version>
<scope>provided</scope>
</dependency>
idea本地调试报错:
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf
解决方案:
删掉provided
Demo
import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}
object StreamAPP {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setMaster("local[3]").setAppName("streamTest")
val ssc = new StreamingContext(conf, Seconds(5))
val lines = ssc.socketTextStream("hadoop001", 9999)
val value = lines.flatMap(_.split(",")).map(x => (x, 1)).reduceByKey(_ + _)
value.print()
ssc.start()
ssc.awaitTermination()
}
}
启动9999端口发送数据
[wzj@hadoop001 ~]$ nc -lk 9999
123
123,234,456
123,234,456
123,234,456
123,234,456
123,234,456
123,234,456
123,234,456
123,234,456
123,234,456
idea输出结果:
-------------------------------------------
Time: 1594392930000 ms
-------------------------------------------
(234,4)
(456,4)
(123,4)
-------------------------------------------
Time: 1594392935000 ms
-------------------------------------------
(234,2)
(456,2)
(123,2)
-------------------------------------------
Time: 1594392940000 ms
-------------------------------------------
欢迎关注公众号,一起愉快的交流