Spark Streaming

spark-submit的使用（生产环境使用）

nc -lk 9999 :监听9999端口

spark-submit --master local[2] \
--class org.apache.spark.examples.streaming.NetworkWordCount  \
--name NetworkWordCount \
/root/soft_install/spark-2.2.0-bin-2.6.0-cdh5.7.0/examples/jars/spark-examples_2.11-2.2.0.jar 192.168.145.128 9999

spark-shell的使用(测试)

nc -lk 9999 :监听9999端口

spark-submit --master local[2] \
import org.apache.spark.streaming.{Seconds, StreamingContext} \
val ssc = new StreamingContext(sc, Seconds(1)) \
val lines = ssc.socketTextStream("192.168.145.128", 9999) \
val words = lines.flatMap(_.split(" "))\
val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)\
wordCounts.print()\
ssc.start()\
ssc.awaitTermination()\

工作原理：粗粒度

Spark Streaming接收到实时数据流，把数据按照指定的时间段切成一片片小的数据块，然后把小的数据块传给Spark Engine处理

工作原理：细粒度

Spark Streaming

概念

StreamingContext

是基于SparkContext

DStream

DStream represents a continuous stream of data,由多个RDD组成，所以对DStream的操作，其实是对多个RDD进行操作