Spark Streaming实时流处理实战笔记七

Spark Streaming核心概念

核心概念

核心概念之StreamingContext

在IDEA中 搜索StreamingContext.scala

def this(sparkContext: SparkContext, batchDuration: Duration) = {
this(sparkContext, null, batchDuration)
}

def this(conf: SparkConf, batchDuration: Duration) = {
this(StreamingContext.createNewSparkContext(conf), null, batchDuration)
}

batch interval可以根据你的应用程序的需求的延迟要求以及集群可用的资源情况来设置

After a context is defined, you have to do the following.

1. Define the input sources by creating input DStreams.
2. Define the streaming computations by applying transformation and
   output operations to DStreams.
3. Start receiving data and processing it using
   streamingContext.start().
4. Wait for the processing to be stopped (manually or due to any error)
   using streamingContext.awaitTermination().
5. The processing can be manually stopped using
   streamingContext.stop().

一旦StreamingContext定义好之后，就可以做一些事情

Discretized Streams (DStreams)

Internally, a DStream is represented by a continuous series of RDDs
Each RDD in a DStream contains data from a certain interval

对DStream操作算子，比如map/flatMap，其实底层会被翻译为对DStream中的每个RDD都做相同因为一个DStream是由不同批次的RDD所构成的。

Input DStreams and Receivers

Input DStreams are DStreams representing the stream of input data received from streaming sources.

Every input DStream (except file stream, discussed later in this section) is associated with a Receiver (Scala doc, Java doc) object which
receives the data from a source and stores it
in Spark’s memory for processing.

Transformations

Output Operations

案例实战

案例一：Spark Streaming处理socket数据

报错 ：java.lang.NoClassDefFoundError: net/jpountz/util/SafeUtils

百度查找maven repository

案例二：Spark Streaming处理HDFS文件数据

文件系统对接的注意事项（官网查询）：

All files must be in the same data format.