InputStream是所有输入流的抽象基类。 这个类提供start()和stop()方法,这两个方法被Spark 流系统用来启动开始接收数据和停止接收数据。
如果输入流仅在运行在driver结点的服务和线程产生的新数据生成RDD,那么可能直接继承这个InputStream。以FileINputDStream为例,它是InputStream的一个子类,在driver端监控一个HDFS目录,如果有新文件生成,则用新文件生成RDDs。
如果实现的输入流需要在工作结点运行一个接收器,使用[[org.apache.spark.streaming.dstream.ReceiverInputDStream]]作为父类。
ReceiverInputDStream是一个抽像类,用于需要在工作结点启动一个接收器来接收外部数据。 ReceiverInputDStream的具体实现必须定义getReceiver方法来得到一个[[org.apache.spark.streaming.receiver.Receiver]] 对象。Receiver对象会发送到工作结点来接收数据。
/**
* This is the abstract base class for all input streams. This class provides methods
* start() and stop() which are called by Spark Streaming system to start and stop
* receiving data, respectively.
* Input streams that can generate RDDs from new data by running a service/thread only on
* the driver node (that is, without running a receiver on worker nodes), can be
* implemented by directly inheriting this InputDStream. For example,
* FileInputDStream, a subclass of InputDStream, monitors a HDFS directory from the driver for
* new files and generates RDDs with the new files. For implementing input streams
* that requires running a receiver on the worker nodes, use
* [[org.apache.spark.streaming.dstream.ReceiverInputDStream]] as the parent class.
*
* @param _ssc Streaming context that will execute this input stream
*/
abstract class InputDStream[T: ClassTag](_ssc: StreamingContext)
extends DStream[T](_ssc) {
/**
* Abstract class for defining any [[org.apache.spark.streaming.dstream.InputDStream]]
* that has to start a receiver on worker nodes to receive external data.
* Specific implementations of ReceiverInputDStream must
* define [[getReceiver]] function that gets the receiver object of type
* [[org.apache.spark.streaming.receiver.Receiver]] that will be sent
* to the workers to receive data.
* @param _ssc Streaming context that will execute this input stream
* @tparam T Class type of the object of this stream
*/
abstract class ReceiverInputDStream[T: ClassTag](_ssc: StreamingContext)
extends InputDStream[T](_ssc) {