Spark 2.0 streaming中 InputDStream 和 ReceiverInputDStream 的区别-CSDN博客

本文链接：https://blog.csdn.net/houzhizhen/article/details/53809105

InputStream是所有输入流的抽象基类。这个类提供start()和stop()方法，这两个方法被Spark 流系统用来启动开始接收数据和停止接收数据。

如果输入流仅在运行在driver结点的服务和线程产生的新数据生成RDD，那么可能直接继承这个InputStream。以FileINputDStream为例，它是InputStream的一个子类，在driver端监控一个HDFS目录，如果有新文件生成，则用新文件生成RDDs。

如果实现的输入流需要在工作结点运行一个接收器，使用[[org.apache.spark.streaming.dstream.ReceiverInputDStream]]作为父类。

ReceiverInputDStream是一个抽像类，用于需要在工作结点启动一个接收器来接收外部数据。 ReceiverInputDStream的具体实现必须定义getReceiver方法来得到一个[[org.apache.spark.streaming.receiver.Receiver]] 对象。Receiver对象会发送到工作结点来接收数据。

/**
 * This is the abstract base class for all input streams. This class provides methods
 * start() and stop() which are called by Spark Streaming system to start and stop
 * receiving data, respectively.
 * Input streams that can generate RDDs from new data by running a service/thread only on
 * the driver node (that is, without running a receiver on worker nodes), can be
 * implemented by directly inheriting this InputDStream. For example,
 * FileInputDStream, a subclass of InputDStream, monitors a HDFS directory from the driver for
 * new files and generates RDDs with the new files. For implementing input streams
 * that requires running a receiver on the worker nodes, use
 * [[org.apache.spark.streaming.dstream.ReceiverInputDStream]] as the parent class.
 *
 * @param _ssc Streaming context that will execute this input stream
 */
abstract class InputDStream[T: ClassTag](_ssc: StreamingContext)
  extends DStream[T](_ssc) {

/**
 * Abstract class for defining any [[org.apache.spark.streaming.dstream.InputDStream]]
 * that has to start a receiver on worker nodes to receive external data.
 * Specific implementations of ReceiverInputDStream must
 * define [[getReceiver]] function that gets the receiver object of type
 * [[org.apache.spark.streaming.receiver.Receiver]] that will be sent
 * to the workers to receive data.
 * @param _ssc Streaming context that will execute this input stream
 * @tparam T Class type of the object of this stream
 */
abstract class ReceiverInputDStream[T: ClassTag](_ssc: StreamingContext)
  extends InputDStream[T](_ssc) {