作者:大数据技术研发人员:谢彪
1、ReceiverSupervisor是处理Receiver接收的数据,“Receive”不断接收数据并把数据交给ReceiverSupervisor进行处理,ReceiverSupervisor接收到数据之后,会“存储数据”,之后把“信息”汇报给“ReceiverTracker”,其实是汇报给ReceiverTracker之中的IPC。
2、"BlockGenerator"是存储数据,它继承于“rateLimiter”, rateLimiter是限制存储数据,在Spark Streaming接收数据过程中我们无法控制“接收数据”的速度,但可以控制“rateLimiter”来存储数据速度,从而控制"存储数据"的速度,BlockGenerator的产生是由ReceiverSupervisor其里面有一个类ReceiverSupervisorImpl的CreatBlockGenerator产生的,代码:
valnewBlockgenerator = new BlockGenerator(blockGenerationListener,Streamid, env.cof、、、、)
以上作用显而可见,1个BlockGenerator只服务于1个inputDstream.
接收数据的时候肯定有一个循环器不断的接收数据,接收到数据肯定也有存储器,存储过之后向Driver汇报。接收数据和存储数据当然要分为两个不同的模块。
Sparkstreaming接收数据的整个流程类似于MVC模式,M就是Receiver,V就是Driver,C就是ReceiverSupervisor
ReceiverSupervisorImpl是receiver的监控器,同时负责receiver的写操作 这个方法需要传入一个Iterator,实时上里边就只有一个Receiver
valstartReceiverFunc: Iterator[Receiver[_]] => Unit =
(iterator:Iterator[Receiver[_]]) => {
if(!iterator.hasNext) {
thrownew SparkException(
"Couldnot start receiver as object not found.")
}
//重试次数必须是0,Receiver失败后只能重新调度启动
if(TaskContext.get().attemptNumber() == 0) {
获得receiver,这个receiver是根据数据输入来源InputDstream获得的receiver。以SocketInputDstream为例,它的receiver就是SocketReceiver.这里的receiver只是一个引用,并没有被实例化。作为一个参数传入ReceiverSupervisorImpl
valreceiver = iterator.next()
assert(iterator.hasNext== false)
valsupervisor = new ReceiverSupervisorImpl(
receiver,SparkEnv.get, serializableHadoopConf.value, checkpointDirOption)
supervisor.start()
supervisor.awaitTermination()
}else {
//It's restarted by TaskScheduler, but we want to reschedule it again. So exitit.
}
}
//Create the RDD using the scheduledLocations to run the receiver in aSpark job
//把receiver封装进一个RDD
valreceiverRDD: RDD[Receiver[_]] =
if(scheduledLocations.isEmpty) {
ssc.sc.makeRDD(Seq(receiver),1)
}else {
valpreferredLocations = scheduledLocations.map(_.toString).distinct
ssc.sc.makeRDD(Seq(receiver-> preferredLocations))
}
receiverRDD.setName(s"Receiver$receiverId")
ssc.sparkContext.setJobDescription(s"Streamingjob running receiver $receiverId")
ssc.sparkContext.setCallSite(Option(ssc.getStartSite()).getOrElse(Utils.getCallSite()))
为了启动Receiver启动了一个spark作业,每一个Receiver的启动都会有一个作业来负责,Receiver是一个一个的启动的如果是将所有的Receiver作为一个作业的不同tesk来启动会有很多弱点。
1.Reciver启动可能失败进而导致应用程序失败
2.运行的过程中会有任务倾斜的问题,将所有的Receiver作为一个作业的不同tesk来运行是采用的spark core的调度方式,在很不幸的情况下会出现所有Receiver运行在一个节点上,Receiver要不断的接收数据,需要消耗很多资源,就会导致这个节点负载特别大。将每个Receiver都作为一个job来运行就会最大可能的负载均衡,不过这样也有可能失败,失败之后不会重试job,而是从新schedule提交一个新的job来运行Receiver,并且不会在之前运行的executor上启动,只要sparkstreaming程序不停止,假如Receiver出故障就会不休止的进行重新echedule并启动,确保Receiver一定会启动还有很重要的一点是,当重新启动一个Receiver时,是用一个线程池在新的线程中启动的
valfuture = ssc.sparkContext.submitJob[Receiver[_], Unit, Unit](
receiverRDD,startReceiverFunc, Seq(0), (_, _) => Unit, ())
//We will keep restarting the receiver job until ReceiverTracker is stopped
future.onComplete{
caseSuccess(_) =>
if(!shouldStartReceiver) {
onReceiverJobFinish(receiverId)
}else {
logInfo(s"RestartingReceiver $receiverId")
self.send(RestartReceiver(receiver))
}
caseFailure(e) =>
if(!shouldStartReceiver) {
onReceiverJobFinish(receiverId)
}else {
logError("Receiverhas been stopped. Try to restart it.", e)
logInfo(s"RestartingReceiver $receiverId")
self.send(RestartReceiver(receiver))
}
}(submitJobThreadPool)
logInfo(s"Receiver${receiver.streamId} started")
}
ReceiverSupervisorImpl负责处理Receiver接收到的数据,处理之后汇报给ReceiverTracker,所以ReceiverSupervisorImpl内部有和ReceiverTracker进行通信的endpoint。这个负责向ReceiverTracker发送消息。
private valtrackerEndpoint= RpcUtils.makeDriverRef("ReceiverTracker", env.conf,env.rpcEnv)
这个负责接收ReceiverTracker发送的消息,CleanupOldBlocks是用来清除运行完的每个batch的Blocks,UpdateRateLimit是用来随时调整限流(限流其实是限的数据存储的速度)。
/** RpcEndpointRef forreceiving messages from the ReceiverTracker in the driver */
private valendpoint=env.rpcEnv.setupEndpoint(
"Receiver-"+streamId+"-"+ System.currentTimeMillis(),newThreadSafeRpcEndpoint {
override valrpcEnv: RpcEnv =env.rpcEnv
override defreceive:PartialFunction[Any, Unit] = {
caseStopReceiver=>
logInfo("Receivedstop signal")
ReceiverSupervisorImpl.this.stop("Stoppedby driver", None)
caseCleanupOldBlocks(threshTime)=>
logDebug("Receiveddelete old batch signal")
cleanupOldBlocks(threshTime)
caseUpdateRateLimit(eps) =>
logInfo(s"Receiveda new rate limit:$eps.")
registeredBlockGenerators.foreach {bg =>
bg.updateRate(eps)
}
}
})
下面是ReceiverSupervisor的start方法
def start() {
onStart()
startReceiver()
}
在onStart中启动的是BlockGenerator,BlockGenerator是把接收到的一条一条的数据生成block存储起来,一个BlockGenerator只服务于一个Receiver。所以BlockGenerator要在Receiver启动之前启动。
override protected defonStart() {
registeredBlockGenerators.foreach { _.start() }
}
BlockGenerator种有一个定时器。这个定时器每隔一定(默认是200ms,和设定的batchduration无关)的时间就执行如下方法。这个方法就是把接收到的数据一条一条的放入到这个buffer缓存中,再把这个buffer按照一定的时间或者尺寸合并成block。除了定时器以外还有一条线程不停的把生成的block交给blockmanager存储起来。
/** Change the buffer towhich single records are added to. */
private defupdateCurrentBuffer(time: Long): Unit = {
try{
varnewBlock: Block =null
synchronized{
if(currentBuffer.nonEmpty){
valnewBlockBuffer=currentBuffer
currentBuffer=newArrayBuffer[Any]
valblockId =StreamBlockId(receiverId, time - blockIntervalMs)
listener.onGenerateBlock(blockId)
newBlock = newBlock(blockId,newBlockBuffer)
}
}
if(newBlock !=null) {
blocksForPushing.put(newBlock) // put is blocking when queue is full
}
} catch{
caseie:InterruptedException=>
logInfo("Block updatingtimer thread was interrupted")
casee:Exception=>
reportError("Errorin block updating thread", e)
}
}
下面来看startReceiver方法
/** Start receiver */
defstartReceiver(): Unit = synchronized {
try{
if(onReceiverStart()) {
logInfo("Startingreceiver")
receiverState=Started
receiver.onStart()
logInfo("Called receiveronStart")
} else{
// The driver refused us
stop("Registeredunsuccessfully because Driver refused to start receiver " +streamId, None)
}
} catch{
caseNonFatal(t) =>
stop("Error startingreceiver "+streamId,Some(t))
}
}
在启动Receiver之前还要向ReceiverTracker请求是否可以启动Receiver。当返回是true才会启动。ReceiverTracker接收到汇报的信息就把注册Receiver的信息。
override protected defonReceiverStart():Boolean = {
valmsg =RegisterReceiver(
streamId,receiver.getClass.getSimpleName,host,executorId,endpoint)
trackerEndpoint.askWithRetry[Boolean](msg)
}
Receiver的启动只是调用receiver.onStart(),Receiver就在work节点上运行了。
以SocketReceiver为例我看看它的onStart方法
private[streaming]
classSocketReceiver[T: ClassTag](
host: String,
port: Int,
bytesToObjects: InputStream => Iterator[T],
storageLevel: StorageLevel
) extendsReceiver[T](storageLevel)withLogging{
defonStart(){
// Start the thread that receives data over aconnection
newThread("SocketReceiver") {
setDaemon(true)
override defrun() { receive() }
}.start()
}
defonStop(){
// There is nothing much to do as the threadcalling receive()
// is designed to stop by itselfisStopped() returns false
}
/** Create a socket connection and receive data untilreceiver is stopped */
defreceive() {
varsocket: Socket =null
try{
logInfo("Connecting to"+ host +":"+ port)
socket = newSocket(host, port)
logInfo("Connected to"+ host +":"+ port)
valiterator =bytesToObjects(socket.getInputStream())
while(!isStopped && iterator.hasNext) {
store(iterator.next)
}
if(!isStopped()) {
restart("Socketdata stream had no more data")
} else{
logInfo("Stoppedreceiving")
}
} catch{
casee:Java.net.ConnectException =>
restart("Errorconnecting to "+host +":"+ port, e)
caseNonFatal(e) =>
logWarning("Errorreceiving data",e)
restart("Errorreceiving data",e)
} finally{
if(socket !=null) {
socket.close()
logInfo("Closedsocket to "+host +":"+ port)
}
}
}
}
作者:大数据技术研发人员:谢彪
-
资料来源于:DT_大数据梦工厂(Spark发行版本定制)
-
DT大数据梦工厂微信公众号:DT_Spark
-
新浪微博:http://www.weibo.com/ilovepains
-
王家林老师每晚20:00免费大数据实战
YY直播:68917580