Spark Streaming自定义Receiver

一 背景

Spark社区为Spark Streaming提供了很多数据源接口，但是有些比较偏的数据源没有覆盖，由于公司技术栈选择，用了阿里云的MQ服务ONS，要做实时需求，要自己编写Receiver

二 技术实现

1.官网的例子已经比较详细，但是进入实践还需要慢慢调试，官方文档。

2.实现代码，由三部分组成，receiver，inputstream，util

3.receiver代码

import java.io.Serializable
import java.util.Properties

import com.aliyun.openservices.ons.api._
import com.aliyun.openservices.ons.api.impl.ONSFactoryImpl
import org.apache.spark.internal.Logging
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.receiver.Receiver

class OnsReceiver(
    cid: String,
    accessKey: String,
    secretKey: String,
    addr: String,
    topic: String,
    tag: String,
    func: Message => Array[Byte])
  extends Receiver[Array[Byte]](StorageLevel.MEMORY_AND_DISK_2) with Serializable with Logging {
  receiver =>

  private var consumer: Consumer = null
  private var workerThread: Thread = null

  override def onStart(): Unit = {
    workerThread = new Thread(new Runnable {
      override def run(): Unit =