基于supermap实现对接udp协议的streaming流服务

版权声明:本文为博主原创文章,欢迎转载,SuperMap技术问答社区 http://ask.supermap.com https://blog.csdn.net/supermapsupport/article/details/80165726
    在实时流数据处理方面,supermap iobjects for spark产品的streaming模块已实现了对socket,http,kafka,jms等
来源的数据处理,但依然无法满足每一种业务的需求,本篇介绍如何对接udp服务来源的数据流,以实时接收北大导航返回
的gps点数据为示例进行处理及输出。

1. 数据介绍

每个gps点坐标是以二进制数据包返回,数据体包含了经纬度,速度,方向,高程等信息。gps点如下:
aaaacccc0002000000353138383535343036383432000000000000000000a7e7dd58502c5d40575a46ea3d5d40404a0073000000ef97e207021c12010e0700

2. 解析数据

根据数据包的结构进行解析,得到字符串格式的点坐标信息,其主要代码如下:
        public String doParse() 
        {
            if (buffer == null) {
                return "";
            }
            //设备编号
            byte[] code = new byte[20];
            buffer.position(10);
            buffer.get(code);
            String codeStr = new String(code).trim();
            byte[] lou = new byte[8];
            buffer.get(lou);
            //经度
            double longitude = bytes2Double(lou);
            byte[] lat = new byte[8];
            buffer.get(lat);
            //纬度
            double latitude = bytes2Double(lat);
            byte[] speed = new byte[2];
            buffer.get(speed);
            //速度
            double catSpeed = byteToShort(speed);
            byte[] direction = new byte[2];
            buffer.get(direction);
            //方向
            short carDirec = byteToShort(direction);
            byte[] altitude = new byte[2];
            buffer.get(altitude);
            //高程
            short carAltitude = byteToShort(altitude);
            byte[] precision = new byte[2];
            buffer.get(precision);
            //精度
            short dataPrecision = byteToShort(precision);
            byte[] year = new byte[2];
            buffer.get(year);
            //时间
            Calendar calendar = Calendar.getInstance();
            calendar.set(byteToShort(year), buffer.get(), buffer.get(), buffer.get(), buffer.get(), buffer.get());
            Date date = calendar.getTime();
            SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
            String dateStr = df.format(date);
            return codeStr + "," + dateStr + "," + longitude + "," + latitude + "," + carAltitude + "," + carDirec + "," + catSpeed + "," + dataPrecision;

        }

3. 数据发送端

数据发送端采用udp协议进行数据的实时发送,udp服务无需连接,其主要代码,如下:
     InetAddress address = InetAddress.getByName("localhost");
                String ip = address.getHostAddress();
                System.out.println(ip);
                DatagramSocket ds = new DatagramSocket();
                while (true) 
                {
                    //发送给本机的地址,端口为5467
                    DatagramPacket dp = new DatagramPacket(str.getBytes(), 0, str.length(), address, 5467);
                    ds.send(dp);
                    Thread.sleep(1000);
                    System.out.println("发送完成");
                }

4. 数据接收器

基于超图streaming模块里的ReceiverBase接口实现UDP服务数据流的接收器,代码清单如下:
    class CustomUDPReceiver(val port: Int) extends ReceiverBase with Serializable {
      override def receive(ssc: StreamingContext): DStream[String] = {
        ssc.receiverStream(new SocketUDPStreamingReceiver(port))
      }
    }
    // 使用Memory_ONLY来提升性能
    class SocketUDPStreamingReceiver(port: Int) extends Receiver[String](StorageLevel.MEMORY_ONLY) {
      def onStart(): Unit = {
        val start = new Runnable() {
          def run(): Unit = {
            val executorService = Executors.newCachedThreadPool()
            val catchAndSend = new Runnable() {
              def run(): Unit = {
                val ds = new DatagramSocket(port)
                val length = 10000
                val buf = new Array[Byte](length)
                val dp = new DatagramPacket(buf, length)
                ds.receive(dp)
                val message = new String(buf, 0, dp.getLength);
                message.split("\n").foreach(line => store(line))
                ds.close()
                restart("")
              }
            }
            executorService.submit(catchAndSend)
          }
        }
        new Thread(start).start()
      }
      def onStop(): Unit = 
      {
      }
    }

5. 构建流模型文件

构建流模型文件的接收器节点,数据处理节点,过滤节点,输出节点等信息,以下为接收器节点代码:
    "stream": {
                         "nodeDic": {
                           "CustomUDPReceiver": {
                             "port": 5467,
                             "reader": {
                               "separator": ",",
                               "className": "com.supermap.bdt.streaming.formatter.CSVFormatter"
                             },
                             "metadata": {
                               "title": "",
                               "epsg": 4326,
                               "fieldInfos": [
                                 {
                                   "name": "id",
                                   "source": "0",
                                   "nType": "TEXT"
                                 },
                                 {
                                   "name": "datatime",
                                   "source": "1",
                                   "nType": "DATETIME"
                                 },
                                 {
                                   "name": "X",
                                   "source": "2",
                                   "nType": "DOUBLE"
                                 },
                                 {
                                   "name": "Y",
                                   "source": "3",
                                   "nType": "DOUBLE"
                                 },
                                 {
                                   "name": "carAltitude",
                                   "source": "4",
                                   "nType": "SINGLE"
                                 },
                                 {
                                   "name": "carDirection",
                                   "source": "5",
                                   "nType": "SINGLE"
                                 },
                                 {
                                   "name": "carSpeed",
                                   "source": "6",
                                   "nType": "SINGLE"
                                 },
                                 {
                                   "name": "carPrecision",
                                   "source": "7",
                                   "nType": "SINGLE"
                                 }
                               ],
                               "featureType": "POINT",
                               "idFieldName": "id",
                               "dateTimeFormat": "yyyy-MM-dd HH:mm:ss"
                             },
                             "name": "CustomUDPReceiver",
                             "caption": "",
                             "description": "",
                             "prevNodes": [],
                             "nextNodes": [
                               "esAppendSender"
                             ],
                             "className": "CustomUDPReceiver"
                           }

6. 启动

启动实时流程序前,先启动udp服务的发送端,其主要代码如下:
    val startupRun = Startup.fromJson(json)
    StreamingRunner.run(startupRun.asInstanceOf[StartupDefault])

7. 结果如图

udp
完整代码地址

阅读更多

扫码向博主提问

SuperMap技术控

非学,无以致疑;非问,无以广识
去开通我的Chat快问
想对作者说点什么?

博主推荐

换一批

没有更多推荐了,返回首页