Nacos源码系列之UDP（三）

一、UDP

User Datagram Protocol 用户数据协议，属于传输层的一种协议。无需建立连接就可以发送封装的 IP 数据包，不对传送数据包进行可靠性保证，即无法得知报文是否安全完整到达。

优势：
具有较好的实时性，工作效率较高；
网络开销小。

劣势：
无法保证可靠性、顺序性等

二、Java实现UDP

public class Send {

    public static void main(String[] args) {
        byte[] data="hello world".getBytes();
        try {
            DatagramPacket datagramPacket=new DatagramPacket(data,data.length,
                    InetAddress.getByName("127.0.0.1"),6000);
            DatagramSocket socket=new DatagramSocket();
            socket.send(datagramPacket);
            socket.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

public class Receive {
    public static void main(String[] args) throws IOException {
        DatagramSocket socket=new DatagramSocket(6000);
        byte[] data=new byte[1024];
        DatagramPacket datagramPacket=new DatagramPacket(data,data.length);
        socket.receive(datagramPacket);
        System.out.println(new String(data,0,datagramPacket.getLength())+" ip:"
                +datagramPacket.getAddress().getHostAddress()+" port:"+datagramPacket.getPort());
        socket.close();

    }
}

测试：

注意：先执行receive，然后等待send消息发送。

三、nacos中的主要UDP通信

在Nacos Naming中，使用http/udp的推送模型。由于UDP无ack，所以client要轮询查询保证数据一致性。

Nacos Naming的UDP通知可以分为以下几个主要流程。

1、Client上报ip+port（http）

Nacos Client在服务发现时，会通过NamingHttpClientProxy调用Nacos Naming的/instance/list接口，传入ip、udpPort。该接口有v1、v2版本。这里以v1为例。位置：com.alibaba.nacos.naming.controllers.InstanceController。

 //InstanceController
 
 @GetMapping("/list")
    @Secured(parser = NamingResourceParser.class, action = ActionTypes.READ)
    public Object list(HttpServletRequest request) throws Exception {
    ...
       String clientIP = WebUtils.optional(request, "clientIP", StringUtils.EMPTY);
       int udpPort = Integer.parseInt(WebUtils.optional(request, "udpPort", "0"));
    ...
    Subscriber subscriber = new Subscriber(clientIP + ":" + udpPort, agent, app, clientIP, namespaceId, serviceName,
                udpPort, clusters);
        return getInstanceOperator().listInstance(namespaceId, serviceName, subscriber, clusters, healthyOnly);
    
    }
    
    
  public ServiceInfo listInstance(String namespaceId, String serviceName, Subscriber subscriber, String cluster,
            boolean healthOnly) throws Exception {
            
   ...
   if (subscriber.getPort() > 0 && pushService.canEnablePush(subscriber.getAgent())) {
                subscriberServiceV1.addClient(namespaceId, serviceName, cluster, subscriber.getAgent(),
                        new InetSocketAddress(clientIP, subscriber.getPort()), pushDataSource, StringUtils.EMPTY,
                        StringUtils.EMPTY);
                cacheMillis = switchDomain.getPushCacheMillis(serviceName);
            }
   ...
}


    public void addClient(String namespaceId, String serviceName, String clusters, String agent,
            InetSocketAddress socketAddr, DataSource dataSource, String tenant, String app) {
        
        PushClient client = new PushClient(namespaceId, serviceName, clusters, agent, socketAddr, dataSource, tenant,
                app);
        addClient(client);
    }
    
    
    
 private final ConcurrentMap<String, ConcurrentMap<String, PushClient>> clientMap = new ConcurrentHashMap<>();

    
     public void addClient(PushClient client) {
        // client is stored by key 'serviceName' because notify event is driven by serviceName change
        String serviceKey = UtilsAndCommons.assembleFullServiceName(client.getNamespaceId(), client.getServiceName());
        ConcurrentMap<String, PushClient> clients = clientMap.get(serviceKey);
        if (clients == null) {
            clientMap.putIfAbsent(serviceKey, new ConcurrentHashMap<>(1024));
            clients = clientMap.get(serviceKey);
        }
        
        PushClient oldClient = clients.get(client.toString());
        if (oldClient != null) {
            oldClient.refresh();
        } else {
            PushClient res = clients.putIfAbsent(client.toString(), client);
            if (res != null) {
                Loggers.PUSH.warn("client: {} already associated with key {}", res.getAddrStr(), res);
            }
            Loggers.PUSH.debug("client: {} added for serviceName: {}", client.getAddrStr(), client.getServiceName());
        }
    }
   

2、Naming通知服务实例变化

事件发布：

当服务实例发生变化后，会添加task。
位置：com.alibaba.nacos.naming.consistency.ephemeral.distro.DistroConsistencyServiceImpl.Notifier。
notifier.addTask(key, DataOperation.CHANGE);

最终会进入到
com.alibaba.nacos.naming.core.Service#onChange

public void onChange(String key, Instances value) throws Exception {
 ...
     updateIPs(value.getInstanceList(), KeyBuilder.matchEphemeralInstanceListKey(key));
...
}

public void updateIPs(Collection<Instance> instances, boolean ephemeral) {
...

        getPushService().serviceChanged(this);
...
  }

重点来了！最后会来到UdpPushService的serviceChanged()。UdpPushService实现了ApplicationListener，Spring提供的发布订阅模型实现，最终调用publishEvent()将事件发布出去。

public class UdpPushService implements ApplicationContextAware, ApplicationListener<ServiceChangeEvent> {

    public void serviceChanged(Service service) {
        this.applicationContext.publishEvent(new ServiceChangeEvent(this, service));
    }
  }

事件订阅：

com.alibaba.nacos.naming.push.UdpPushService

 @Override
    public void onApplicationEvent(ServiceChangeEvent event) {
        // If upgrade to 2.0.X, do not push for v1.
        if (ApplicationUtils.getBean(UpgradeJudgement.class).isUseGrpcFeatures()) {
            return;
        }
        Service service = event.getService();
        String serviceName = service.getName();
        String namespaceId = service.getNamespaceId();
        //merge some change events to reduce the push frequency:
        if (futureMap.containsKey(UtilsAndCommons.assembleFullServiceName(namespaceId, serviceName))) {
            return;
        }
        Future future = GlobalExecutor.scheduleUdpSender(() -> {
            try {
                Loggers.PUSH.info(serviceName + " is changed, add it to push queue.");
                ConcurrentMap<String, PushClient> clients = subscriberServiceV1.getClientMap()
                        .get(UtilsAndCommons.assembleFullServiceName(namespaceId, serviceName));
                if (MapUtils.isEmpty(clients)) {
                    return;
                }
                
                Map<String, Object> cache = new HashMap<>(16);
                long lastRefTime = System.nanoTime();
                for (PushClient client : clients.values()) {
                    if (client.zombie()) {
                        Loggers.PUSH.debug("client is zombie: " + client);
                        clients.remove(client.toString());
                        Loggers.PUSH.debug("client is zombie: " + client);
                        continue;
                    }
                    
                    AckEntry ackEntry;
                    Loggers.PUSH.debug("push serviceName: {} to client: {}", serviceName, client);
                    String key = getPushCacheKey(serviceName, client.getIp(), client.getAgent());
                    byte[] compressData = null;
                    Map<String, Object> data = null;
                    if (switchDomain.getDefaultPushCacheMillis() >= 20000 && cache.containsKey(key)) {
                        org.javatuples.Pair pair = (org.javatuples.Pair) cache.get(key);
                        compressData = (byte[]) (pair.getValue0());
                        data = (Map<String, Object>) pair.getValue1();
                        
                        Loggers.PUSH.debug("[PUSH-CACHE] cache hit: {}:{}", serviceName, client.getAddrStr());
                    }
                    
                    if (compressData != null) {
                        ackEntry = prepareAckEntry(client, compressData, data, lastRefTime);
                    } else {
                        ackEntry = prepareAckEntry(client, prepareHostsData(client), lastRefTime);
                        if (ackEntry != null) {
                            cache.put(key,
                                    new org.javatuples.Pair<>(ackEntry.getOrigin().getData(), ackEntry.getData()));
                        }
                    }
                    
                    Loggers.PUSH.info("serviceName: {} changed, schedule push for: {}, agent: {}, key: {}",
                            client.getServiceName(), client.getAddrStr(), client.getAgent(),
                            (ackEntry == null ? null : ackEntry.getKey()));
                    
                    udpPush(ackEntry);
                }
            } catch (Exception e) {
                Loggers.PUSH.error("[NACOS-PUSH] failed to push serviceName: {} to client, error: {}", serviceName, e);
                
            } finally {
                futureMap.remove(UtilsAndCommons.assembleFullServiceName(namespaceId, serviceName));
            }
            
        }, 1000, TimeUnit.MILLISECONDS);
        
        futureMap.put(UtilsAndCommons.assembleFullServiceName(namespaceId, serviceName), future);
        
    }
    
     
     //push方法
    private static AckEntry udpPush(AckEntry ackEntry) {
        if (ackEntry == null) {
            Loggers.PUSH.error("[NACOS-PUSH] ackEntry is null.");
            return null;
        }
        
        if (ackEntry.getRetryTimes() > Constants.UDP_MAX_RETRY_TIMES) {
            Loggers.PUSH.warn("max re-push times reached, retry times {}, key: {}", ackEntry.getRetryTimes(),
                    ackEntry.getKey());
            ackMap.remove(ackEntry.getKey());
            udpSendTimeMap.remove(ackEntry.getKey());
            MetricsMonitor.incrementFailPush();
            return ackEntry;
        }
        
        try {
            if (!ackMap.containsKey(ackEntry.getKey())) {
                MetricsMonitor.incrementPush();
            }
            ackMap.put(ackEntry.getKey(), ackEntry);
            udpSendTimeMap.put(ackEntry.getKey(), System.currentTimeMillis());
            
            Loggers.PUSH.info("send udp packet: " + ackEntry.getKey());
            udpSocket.send(ackEntry.getOrigin());
            
            ackEntry.increaseRetryTime();
            
            GlobalExecutor.scheduleRetransmitter(new Retransmitter(ackEntry),
                    TimeUnit.NANOSECONDS.toMillis(Constants.ACK_TIMEOUT_NANOS), TimeUnit.MILLISECONDS);
            
            return ackEntry;
        } catch (Exception e) {
            Loggers.PUSH.error("[NACOS-PUSH] failed to push data: {} to client: {}, error: {}", ackEntry.getData(),
                    ackEntry.getOrigin().getAddress().getHostAddress(), e);
            ackMap.remove(ackEntry.getKey());
            udpSendTimeMap.remove(ackEntry.getKey());
            MetricsMonitor.incrementFailPush();
            
            return null;
        }
    }

3、Client接收服务实例变化，并发送ack

Client的接收可以从单元测试入手，位置：com.alibaba.nacos.client.naming.core.PushReceiverTest

@Test 
public void testTestRunWithDump() throws InterruptedException, IOException {
        ServiceInfoHolder holder = Mockito.mock(ServiceInfoHolder.class);
        final PushReceiver pushReceiver = new PushReceiver(holder);
        final ExecutorService executorService = Executors.newFixedThreadPool(1);
        executorService.submit(new Runnable() {
            @Override
            public void run() {
                pushReceiver.run();
            }
        });
        TimeUnit.MILLISECONDS.sleep(10);
        
        PushReceiver.PushPacket pack1 = new PushReceiver.PushPacket();
        pack1.type = "dump";
        pack1.data = "pack1";
        pack1.lastRefTime = 1;
        final String res1 = udpClientRun(pack1, pushReceiver);
        Assert.assertEquals("{\"type\": \"dump-ack\", \"lastRefTime\": \"1\", \"data\":\"{}\"}", res1);
        verify(holder, times(1)).getServiceInfoMap();
        
    }

private String udpClientRun(PushReceiver.PushPacket pack, PushReceiver pushReceiver) throws IOException {
    final int udpPort = pushReceiver.getUdpPort();
    String json = JacksonUtils.toJson(pack);
    final byte[] bytes = IoUtils.tryCompress(json, "UTF-8");
    final DatagramSocket datagramSocket = new DatagramSocket();
    datagramSocket.send(new DatagramPacket(bytes, bytes.length, InetAddress.getByName("localhost"), udpPort));
    byte[] buffer = new byte[20480];
    final DatagramPacket datagramPacket = new DatagramPacket(buffer, buffer.length);
    datagramSocket.receive(datagramPacket);
    final byte[] data = datagramPacket.getData();
    String res = new String(data, StandardCharsets.UTF_8);
    return res.trim();
}

上面涉及到一个核心处理类PushReceiver。

//init
   public PushReceiver(ServiceInfoHolder serviceInfoHolder) {
        try {
            this.serviceInfoHolder = serviceInfoHolder;
            String udpPort = getPushReceiverUdpPort();
            if (StringUtils.isEmpty(udpPort)) {
                this.udpSocket = new DatagramSocket();
            } else {
                //如果有udpPort，创建ip+port的socket
                this.udpSocket = new DatagramSocket(new InetSocketAddress(Integer.parseInt(udpPort)));
            }
            this.executorService = new ScheduledThreadPoolExecutor(1, new ThreadFactory() {
                @Override
                public Thread newThread(Runnable r) {
                    Thread thread = new Thread(r);
                    thread.setDaemon(true);
                    thread.setName("com.alibaba.nacos.naming.push.receiver");
                    return thread;
                }
            });
            
            this.executorService.execute(this);
        } catch (Exception e) {
            NAMING_LOGGER.error("[NA] init udp socket failed", e);
        }
    }

//run()
    @Override
    public void run() {
        while (!closed) {
            try {
                
                // byte[] is initialized with 0 full filled by default
                byte[] buffer = new byte[UDP_MSS];
                DatagramPacket packet = new DatagramPacket(buffer, buffer.length);
                
                //socket接收数据包
                udpSocket.receive(packet);
                
                String json = new String(IoUtils.tryDecompress(packet.getData()), UTF_8).trim();
                NAMING_LOGGER.info("received push data: " + json + " from " + packet.getAddress().toString());
                //转换成entity
                PushPacket pushPacket = JacksonUtils.toObj(json, PushPacket.class);
                String ack;
                //根据type组装ack消息
                if (PUSH_PACKAGE_TYPE_DOM.equals(pushPacket.type) || PUSH_PACKAGE_TYPE_SERVICE.equals(pushPacket.type)) {
                    serviceInfoHolder.processServiceInfo(pushPacket.data);
                    
                    // send ack to server
                    ack = "{\"type\": \"push-ack\"" + ", \"lastRefTime\":\"" + pushPacket.lastRefTime + "\", \"data\":"
                            + "\"\"}";
                } else if (PUSH_PACKAGE_TYPE_DUMP.equals(pushPacket.type)) {
                    // dump data to server
                    ack = "{\"type\": \"dump-ack\"" + ", \"lastRefTime\": \"" + pushPacket.lastRefTime + "\", \"data\":"
                            + "\"" + StringUtils.escapeJavaScript(JacksonUtils.toJson(serviceInfoHolder.getServiceInfoMap()))
                            + "\"}";
                } else {
                    // do nothing send ack only
                    ack = "{\"type\": \"unknown-ack\"" + ", \"lastRefTime\":\"" + pushPacket.lastRefTime
                            + "\", \"data\":" + "\"\"}";
                }
                
                //将ack发送出去
                udpSocket.send(new DatagramPacket(ack.getBytes(UTF_8), ack.getBytes(UTF_8).length,
                        packet.getSocketAddress()));
            } catch (Exception e) {
                if (closed) {
                    return;
                }
                NAMING_LOGGER.error("[NA] error while receiving push data", e);
            }
        }
    }

而在实际项目中，客户端通过NamingHttpClientProxy的构造方法，初始化PushReceiver。

   public NamingHttpClientProxy(String namespaceId, SecurityProxy securityProxy, ServerListManager serverListManager,
            Properties properties, ServiceInfoHolder serviceInfoHolder) {
        super(securityProxy, properties);
        this.serverListManager = serverListManager;
        this.setServerPort(DEFAULT_SERVER_PORT);
        this.namespaceId = namespaceId;
        this.beatReactor = new BeatReactor(this, properties);
        this.pushReceiver = new PushReceiver(serviceInfoHolder);
        this.maxRetry = ConvertUtils.toInt(properties.getProperty(PropertyKeyConst.NAMING_REQUEST_DOMAIN_RETRY_COUNT,
                String.valueOf(UtilAndComs.REQUEST_DOMAIN_RETRY_COUNT)));
    }

4、Naming接收ack

初始化了udpsocket和处理线程，在Receiver中接收udp数据包。

    static {
        try {
            udpSocket = new DatagramSocket();
            
            Receiver receiver = new Receiver();
            
            Thread inThread = new Thread(receiver);
            inThread.setDaemon(true);
            inThread.setName("com.alibaba.nacos.naming.push.receiver");
            inThread.start();
            
        } catch (SocketException e) {
            Loggers.SRV_LOG.error("[NACOS-PUSH] failed to init push service");
        }
    }

  public static class Receiver implements Runnable {
        
        @Override
        public void run() {
            while (true) {
                byte[] buffer = new byte[1024 * 64];
                DatagramPacket packet = new DatagramPacket(buffer, buffer.length);
                
                try {
                    udpSocket.receive(packet);
                    
                    String json = new String(packet.getData(), 0, packet.getLength(), StandardCharsets.UTF_8).trim();
                    AckPacket ackPacket = JacksonUtils.toObj(json, AckPacket.class);
                    
                    InetSocketAddress socketAddress = (InetSocketAddress) packet.getSocketAddress();
                    String ip = socketAddress.getAddress().getHostAddress();
                    int port = socketAddress.getPort();
                    
                    if (System.nanoTime() - ackPacket.lastRefTime > Constants.ACK_TIMEOUT_NANOS) {
                        Loggers.PUSH.warn("ack takes too long from {} ack json: {}", packet.getSocketAddress(), json);
                    }
                    
                    String ackKey = AckEntry.getAckKey(ip, port, ackPacket.lastRefTime);
                    AckEntry ackEntry = ackMap.remove(ackKey);
                    if (ackEntry == null) {
                        throw new IllegalStateException(
                                "unable to find ackEntry for key: " + ackKey + ", ack json: " + json);
                    }
                    
                    long pushCost = System.currentTimeMillis() - udpSendTimeMap.get(ackKey);
                    
                    Loggers.PUSH
                            .info("received ack: {} from: {}:{}, cost: {} ms, unacked: {}, total push: {}", json, ip,
                                    port, pushCost, ackMap.size(), MetricsMonitor.getTotalPushMonitor().get());
                    
                    MetricsMonitor.incrementPushCost(pushCost);
                    
                    udpSendTimeMap.remove(ackKey);
                    
                } catch (Throwable e) {
                    Loggers.PUSH.error("[NACOS-PUSH] error while receiving ack data", e);
                }
            }
        }
        
    }
//ack entity
public class AckEntry {
    
    public AckEntry(String key, DatagramPacket packet) {
        this.key = key;
        this.origin = packet;
    }
    
    private String key;
    //传递数据包
    private DatagramPacket origin;
    //重试次数
    private AtomicInteger retryTimes = new AtomicInteger(0);
    
    private Map<String, Object> data;
    
}

四、小结

• 对比TCP、UDP，各自有适合的场景。对不同的业务场景需要，选择响应的传输方式。
• nacos naming：UDP+补偿机制。UDP虽然没有ack机制，但可以在接收消息后再次发出send来实现消息确认。
• 解耦：发布-订阅模式，socket的send、receive。