Netty实现RPC框架

框架核心：

• 注册中心：使用zk，通过创建临时顺序节点保存服务的提供者和消费者；
• 服务提供者：使用自定义注解的方式暴露服务；
• 服务消费者：在properties文件中配置消费的服务；缓存服务提供者地址，并对提供者注册监听，当提供者变更时，更新本地缓存
• 服务治理平台：在提供者和消费者上注册监听，缓存服务的提供者和消费者。

一、原理介绍

参考dubbo的服务注册方式，使用zk作为注册中心，服务的生产者和消费者都在服务名下面创建临时顺序节点：

例如上图，

1）服务提供者：假设提供者现在要暴露ServiceB服务，那么提供者应用在启动时会在“/registry/xxx.xxx.ServiceB/providers”目录下创建一个临时顺序节点“/dataxxxxxxx”，并把自己的ip、服务端口号写进去，然后启动Netty服务端。

2）服务消费者：假设消费者现在要消费ServiceB服务，那么在消费者应用启动时，首先在“/registry/xxx.xxx.ServiceB/consumers”目录下创建临时顺序节点，并把自己的ip写进去。

3）服务调用（Netty）：服务消费者使用本地缓存保存提供者地址，不会主动去zk查询，只会在Watcher事件触发时更新本地缓存，当需要调用远程服务时，在这个本地缓存中找到服务提供者，然后启动Netty客户端，与服务提供者端的Netty进行通信，即通过Netty将服务名、方法名、参数发给提供者，提供者收到请求后，使用动态代理执行指定的方法，然后返回给消费者。

这其中用到序列化、编解码，序列化框架用的Protobuf，编解码用的Netty的ByteToMessageDecoder、MessageToByteEncoder。

服务调用过程时序图（图片来源：https://github.com/hu1991die/netty-rpc）：

 

二、代码实现

这里贴出主干代码，详细代码参考github（https://github.com/hfutlilong/rpc）

2.1 服务提供者

暴露服务：

/**
 * RPC注解
 */
@Target({ ElementType.TYPE })
@Retention(RetentionPolicy.RUNTIME)
@Component
public @interface RpcService {
}


public interface HelloService {
    String hello(String name);
}


@RpcService
public class HelloServiceImpl implements HelloService {
    @Override
    public String hello(String name) {
        return "Hello! " + name;
    }
}

扫描所有暴露的服务的接口名：

/* 获取所有带@RpcService注解的Spring Bean */
Map<String, Object> serviceBeanMap = applicationContext.getBeansWithAnnotation(RpcService.class);
if (MapUtils.isEmpty(serviceBeanMap)) {
    return;
}

for (Object serviceBean : serviceBeanMap.values()) {
    Class<?>[] interfaces = serviceBean.getClass().getInterfaces();
    for (Class<?> interfaceClass : interfaces) {
        String interfaceName = interfaceClass.getName();
        handlerMap.put(interfaceName, serviceBean);
    }
}

将服务接口注册到zk：

/**
 * 服务注册
 */
public class ServiceRegistry {

    /* 注册地址 */
    private String registryAddress;

    private CuratorFramework client;

    public ServiceRegistry(String registryAddress) {
        this.registryAddress = registryAddress;
    }

    public void registerProvider(Set<String> serviceNames, String serverAddress) throws Exception {
        client = ZkUtil.connectZkServer(registryAddress);

        for (String serviceName : serviceNames) {
            // 创建的临时节点示例：/registry/xxx.service/providers/data_0001
            String dataPath = Constant.ZkConstant.ZK_SEPERATOR + Constant.ZkConstant.SERVICE_ROOT_PATH
                    + Constant.ZkConstant.ZK_SEPERATOR + serviceName + Constant.ZkConstant.ZK_SEPERATOR
                    + Constant.ZkConstant.ZK_PROVIDERS_PATH + Constant.ZkConstant.ZK_SEPERATOR
                    + Constant.ZkConstant.ZK_PATH_PREFIX;
            /* 创建zk节点 */
            ZkUtil.createZkNode(client, dataPath, serverAddress);
        }

        ZkUtil.addJvmHook(client);
    }
}

ZkUtil工具类：

/**
 * @Description Zookeeper工具类
 * @Author lilong
 * @Date 2019-04-11 19:28
 */
public class ZkUtil {
    /**
     * 建立连接
     *
     * @param registryAddress
     * @return
     */
    public static CuratorFramework connectZkServer(String registryAddress) {
        CuratorFramework client = CuratorFrameworkFactory.newClient(registryAddress,
                new ExponentialBackoffRetry(Constant.ZkConstant.BASE_SLEEP_TIME_MS, Constant.ZkConstant.MAX_RETRIES));
        client.start();
        return client;
    }

    /**
     * 创建zk临时顺序节点
     *
     * @param path zk节点路径
     * @param data zk节点保存的内容
     * @throws Exception
     */
    public static void createZkNode(CuratorFramework client, String path, String data) throws Exception {
        if (StringUtils.isBlank(data)) {
            client.create().creatingParentContainersIfNeeded().withProtection()
                    .withMode(CreateMode.EPHEMERAL_SEQUENTIAL).forPath(path);
        } else {
            byte[] bytes = data.getBytes();
            client.create().creatingParentContainersIfNeeded().withProtection()
                    .withMode(CreateMode.EPHEMERAL_SEQUENTIAL).forPath(path, bytes);
        }
    }

    /**
     * 注册钩子
     */
    public static void addJvmHook(CuratorFramework client) {
        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            if (client != null) {
                client.close();
            }
        }));
    }
}

 

2.2 服务消费者

在配置文件中配置要消费的服务：

# 消费哪些服务，英文分号“;”隔开
consumer.services=com.netty.rpc.service.HelloService

监控提供者：

/**
 * 服务发现和监控
 */
public class ServiceDiscovery {
    private static final Logger LOGGER = LoggerFactory.getLogger(ServiceDiscovery.class);
    /* 保存服务提供者地址 */
    private volatile Map<String, List<String>> serviceProviderMap = new ConcurrentHashMap<>();
    /* 保存服务消费者地址 */
    private volatile Map<String, List<String>> serviceConsumerMap = new ConcurrentHashMap<>();
    /* 消费哪些服务 */
    private String consumerServices;
    private CuratorFramework client;

    public ServiceDiscovery(String registryAddress, String consumerServices) throws Exception {
        this.consumerServices = consumerServices;
        /* 连接zk服务 */
        client = ZkUtil.connectZkServer(registryAddress);
        /* 监听zk服务 */
        watchZkNode();
        ZkUtil.addJvmHook(client);
    }

    /**
     * 服务发现
     * 
     * @return
     */
    public String discovery(String service) {
        List<String> providers = serviceProviderMap.get(service);
        int size = providers.size();
        if (size == 0) {
            return null;
        } else if (size == 1) {
            return providers.get(0);
        } else {
            // 多个提供者，随机选择一个
            return providers.get(ThreadLocalRandom.current().nextInt(size));
        }
    }

    /**
     * 监听服务节点，并在服务下面注册消费者，并监听消费者
     */
    private void watchZkNode() throws Exception {
        String host = NetwokUtils.getLocalhost(); // 获取本机地址
        String[] serviceNames = consumerServices.split(";");
        for (int i = 0; i < serviceNames.length; i++) {
            String serviceName = serviceNames[i];
            registerConsumer(serviceName, host); // 在服务目录注册消费者
            watchProducerNode(serviceName); // 监听生产者
            watchConsumerNode(serviceName); // 监听消费者
        }
    }

    /**
     * 注册消费者
     */
    private void registerConsumer(String serviceName, String host) throws Exception {
        // 创建的临时节点示例：/registry/xxx.service/providers/data_0001
        String dataPath = Constant.ZkConstant.ZK_SEPERATOR + Constant.ZkConstant.SERVICE_ROOT_PATH
                + Constant.ZkConstant.ZK_SEPERATOR + serviceName + Constant.ZkConstant.ZK_SEPERATOR
                + Constant.ZkConstant.ZK_CONSUMERS_PATH + Constant.ZkConstant.ZK_SEPERATOR
                + Constant.ZkConstant.ZK_PATH_PREFIX;
        /* 创建zk节点 */
        ZkUtil.createZkNode(client, dataPath, host);
    }

    /**
     * 监听服务提供者
     * 
     * @param service
     * @throws Exception
     */
    private void watchProducerNode(String service) throws Exception {
        // /registry/a.b.c.Service/providers
        String providerPath = Constant.ZkConstant.ZK_SEPERATOR + Constant.ZkConstant.SERVICE_ROOT_PATH
                + Constant.ZkConstant.ZK_SEPERATOR + service + Constant.ZkConstant.ZK_SEPERATOR
                + Constant.ZkConstant.ZK_PROVIDERS_PATH;
        // 监听服务节点
        List<String> providers = getChildrenContent(service, providerPath, NodeTypeEnum.PROVIDER);
        serviceProviderMap.put(service, providers);
    }

    /**
     * 监听服务消费者
     * 
     * @param service
     * @throws Exception
     */
    private void watchConsumerNode(String service) throws Exception {
        // /registry/a.b.c.Service/consumers
        String consumerPath = Constant.ZkConstant.ZK_SEPERATOR + Constant.ZkConstant.SERVICE_ROOT_PATH
                + Constant.ZkConstant.ZK_SEPERATOR + service + Constant.ZkConstant.ZK_SEPERATOR
                + Constant.ZkConstant.ZK_CONSUMERS_PATH;
        // 监听消费者
        List<String> consumers = getChildrenContent(service, consumerPath, NodeTypeEnum.CONSUMER);
        serviceConsumerMap.put(service, consumers);
    }

    /**
     * 获取路径path下的所有子节点的内容
     * 
     * @param path
     * @return
     * @throws Exception
     */
    public List<String> getChildrenContent(String service, String path, NodeTypeEnum nodeType) throws Exception {
        List<String> children = client.getChildren().usingWatcher(new ZKWatcher(service, path, nodeType)).forPath(path);
        if (children == null || children.size() == 0) {
            LOGGER.debug("path {} has no children", path);
            return new ArrayList<>();
        }
        return getZkNodesContent(path, children);
    }

    /**
     * zookeeper监听节点数据变化
     */
    private class ZKWatcher implements CuratorWatcher {
        private String service;
        private String path;
        NodeTypeEnum nodeType;
        
        public ZKWatcher(String service, String path, NodeTypeEnum nodeType) {
            this.service = service;
            this.path = path;
            this.nodeType = nodeType;
        }

        public void process(WatchedEvent event) throws Exception {
            if (event.getType() == Event.EventType.NodeChildrenChanged) { // 监听子节点的变化
                List<String> children = client.getChildren().usingWatcher(new ZKWatcher(service, path, nodeType))
                        .forPath(path);
                switch (nodeType) {
                    case PROVIDER:
                        serviceProviderMap.put(service, getZkNodesContent(path, children));
                        break;
                    case CONSUMER:
                        serviceConsumerMap.put(service, getZkNodesContent(path, children));
                        break;
                    default:
                        // 节点类型要么是生产者、要么是消费者，不允许是其他类型
                        throw new RuntimeException("param nodeType error.");
                }
            }
        }
    }

    private enum NodeTypeEnum {
        PROVIDER, CONSUMER
    }

    /**
     * 获取zk节点的内容
     * 
     * @param path
     * @param children
     * @return
     * @throws Exception
     */
    private List<String> getZkNodesContent(String path, List<String> children) throws Exception {
        List<String> childrenContent = new ArrayList<>();
        if (children != null && children.size() > 0) {
            for (String child : children) {
                String childPath = path + Constant.ZkConstant.ZK_SEPERATOR + child;
                byte[] b = client.getData().forPath(childPath);
                String value = new String(b, StandardCharsets.UTF_8);
                if (StringUtils.isNotBlank(value)) {
                    childrenContent.add(value);
                }
            }
        }
        return childrenContent;
    }
}

2.3 服务治理平台

把上面的生产者和消费者组装起来：

package com.netty.rpc.entity;

import java.util.List;

/**
 * 服务治理详情
 */
public class SoaVO {
    /**
     * 服务
     */
    String service;

    /**
     * 提供者
     */
    List<String> providers;

    /**
     * 消费者
     */
    List<String> consumers;

    // getter/setter
}

汇总服务的生产者和消费者：

public class SoaService {
    private ServiceDiscovery serviceDiscovery;

    public SoaService(ServiceDiscovery serviceDiscovery) {
        this.serviceDiscovery = serviceDiscovery;
    }

    public Map<String, SoaVO> getAllServices() {
        Map<String, List<String>> providers = serviceDiscovery.getServiceProviderMap();
        Map<String, List<String>> consumers = serviceDiscovery.getServiceConsumerMap();

        Map<String, SoaVO> allServicesMap = new HashMap<>();
        for (Map.Entry<String, List<String>> entry : providers.entrySet()) {
            SoaVO soaVO = new SoaVO();
            soaVO.setService(entry.getKey());
            soaVO.setProviders(entry.getValue());
            allServicesMap.put(entry.getKey(), soaVO);
        }

        for (Map.Entry<String, List<String>> entry : consumers.entrySet()) {
            String service = entry.getKey();
            SoaVO soaVO = new SoaVO();
            if (allServicesMap.containsKey(service)) {
                soaVO = allServicesMap.get(service);
            }
            soaVO.setService(entry.getKey());
            soaVO.setConsumers(entry.getValue());
            allServicesMap.put(service, soaVO);
        }

        return allServicesMap;
    }
}

 

三、测试

/**
 * 消费者启动类
 */
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {"classpath:consumer/dubbo-consumer.xml"})
public class HelloServiceTest {
    @Resource
    private RpcProxy rpcProxy;

    @Resource
    private SoaService soaService;

    @Test
    public void helloTest(){
        HelloService helloService = rpcProxy.create(HelloService.class);
        String result = helloService.hello("world");
        System.out.println("============>result: " + result);
        Assert.assertEquals("Hello! world", result);

        Map<String, SoaVO> soaVOMap = soaService.getAllServices();
        System.out.println("####### SOA：" + JSON.toJSONString(soaVOMap));

        try {
            Thread.sleep(5 * 60 * 1000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

打印RPC调用结果和SOA服务治理情况：

============>result: Hello! world
####### SOA：{"com.netty.rpc.service.HelloService":
{"consumers":["192.168.74.1"],
"providers":["192.168.74.1:8888"],
"service":"com.netty.rpc.service.HelloService"}}

 

启动./zkCli.sh查看生产者和消费者：

生产者：

 

消费者：

 

四、其他

本文代码托管在github：
https://github.com/hfutlilong/rpc

 

参考内容：

https://github.com/hu1991die/netty-rpc

https://my.oschina.net/huangyong/blog/361751