hadoop 2.6 源码 解读之 DFSClient方法调用的RPC流程

DFSClient 是应用程序访问hdfs的主要入口类
其方法调用最终通过RPC通信触发服务端响应请求。
以 rename方法为例，介绍整个流程

 public void rename(String src, String dst, Options.Rename... options)
      throws IOException {
    checkOpen();
    try {
      namenode.rename2(src, dst, options);
    }
    ...
    }

最终调用namenode 的方法
##接下来 看下 namenode 是怎么构造出来的

先介绍两个协议

ClientProtocol协议

该协议定义了和名字节点交互的所有方法（如rename等方法）

@InterfaceAudience.Private
@InterfaceStability.Evolving
@KerberosInfo(
    serverPrincipal = DFSConfigKeys.DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY)
@TokenInfo(DelegationTokenSelector.class)
public interface ClientProtocol {
...
}

由此可见 ClientProtocol 其实就是一个 interface 定义

它有两个重要的子类

• ClientNamenodeProtocolTranslatorPB 用于 client 端，发送请求
• NameNodeRpcServer 用于server端，处理请求

###ClientNamenodeProtocolPB 协议

ClientNamenodeProtocol.proto 生成 ClientNamenodeProtocolProtos.java
ClientNamenodeProtocolProtos 定义了 ClientNamenodeProtocol.BlockingInterface 接口
ClientNamenodeProtocolPB 继承了 这个 BlockingInterface

public interface ClientNamenodeProtocolPB extends 
  ClientNamenodeProtocol.BlockingInterface {
}

ClientNamenodeProtocolPB 这个接口协议 用处 是将ClientProtocol 各种方法 进行 protobuf 序列化包装，以便RPC进行调用

DFSClient 构造函数中

this.namenode = proxyInfo.getProxy();

proxyInfo 的来历是

proxyInfo = NameNodeProxies.createProxy(conf, nameNodeUri,
          ClientProtocol.class, nnFallbackToSimpleAuth);

在非HA模式下 createProxy最终执行于

if (failoverProxyProvider == null) {
      // Non-HA case
      return createNonHAProxy(conf, NameNode.getAddress(nameNodeUri), xface,
          UserGroupInformation.getCurrentUser(), true, fallbackToSimpleAuth);
    } 

createNonHAProxy执行于

T proxy;
    if (xface == ClientProtocol.class) {
      proxy = (T) createNNProxyWithClientProtocol(nnAddr, conf, ugi,
          withRetries, fallbackToSimpleAuth);
    }

createNNProxyWithClientProtocol 关键代码如下

private static ClientProtocol createNNProxyWithClientProtocol
{
...
ClientNamenodeProtocolPB proxy = RPC.getProtocolProxy(
        ClientNamenodeProtocolPB.class, version, address, ugi, conf,
        NetUtils.getDefaultSocketFactory(conf),
        org.apache.hadoop.ipc.Client.getTimeout(conf), defaultPolicy,
        fallbackToSimpleAuth).getProxy();

...
return new ClientNamenodeProtocolTranslatorPB(proxy);
...

}

由此可以看出 DFSClient namenode 指向的是ClientNamenodeProtocolTranslatorPB 对象

重点 看下 RPC.getProtocolProxy,方法中的关键代码

return getProtocolEngine(protocol, conf).getProxy(protocol, clientVersion,
        addr, ticket, conf, factory, rpcTimeout, connectionRetryPolicy,
        fallbackToSimpleAuth);

getProtocolEngine(protocol, conf) 获得是 ProtobufRpcEngine

ProtobufRpcEngine.getProxy 如下

final Invoker invoker = new Invoker(protocol, addr, ticket, conf, factory,
        rpcTimeout, connectionRetryPolicy, fallbackToSimpleAuth);
    return new ProtocolProxy<T>(protocol, (T) Proxy.newProxyInstance(
        protocol.getClassLoader(), new Class[]{protocol}, invoker), false);

返回的是 ClientNamenodeProtocolPB的代理对象，这个对象 封装在 ProtocolProxy对象中并返回。
从此 DFSClient方法调用 都转换为 ClientNamenodeProtocolPB代理对象的方法调用

看看这个代理究竟做了什么事
这边使用的java动态代理机制，
关键逻辑 在Invoker 的覆写方法中 invoke() 中

public Object invoke(Object proxy, Method method, Object[] args)
{

val = (RpcResponseWrapper) client.call(RPC.RpcKind.RPC_PROTOCOL_BUFFER,
            new RpcRequestWrapper(rpcRequestHeader, theRequest), remoteId,
            fallbackToSimpleAuth);
}

最终调用connection.sendRpcRequest，将请求发出去

final Call call = createCall(rpcKind, rpcRequest);
    Connection connection = getConnection(remoteId, call, serviceClass,
      fallbackToSimpleAuth);
    try {
      connection.sendRpcRequest(call);                 // send the rpc request
    }        

sendRpcRequest的实现如下，通过线程池发出请求

synchronized (sendRpcRequestLock) {
        Future<?> senderFuture = sendParamsExecutor.submit(new Runnable() {
          @Override
          public void run() {
            try {
              synchronized (Connection.this.out) {
                if (shouldCloseConnection.get()) {
                  return;
                }
                
                if (LOG.isDebugEnabled())
                  LOG.debug(getName() + " sending #" + call.id);
         
                byte[] data = d.getData();
                int totalLength = d.getLength();
                out.writeInt(totalLength); // Total Length
                out.write(data, 0, totalLength);// RpcRequestHeader + RpcRequest
                out.flush();
              }
            } catch (IOException e) {
              // exception at this point would leave the connection in an
              // unrecoverable state (eg half a call left on the wire).
              // So, close the connection, killing any outstanding calls
              markClosed(e);
            } finally {
              //the buffer is just an in-memory buffer, but it is still polite to
              // close early
              IOUtils.closeStream(d);
            }
          }
        });
      
        try {
          senderFuture.get();
        } catch (ExecutionException e) {
          Throwable cause = e.getCause();
          
          // cause should only be a RuntimeException as the Runnable above
          // catches IOException
          if (cause instanceof RuntimeException) {
            throw (RuntimeException) cause;
          } else {
            throw new RuntimeException("unexpected checked exception", cause);
          }
        }
      }

rename 分析参考

参考链接：
https://psiitoy.github.io/2018/02/01/[%E6%BA%90%E7%A0%81]Hadoop%20HDFS%E6%80%BB%E7%BB%93(4)%20%E6%95%B0%E6%8D%AE%E8%AF%BB%E5%86%99%E6%B5%81%E7%A8%8B%E5%88%86%E6%9E%90/

注意事项：
S3 Rename 操作，因为它是通过 Copy + Delete 实现的