hadoop RPC框架源码分析

gezooo

于 2022-04-02 09:56:31 发布

阅读量259

点赞数

文章标签： hadoop 源码 rpc

本文链接：https://blog.csdn.net/gezooo/article/details/123913588

版权

参考： Hadoop RPC框架_thomas0yang的博客-CSDN博客_hadoop rpc

Hadoop RPC分析

SERVER：

LISTENER：打开客户端的请求，selector OP_ACCEPT,
ServerSocketChannel server = (ServerSocketChannel) key.channel();
Reader reader = getReader();
Connection c = connectionManager.register(channel);
key.attach(c); // so closeCurrentConnection can get the object
reader.addConnection(c);

READER 数组，维护了一个pendingConnecitons列表，定期的监听pendingConnecitons的 selector OP_READ, 读取输入的请求

int size = pendingConnections.size();
for (int i=size; i>0; i--) {
Connection conn = pendingConnections.take();
conn.channel.register(readSelector, SelectionKey.OP_READ, conn);
}
readSelector.select();


processRpcRequest:

rpcRequest = ReflectionUtils.newInstance(rpcRequestClass, conf);
rpcRequest.readFields(dis);

产生一个CALL
Call call = new Call(header.getCallId(), header.getRetryCount(),
rpcRequest, this, ProtoUtil.convert(header.getRpcKind()),
header.getClientId().toByteArray(), traceScope, callerContext);

加入callQueue
callQueue.put(call);


HANDLER

final Call call = callQueue.take(); // pop the queue; maybe blocked here
执行SERVER端的调用，获取call的结果
value = call(call.rpcKind, call.connection.protocolName, call.rpcRequest,
call.timestamp);
注： call的实现是有RpcEngine实现的， Hadoop的RcpEngine有两种，WritableRpcEngine, 一个是ProtobufRpcEngine
setupResponse(buf, call, returnStatus, detailedErr,
value, errorClass, error); //设置Response,把要返回的数据准备好一个response
call.sendResponse();
-->call.connection.sendResponse
--> responder.doRespond(call);

RESPONDER
先直接写一部分数据，如果一次没写完，在responsder 注册 OP_WRITE
channel.register(writeSelector, SelectionKey.OP_WRITE, call);
后面异步线程慢慢写
doAsyncWrite(key);

客户端

CLIENT

call方法的摘要
final Call call = createCall(rpcKind, rpcRequest);
Connection connection = getConnection(remoteId, call, serviceClass,fallbackToSimpleAuth);
--> connection.setupIOstreams(fallbackToSimpleAuth);
setupConnection();
this.socket = socketFactory.createSocket();
SocketChannel ch = socket.getChannel();
int ret = selector.select((SelectableChannel)channel,
SelectionKey.OP_CONNECT, timeoutLeft);
InputStream inStream = NetUtils.getInputStream(socket);
OutputStream outStream = NetUtils.getOutputStream(socket);
writeConnectionHeader(outStream);


connection.sendRpcRequest(call);
//准备写入的数据，buffer
final DataOutputBuffer d = new DataOutputBuffer();
RpcRequestHeaderProto header = ProtoUtil.makeRpcRequestHeader(
call.rpcKind, OperationProto.RPC_FINAL_PACKET, call.id, call.retry,
clientId);
header.writeDelimitedTo(d);
call.rpcRequest.write(d);
异步并发的写入 connection的IOSTREAM，这里用并发提高处理请求的速度
Future<?> senderFuture = sendParamsExecutor.submit(new Runnable(){....})
byte[] data = d.getData();
int totalLength = d.getLength();
out.writeInt(totalLength); // Total Length
out.write(data, 0, totalLength);// RpcRequestHeader + RpcRequest
out.flush();
senderFuture.get(); 等待写入完成，

等待返回结果
synchronized (call) {
while (!call.done) {
try {
call.wait(); // wait for the result
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
throw new InterruptedIOException("Call interrupted");
}
}
没有报错就返回：
return call.getRpcResponse();


返回的response如何处理呢？

由Connection 异步的处理，一个线程，不停的监听

private class Connection extends Thread

Connection.run
           try {
           while (waitForWork()) {//wait here for work - read or close connection
           receiveRpcResponse();
           }

waitForWork 解读：
如果conneciton.calls 队列为空，并且等待超过timeout的时间，就会被关闭掉，return false
如果connection需要关闭，就会被关闭掉，return false
只有当calls不为空，并且shouldconnectionclose 标记为false，并且是running, 就继续 true

receiveRpcResponse的解读：

int totalLen = in.readInt(); //这一个调用会把线程阻塞，直到有数据返回，所以每一个conneciton只能同时处理一个call.

当返回的状态为SUCCESS之后，
就
Writable value = ReflectionUtils.newInstance(valueClass, conf);
               value.readFields(in); // read value
               calls.remove(callId);
               call.setRpcResponse(value);
callComplete();
this.done = true;
                                       notify(); // 这里处理完call, 就会notfiy wait的call.

根据ClientRMService 分析一下具体Protocal的实现原理

ClientRMService implemeng ApplicationClientProtocal

在serviceStart方法里会启动一个server
YarnRPC rpc = YarnRPC.create(conf);
this.server =
rpc.getServer(ApplicationClientProtocol.class, this,
clientBindAddress,
conf, this.rmDTSecretManager,
conf.getInt(YarnConfiguration.RM_CLIENT_THREAD_COUNT,
YarnConfiguration.DEFAULT_RM_CLIENT_THREAD_COUNT));


YarnRPC的实现类：org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
HadoopYarnProtoRPC.getServer
   RpcFactoryProvider.getServerFactory(conf).getServer(protocol,
   instance, addr, conf, secretManager, numHandlers, portRangeConfig);

   org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.gerServer
   获取
       Constructor<?> constructor = serviceCache.get(protocol);
       pbServiceImplClazz = localConf
.getClassByName(getPbServiceImplClassName(protocol));
String srcPackagePart = getPackageName(clazz);
                               String srcClassName = getClassName(clazz);
                               String destPackagePart = srcPackagePart + "." + impl.pb.service;
                               String destClassPart = srcClassName + PB_IMPL_CLASS_SUFFIX;
                               return destPackagePart + "." + PBServiceImpl;

               注：org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl
               这个类是一个包装类，并没有直接实现 ApplicationClientProtocal 接口，而是包装了一个实现了ApplicationClientProtocal接口的 ClientRMService，所有服务转发给clientRMService处理的
               这个类是RPC框架中的一个连接器类，连接了RPCServer接受到请求之后，转发给真实实现了协议接口的服务处理。
               这个类有固定的一些包名字和类名字，后面在RPC框架里找到协议实现的时候根据命名规范找到具体的协议连接器类

这个类实现的是 ApplicationClientProtocolPB 接口，
public class ApplicationClientProtocolPBServiceImpl implements ApplicationClientProtocolPB {
ApplicationClientProtocolPB 接口实现了protobuf生成的接口
public interface ApplicationClientProtocolPB extends ApplicationClientProtocolService.BlockingInterface {
ApplicationClientProtocolPBServiceImpl 这个类就是protobuf 接口和 ClientRMService之间的连接器

constructor = pbServiceImplClazz.getConstructor(protocol);
constructor.setAccessible(true);
serviceCache.putIfAbsent(protocol, constructor);


这里调用ApplicationClientProtocolPBServiceImpl 的构造器，创建出这里调用ApplicationClientProtocolPBServiceImpl 对象，把 ClientRMService 的一个实例作为参数传入
       Object service = null;
       try {
       service = constructor.newInstance(instance);
       }

       获取服务实现的接口，它实现了ApplicationClientProtocolPB 接口
       Class<?> pbProtocol = service.getClass().getInterfaces()[0];
       Method method = protoCache.get(protocol);

       if (method == null) {
   Class<?> protoClazz = null;
   try {
   protoClazz = localConf.getClassByName(getProtoClassName(protocol));
                       org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService
                       由下面的proto文件生成
                       hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/applicationclient_protocol.proto
                       在proto文件里定义了包路径以及生成的class的名称
                       option java_package = "org.apache.hadoop.yarn.proto";
                                       option java_outer_classname = "ApplicationClientProtocol";
   } catch (ClassNotFoundException e) {
   throw new YarnRuntimeException("Failed to load class: ["
   + getProtoClassName(protocol) + "]", e);
   }

   //在这个类里 org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService
   //会有一个方法： newReflectiveBlockingService
   try {
   method = protoClazz.getMethod("newReflectiveBlockingService",
   pbProtocol.getInterfaces()[0]);
   method.setAccessible(true);
   protoCache.putIfAbsent(protocol, method);
   } catch (NoSuchMethodException e) {
   throw new YarnRuntimeException(e);
   }
   }

   //newReflectiveBlockingService 给的一个参数就是是实现了协议的服务，里面会根据方法的index，调用方法
   //ProtobufRpcEngine 里的Server的call方法代理到blockingService 里的方法调用

   return createServer(pbProtocol, addr, conf, secretManager, numHandlers,
(BlockingService)method.invoke(null, service), portRangeConfig);
//设置engine为 ProtobufRpcEngine
RPC.setProtocolEngine(conf, pbProtocol, ProtobufRpcEngine.class);
           RPC.Server server = new RPC.Builder(conf).setProtocol(pbProtocol)
           .setInstance(blockingService).setBindAddress(addr.getHostName())
           .setPort(addr.getPort()).setNumHandlers(numHandlers).setVerbose(false)
           .setSecretManager(secretManager).setPortRangeConfig(portRangeConfig)
           .build();
           LOG.info("Adding protocol "+pbProtocol.getCanonicalName()+" to the server");
           server.addProtocol(RPC.RpcKind.RPC_PROTOCOL_BUFFER, pbProtocol, blockingService);
           return server;


   RPC.Server的实现
   @Override
       public Writable call(RPC.RpcKind rpcKind, String protocol,
       Writable rpcRequest, long receiveTime) throws Exception {
       return getRpcInvoker(rpcKind).call(this, protocol, rpcRequest,
       receiveTime);
       }

       public static RpcInvoker getRpcInvoker(RPC.RpcKind rpcKind) {
           RpcKindMapValue val = rpcKindMap.get(rpcKind);
           return (val == null) ? null : val.rpcInvoker;
           }

   //在registerProtocalEngine的时候，就把rpcKindMap 里的rpcInvoker设置了
   public static void registerProtocolEngine(RPC.RpcKind rpcKind,
   Class<? extends Writable> rpcRequestWrapperClass,
   RpcInvoker rpcInvoker) {
       RpcKindMapValue old =
       rpcKindMap.put(rpcKind, new RpcKindMapValue(rpcRequestWrapperClass, rpcInvoker));
       if (old != null) {
       rpcKindMap.put(rpcKind, old);
       throw new IllegalArgumentException("ReRegistration of rpcKind: " +
       rpcKind);
       }
       LOG.debug("rpcKind=" + rpcKind +
       ", rpcRequestWrapperClass=" + rpcRequestWrapperClass +
       ", rpcInvoker=" + rpcInvoker);
       }

       //在类ProtobufRpcEngine 初始化的时候，调用了registerProtocolEngine，注册好了
       public class ProtobufRpcEngine implements RpcEngine {
           public static final Log LOG = LogFactory.getLog(ProtobufRpcEngine.class);

           static { // Register the rpcRequest deserializer for WritableRpcEngine
           org.apache.hadoop.ipc.Server.registerProtocolEngine(
           RPC.RpcKind.RPC_PROTOCOL_BUFFER, RpcRequestWrapper.class,
           new Server.ProtoBufRpcInvoker());
           }

           ProtoBufRpcInvoker 是ProtobufRpcEngine 的内部类，实现了RpcInvoker
       static class ProtoBufRpcInvoker implements RpcInvoker

   最终服务端的call就是通过该类实现的，分析一下

   public Writable call(RPC.Server server, String protocol,
Writable writableRequest, long receiveTime) throws Exception {

//RpcRequestWrapper 是在registerProtocolEngine 的时候注册的
注：这里复习一下Server.Reader 线程异步读取Client发送过来的数据：
**************************************************************************************
一下代码引用自Reader里的processRpcRequest 方法里的片段，这里的rpcRequestClass 就是
Class<? extends Writable> rpcRequestClass = getRpcRequestWrapper(header.getRpcKind());
               if (rpcRequestClass != null)
                           return rpcRequestClass;
                           RpcKindMapValue val = rpcKindMap.get(ProtoUtil.convert(rpcKind));

Writable rpcRequest;
       try { //Read the rpc request
       rpcRequest = ReflectionUtils.newInstance(rpcRequestClass, conf);
       rpcRequest.readFields(dis);
       }
************************************************************************************


RpcRequestWrapper request = (RpcRequestWrapper) writableRequest;
RequestHeaderProto rpcRequest = request.requestHeader;
String methodName = rpcRequest.getMethodName();
String protoName = rpcRequest.getDeclaringClassProtocolName();
long clientVersion = rpcRequest.getClientProtocolVersion();
if (server.verbose)
LOG.info("Call: protocol=" + protocol + ", method=" + methodName);

************************************************************************************
创建ProtobufRpcEngine.Server的时候，注册RpcKind和协议的类以及实现
registerProtocolAndImpl(RPC.RpcKind.RPC_PROTOCOL_BUFFER, protocolClass,
protocolImpl);

String protocolName = RPC.getProtocolName(protocolClass);
ProtocolInfo anno = protocol.getAnnotation(ProtocolInfo.class);
                                           return (anno == null) ? protocol.getName() : anno.protocolName();
                    version = RPC.getProtocolVersion(protocolClass);
                                        ProtocolInfo anno = protocol.getAnnotation(ProtocolInfo.class);
                                           if (anno != null) {
                                           version = anno.protocolVersion();
                                           if (version != -1)
                                           return version;
                                           }
                                           try {
                                           Field versionField = protocol.getField("versionID");
                                           versionField.setAccessible(true);
                                           return versionField.getLong(protocol);
                                           }

getProtocolImplMap(rpcKind).put(new ProtoNameVer(protocolName, version), new ProtoClassProtoImpl(protocolClass, protocolImpl));

************************************************************************************

ProtoClassProtoImpl protocolImpl = getProtocolImpl(server, protoName,
clientVersion);

//protocolImpl就是创建Server的时候给的连接器类 ApplicationClientProtocolPBServiceImpl
BlockingService service = (BlockingService) protocolImpl.protocolImpl;
MethodDescriptor methodDescriptor = service.getDescriptorForType()
.findMethodByName(methodName);
if (methodDescriptor == null) {
String msg = "Unknown method " + methodName + " called on " + protocol
+ " protocol.";
LOG.warn(msg);
throw new RpcNoSuchMethodException(msg);
}
Message prototype = service.getRequestPrototype(methodDescriptor);
Message param = prototype.newBuilderForType()
.mergeFrom(request.theRequestRead).build();

Message result;
long startTime = Time.now();
int qTime = (int) (startTime - receiveTime);
Exception exception = null;
try {
server.rpcDetailedMetrics.init(protocolImpl.protocolClass);
result = service.callBlockingMethod(methodDescriptor, null, param);
} catch (ServiceException e) {
exception = (Exception) e.getCause();
throw (Exception) e.getCause();
} catch (Exception e) {
exception = e;
throw e;
} finally {
int processingTime = (int) (Time.now() - startTime);
if (LOG.isDebugEnabled()) {
String msg = "Served: " + methodName + " queueTime= " + qTime +
" procesingTime= " + processingTime;
if (exception != null) {
msg += " exception= " + exception.getClass().getSimpleName();
}
LOG.debug(msg);
}
String detailedMetricsName = (exception == null) ?
methodName :
exception.getClass().getSimpleName();
server.rpcMetrics.addRpcQueueTime(qTime);
server.rpcMetrics.addRpcProcessingTime(processingTime);
server.rpcDetailedMetrics.addProcessingTime(detailedMetricsName,
processingTime);
if (server.isLogSlowRPC()) {
server.logSlowRpcCalls(methodName, processingTime);
}
}
return new RpcResponseWrapper(result);
}


现在来看一下客户端调用的实现

YarnClient.createYarnClient();
YarnClientImpl.rmClient 是一个实现了ApplicationClientProtocol 接口的客户端代理
YarnClientImpl.serviceStart
   rmClient = ClientRMProxy.createRMProxy(getConfig(),ApplicationClientProtocol.class);
           T proxy = RMProxy.<T>getProxy(conf, protocol, rmAddress);
               YarnRPC.create(conf).getProxy(protocol, rmAddress, conf);
               org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getProxy()
                      RpcFactoryProvider.getClientFactory(conf).getClient(protocol, 1,addr, conf);
                      org.apache.hadoop.yarn.factories.impl.pb.RpcClientFactoryPBImpl.getClient(protocol, 1,addr, conf); //和上面的获取SERVER的对应org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.gerServer
                      org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl
                      //对应Server的实现：org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl

                      Class<?> pbClazz = null;
                       try {
                       pbClazz = localConf.getClassByName(getPBImplClassName(protocol));
                       //org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl
                       } catch (ClassNotFoundException e) {
                       throw new YarnRuntimeException("Failed to load class: ["
                       + getPBImplClassName(protocol) + "]", e);
                       }
                       try {
                       constructor = pbClazz.getConstructor(Long.TYPE, InetSocketAddress.class, Configuration.class);

                                               public ApplicationClientProtocolPBClientImpl(long clientVersion,
                                                           InetSocketAddress addr, Configuration conf) throws IOException {
                                                           RPC.setProtocolEngine(conf, ApplicationClientProtocolPB.class,
                                                           ProtobufRpcEngine.class);
                                                           proxy = RPC.getProxy(ApplicationClientProtocolPB.class, clientVersion, addr, conf);
                                                           }

                                                           RPC.getProxy()
                                                                   RPC.getProtocolProxy(protocol, clientVersion, addr, conf).getProxy();

                                                                   ProtobufRpcEngine.getProxy()
                                                                   @Override
                                                                   public <T> ProtocolProxy<T> getProxy(Class<T> protocol, long clientVersion,
                                                                   InetSocketAddress addr, UserGroupInformation ticket, Configuration conf,
                                                                   SocketFactory factory, int rpcTimeout, RetryPolicy connectionRetryPolicy
                                                                   ) throws IOException {
                                                                   return getProxy(protocol, clientVersion, addr, ticket, conf, factory,
                                                                   rpcTimeout, connectionRetryPolicy, null);
                                                                   }

                                                                   ProtobufRpcEngine.getProxy(...)
                                                                           final Invoker invoker = new Invoker(protocol, addr, ticket, conf, factory, rpcTimeout, connectionRetryPolicy, fallbackToSimpleAuth);
                                                                   new ProtocolProxy<T>(protocol, (T) Proxy.newProxyInstance(protocol.getClassLoader(), new Class[]{protocol}, invoker), false);

                                                           最终是由Java的代理类来实现的，最终实现调用是由Invoker处理的



                       constructor.setAccessible(true);
                       cache.putIfAbsent(protocol, constructor);
                       } catch (NoSuchMethodException e) {
                       throw new YarnRuntimeException("Could not find constructor with params: " + Long.TYPE + ", " + InetSocketAddress.class + ", " + Configuration.class, e);
                       }
                       }
                       try {
                       Object retObject = constructor.newInstance(clientVersion, addr, conf);
                       return retObject;
                       } catch (InvocationTargetException e) {
                       throw new YarnRuntimeException(e);
                       } catch (IllegalAccessException e) {
                       throw new YarnRuntimeException(e);
                       } catch (InstantiationException e) {
                       throw new YarnRuntimeException(e);
                       }



                  ProtobufRpcEngine.Invoker.invoke方法：

               @Override
                       public Object invoke(Object proxy, Method method, Object[] args)
                       throws ServiceException {
                       long startTime = 0;
                       if (LOG.isDebugEnabled()) {
                       startTime = Time.now();
                       }

                       if (args.length != 2) { // RpcController + Message
                       throw new ServiceException("Too many parameters for request. Method: ["
                       + method.getName() + "]" + ", Expected: 2, Actual: "
                       + args.length);
                       }
                       if (args[1] == null) {
                       throw new ServiceException("null param while calling Method: ["
                       + method.getName() + "]");
                       }

                       // if Tracing is on then start a new span for this rpc.
                       // guard it in the if statement to make sure there isn't
                       // any extra string manipulation.
                       Tracer tracer = Tracer.curThreadTracer();
                       TraceScope traceScope = null;
                       if (tracer != null) {
                       traceScope = tracer.newScope(RpcClientUtil.methodToTraceString(method));
                       }

                       RequestHeaderProto rpcRequestHeader = constructRpcRequestHeader(method);

                       if (LOG.isTraceEnabled()) {
                       LOG.trace(Thread.currentThread().getId() + ": Call -> " +
                       remoteId + ": " + method.getName() +
                       " {" + TextFormat.shortDebugString((Message) args[1]) + "}");
                       }


                       Message theRequest = (Message) args[1];
                       final RpcResponseWrapper val;
                       try {
                       //最终调用Client的call方法调用远程的服务，Client在上面已经分析过了，到此hadoop RPC分析完毕
                       val = (RpcResponseWrapper) client.call(RPC.RpcKind.RPC_PROTOCOL_BUFFER,
                       new RpcRequestWrapper(rpcRequestHeader, theRequest), remoteId,
                       fallbackToSimpleAuth);

                       } catch (Throwable e) {
                       if (LOG.isTraceEnabled()) {
                       LOG.trace(Thread.currentThread().getId() + ": Exception <- " +
                       remoteId + ": " + method.getName() +
                       " {" + e + "}");
                       }
                       if (traceScope != null) {
                       traceScope.addTimelineAnnotation("Call got exception: " +
                       e.getMessage());
                       }
                       throw new ServiceException(e);
                       } finally {
                       if (traceScope != null) traceScope.close();
                       }

                       if (LOG.isDebugEnabled()) {
                       long callTime = Time.now() - startTime;
                       LOG.debug("Call: " + method.getName() + " took " + callTime + "ms");
                       }

                       Message prototype = null;
                       try {
                       prototype = getReturnProtoType(method);
                       } catch (Exception e) {
                       throw new ServiceException(e);
                       }
                       Message returnMessage;
                       try {
                       returnMessage = prototype.newBuilderForType()
                       .mergeFrom(val.theResponseRead).build();

                       if (LOG.isTraceEnabled()) {
                       LOG.trace(Thread.currentThread().getId() + ": Response <- " +
                       remoteId + ": " + method.getName() +
                       " {" + TextFormat.shortDebugString(returnMessage) + "}");
                       }

                       } catch (Throwable e) {
                       throw new ServiceException(e);
                       }
                       return returnMessage;
                       }