步骤3 为RPC函数的参数和返回值提供Protocol Buffers定义。YARN需要保证每个RPC函数的参数和返回值是采用Protocol Buffers定义的,因此ResourceTracker协议中RegisterNodeManagerRequest、RegisterNodeManagerResponse、NodeHeartbeatRequest和NodeHeartbeatResponse四个参数或者返回值需要使用Protocol Buffers定义,具体如下(见yarn_server_common_service_protos.proto文件):
- import "yarn_protos.proto";
- import "yarn_server_common_protos.proto";
- message RegisterNodeManagerRequestProto {
- optional NodeIdProto node_id = 1;
- optional int32 http_port = 3;
- optional ResourceProto resource = 4;
- }
- message RegisterNodeManagerResponseProto {
- optional MasterKeyProto container_token_master_key = 1;
- optional MasterKeyProto nm_token_master_key = 2;
- optional NodeActionProto nodeAction = 3;
- optional int64 rm_identifier = 4;
- optional string diagnostics_message = 5;
- }
- ... //其他几个参数和返回值的定义
步骤4 为RPC函数的参数和返回值提供Java定义和封装。YARN采用了Protocol Buffers作为参数和返回值的序列化框架,且以原生态.proto文件的方式给出了定义,而具体的Java代码生成需在代码编写之后完成。基于以上考虑,为了更容易使用Protocol Buffers生成的(Java语言)参数和返回值定义,YARN RPC为每个RPC函数的参数和返回值提供Java定义和封装,以参数RegisterNodeManagerRequest为例进行说明。
Java接口定义如下(见Java包org.apache.hadoop.yarn.server.api.protocolrecords):
- public interface RegisterNodeManagerRequest {
- NodeId getNodeId();
- int getHttpPort();
- Resource getResource();
- void setNodeId(NodeId nodeId);
- void setHttpPort(int port);
- void setResource(Resource resource);
- }
Java封装如下(见Java包org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb):
- public class RegisterNodeManagerRequestPBImpl extends
- ProtoBase<RegisterNodeManagerRequestProto> implements RegisterNodeManagerRequest {
- RegisterNodeManagerRequestProto proto = RegisterNodeManagerRequestProto.getDefaultInstance();
- RegisterNodeManagerRequestProto.Builder builder = null;
- private NodeId nodeId = null;
- ...
- @Override
- public NodeId getNodeId() {
- RegisterNodeManagerRequestProtoOrBuilder p = viaProto ? proto : builder;
- if (this.nodeId != null) {
- return this.nodeId;
- }
- if (!p.hasNodeId()) {
- return null;
- }
- this.nodeId = convertFromProtoFormat(p.getNodeId());
- return this.nodeId;
- }
- @Override
- public void setNodeId(NodeId nodeId) {
- maybeInitBuilder();
- if (nodeId == null)
- builder.clearNodeId();
- this.nodeId = nodeId;
- }
- ...
- }
步骤5 为通信协议提供客户端和服务器端实现。客户端代码放在org.apache.hadoop.yarn.server.api.impl.pb.client包中,且类名为ResourceTrackerPBClientImpl,实现如下:
- public class ResourceTrackerPBClientImpl implements ResourceTracker, Closeable {
- private ResourceTrackerPB proxy;
- public ResourceTrackerPBClientImpl(long clientVersion, InetSocketAddress addr, Configuration conf) throws IOException {
- RPC.setProtocolEngine(conf, ResourceTrackerPB.class, ProtobufRpcEngine.class);
- proxy = (ResourceTrackerPB)RPC.getProxy(
- ResourceTrackerPB.class, clientVersion, addr, conf);
- }
- @Override
- public RegisterNodeManagerResponse registerNodeManager(
- RegisterNodeManagerRequest request) throws YarnException,
- IOException {
- RegisterNodeManagerRequestProto requestProto = ((RegisterNodeManagerRequestPBImpl)request).getProto();
- try {
- return new RegisterNodeManagerResponsePBImpl(proxy.registerNodeManager (null, requestProto));
- } catch (ServiceException e) {
- RPCUtil.unwrapAndThrowException(e);
- return null;
- }
- }
- ...
- }