这篇文章主要写 Hadoop RPC Client 的设计 与实现 . 在讲解的时候, 以 ProtobufRpcEngine为实例, 然后分步进行叙述.
一.Client端架构
Client类只有一个入口， 就是call()方法。 代理类会调用Client.call()方法将RPC请求发送到远程服务器， 然后等待远程服务器的响应。 如果远程服务器响应请求时出现异常， 则在call()方法中抛出异常。
在 call 方法中先将远程调用信息封装成一个 Client.Call 对象（保存了完成标志、返回信息、异常信息等），然后得到 connection 对象用于管理 Client 与 Server 的 Socket 连接。
getConnection 方法中通过 setupIOstreams 建立与 Server 的 socket 连接，启动 Connection 线程，监听 socket 读取 server 响应。
call() 方法发送 RCP 请求。
call() 方法调用 Call.wait() 在 Call 对象上等待 Server 响应信息。
Connection 线程收到响应信息设置 Call 对象返回信息字段，并调用 Call.notify() 唤醒 call() 方法线程读取 Call 对象返回值。
二.Client端创建流程
下面是创建Client端的代码.协议采用proto, 所以产生的RpcEngine是 ProtobufRpcEngine. 所以接下来的文章是以ProtobufRpcEngine为蓝本进行源码分析.
我只放了Server端, 详细的代码请查看:
Hadoop3.2.1 【 HDFS 】源码分析 : RPC原理 [六] ProtobufRpcEngine 使用
public static void main(String[] args) throws Exception {
//1. 构建配置对象
Configuration conf = new Configuration();
//2. 设置协议的RpcEngine为ProtobufRpcEngine .
RPC.setProtocolEngine(conf, Server.MetaInfoProtocol.class,
ProtobufRpcEngine.class);
//3. 拿到代理对象
Server.MetaInfoProtocol proxy = RPC.getProxy(Server.MetaInfoProtocol.class, 1L,
new InetSocketAddress("localhost", 7777), conf);
//4. 构建发送请求对象
CustomProtos.GetMetaInfoRequestProto obj = CustomProtos.GetMetaInfoRequestProto.newBuilder().setPath("/meta").build();
//5. 将请求对象传入, 获取响应信息
CustomProtos.GetMetaInfoResponseProto metaData = proxy.getMetaInfo(null, obj);
//6. 输出数据
System.out.println(metaData.getInfo());
}
上面的代码, 主要是分三部分.
1.构建配置对象&设置RpcEngine引擎
2.获取代理对象
3.设置请求参数&通过代理对象请求.
4.处理结果
第一条和第二条,我就不细说了. 这个很简单,就是使用proto定义一个协议, 绑定到RPC.Builder的实现对象里面.
我们直接看这段, 获取代理对象.
Server.MetaInfoProtocol proxy = RPC.getProxy(Server.MetaInfoProtocol.class, 1L,
new InetSocketAddress("localhost", 7777), conf);也是就是通过RPC.getProxy方法获取协议的代理对象.
/**
* Construct a client-side proxy object with the default SocketFactory
* @param <T>
*
* @param protocol 协议
* @param clientVersion 客户端的版本
* @param addr 请求地址
* @param conf 配置文件
* @return a proxy instance
* @throws IOException
*/
public static <T> T getProxy(Class<T> protocol,
long clientVersion,
InetSocketAddress addr, Configuration conf)
throws IOException {
return getProtocolProxy(protocol, clientVersion, addr, conf).getProxy();
}接续看,这里面就一句getProtocolProxy,加断点一直跟进
/**
* Get a protocol proxy that contains a proxy connection to a remote server
* and a set of methods that are supported by the server
*
* @param protocol protocol
* @param clientVersion client's version
* @param addr server address
* @param ticket security ticket
* @param conf configuration
* @param factory socket factory
* @param rpcTimeout max time for each rpc; 0 means no timeout
* @param connectionRetryPolicy retry policy
* @param fallbackToSimpleAuth set to true or false during calls to indicate if
* a secure client falls back to simple auth
* @return the proxy
* @throws IOException if any error occurs
*/
public static <T> ProtocolProxy<T> getProtocolProxy(Class<T> protocol,
long clientVersion,
InetSocketAddress addr,
UserGroupInformation ticket,
Configuration conf,
SocketFactory factory,
int rpcTimeout,
RetryPolicy connectionRetryPolicy,
AtomicBoolean fallbackToSimpleAuth)
throws IOException {
if (UserGroupInformation.isSecurityEnabled()) {
SaslRpcServer.init(conf);
}
return getProtocolEngine(protocol, conf).getProxy(protocol, clientVersion,
addr, ticket, conf, factory, rpcTimeout, connectionRetryPolicy,
fallbackToSimpleAuth, null);
}debug界面是这样的:
核心的是这句
return getProtocolEngine(protocol, conf).getProxy(protocol, clientVersion,
addr, ticket, conf, factory, rpcTimeout, connectionRetryPolicy,
fallbackToSimpleAuth, null);首先通过 getProtocolEngine 获取RPC Engine [ ProtobufRpcEngine] ,
// return the RpcEngine configured to handle a protocol
static synchronized RpcEngine getProtocolEngine(Class<?> protocol,
Configuration conf) {
//从缓存中获取RpcEngine ,
// 这个是提前设置的
// 通过 RPC.setProtocolEngine(conf, MetaInfoProtocol.class,ProtobufRpcEngine.class);
RpcEngine engine = PROTOCOL_ENGINES.get(protocol);
if (engine == null) {
//通过这里 获取RpcEngine的实现类 , 这里我们获取的是 ProtobufRpcEngine.class
Class<?> impl = conf.getClass(ENGINE_PROP+"."+protocol.getName(),
WritableRpcEngine.class);
// impl : org.apache.hadoop.ipc.ProtobufRpcEngine
engine = (RpcEngine)ReflectionUtils.newInstance(impl, conf);
PROTOCOL_ENGINES.put(protocol, engine);
}
return engine;
}
然后再调用 ProtobufRpcEngine的 getProxy方法.将协议,客户端的版本号. socket地址, ticket , 配置文件, socket 的创建工厂对象[ StandardSocketFactory ] , PRC 服务的超时时间, connetion的重试策略,以及权限等信息,传入.
@Override
@SuppressWarnings("unchecked")
public <T> ProtocolProxy<T> getProxy(Class<T> protocol, long clientVersion,
InetSocketAddress addr, UserGroupInformation ticket, Configuration conf,
SocketFactory factory, int rpcTimeout, RetryPolicy connectionRetryPolicy,
AtomicBoolean fallbackToSimpleAuth, AlignmentContext alignmentContext)
throws IOException {
//构造一个实现了InvocationHandler接口的invoker 对象
// (动态代理机制中的InvocationHandler对象会在invoke()方法中代理所有目标接口上的 调用，
// 用户可以在invoke()方法中添加代理操作)
final Invoker invoker = new Invoker(protocol, addr, ticket, conf, factory,
rpcTimeout, connectionRetryPolicy, fallbackToSimpleAuth,
alignmentContext);
//然后调用Proxy.newProxylnstance()获取动态代理对象，并通过ProtocolProxy返回
return new ProtocolProxy<T>(protocol, (T) Proxy.newProxyInstance(
protocol.getClassLoader(), new Class[]{protocol}, invoker), false);
}在getProxy 这个方法中. 主要是分两步,
1. 构造一个实现了InvocationHandler接口的invoker 对象 (动态代理机制中的InvocationHandler对象会在invoke()方法中代理所有目标接口上的 调用， 用户可以在invoke()方法中添加代理操作
2.调用Proxy.newProxylnstance()获取动态代理对象，并通过ProtocolProxy返回
我们先看Invoker的创建.
private Invoker(Class<?> protocol, InetSocketAddress addr,
UserGroupInformation ticket, Configuration conf, SocketFactory factory,
int rpcTimeout, RetryPolicy connectionRetryPolicy,
AtomicBoolean fallbackToSimpleAuth, AlignmentContext alignmentContext)
throws IOException {
this(protocol,
Client.ConnectionId.getConnectionId(addr, protocol, ticket, rpcTimeout, connectionRetryPolicy, conf), conf, factory);
this.fallbackToSimpleAuth = fallbackToSimpleAuth;
this.alignmentContext = alignmentContext;
}主要是:
this(protocol, Client.ConnectionId.getConnectionId(addr, protocol, ticket, rpcTimeout, connectionRetryPolicy, conf), conf, factory);我们先看
Client.ConnectionId.getConnectionId(addr, protocol, ticket, rpcTimeout, connectionRetryPolicy, conf), conf, factory)这里会调用getConnectionId方法 构建一个Client.ConnectionId对象.
ConnectionId : 这个类 持有 请求地址 和 用户的ticketclient 连接 server 的唯一凭证 : [remoteAddress, protocol, ticket]
/**
* Returns a ConnectionId object.
* @param addr Remote address for the connection.
* @param protocol Protocol for RPC.
* @param ticket UGI
* @param rpcTimeout timeout
* @param conf Configuration object
* @return A ConnectionId instance
* @throws IOException
*/
static ConnectionId getConnectionId(InetSocketAddress addr,
Class<?> protocol, UserGroupInformation ticket, int rpcTimeout,
RetryPolicy connectionRetryPolicy, Configuration conf) throws IOException {
//构建重试策略
if (connectionRetryPolicy == null) {
//设置最大重试次数 默认值: 10
final int max = conf.getInt(
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_MAX_RETRIES_KEY,
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_MAX_RETRIES_DEFAULT);
// 设置重试间隔: 1 秒
final int retryInterval = conf.getInt(
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_RETRY_INTERVAL_KEY,
CommonConfigurationKeysPublic
.IPC_CLIENT_CONNECT_RETRY_INTERVAL_DEFAULT);
//创建重试策略实例 RetryUpToMaximumCountWithFixedSleep
// 重试10次, 每次间隔1秒
connectionRetryPolicy = RetryPolicies.retryUpToMaximumCountWithFixedSleep(
max, retryInterval, TimeUnit.MILLISECONDS);
}
//创建ConnectionId :
// 这个类 持有 请求地址 和 用户的ticket
// client 连接 server 的唯一凭证 : [remoteAddress, protocol, ticket]
return new ConnectionId(addr, protocol, ticket, rpcTimeout,
connectionRetryPolicy, conf);
}
在getConnectionId这个方法里面会干两个事, 创一个重试策略 [RetryUpToMaximumCountWithFixedSleep]. 然后构建一个ConnectionId对象.
ConnectionId(InetSocketAddress address, Class<?> protocol,
UserGroupInformation ticket, int rpcTimeout,
RetryPolicy connectionRetryPolicy, Configuration conf) {
// 协议
this.protocol = protocol;
// 请求地址
this.address = address;
//用户 ticket
this.ticket = ticket;
//设置超时时间
this.rpcTimeout = rpcTimeout;
//设置重试策略 默认: 重试10次, 每次间隔1秒
this.connectionRetryPolicy = connectionRetryPolicy;
// 单位 10秒
this.maxIdleTime = conf.getInt(
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECTION_MAXIDLETIME_KEY,
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECTION_MAXIDLETIME_DEFAULT);
// sasl client最大重试次数 5 次
this.maxRetriesOnSasl = conf.getInt(
CommonConfigurationKeys.IPC_CLIENT_CONNECT_MAX_RETRIES_ON_SASL_KEY,
CommonConfigurationKeys.IPC_CLIENT_CONNECT_MAX_RETRIES_ON_SASL_DEFAULT);
//指示客户端将在套接字超时时进行重试的次数，以建立服务器连接。 默认值: 45
this.maxRetriesOnSocketTimeouts = conf.getInt(
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_MAX_RETRIES_ON_SOCKET_TIMEOUTS_KEY,
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_MAX_RETRIES_ON_SOCKET_TIMEOUTS_DEFAULT);
//使用TCP_NODELAY标志绕过Nagle的算法传输延迟。 默认值: true
this.tcpNoDelay = conf.getBoolean(
CommonConfigurationKeysPublic.IPC_CLIENT_TCPNODELAY_KEY,
CommonConfigurationKeysPublic.IPC_CLIENT_TCPNODELAY_DEFAULT);
// 从客户端启用低延迟连接 默认 false
this.tcpLowLatency = conf.getBoolean(
CommonConfigurationKeysPublic.IPC_CLIENT_LOW_LATENCY,
CommonConfigurationKeysPublic.IPC_CLIENT_LOW_LATENCY_DEFAULT
);
// 启用从RPC客户端到服务器的ping操作 默认值: true
this.doPing = conf.getBoolean(
CommonConfigurationKeys.IPC_CLIENT_PING_KEY,
CommonConfigurationKeys.IPC_CLIENT_PING_DEFAULT);
// 设置ping 操作的间隔, 默认值 : 1分钟
this.pingInterval = (doPing ? Client.getPingInterval(conf) : 0);
this.conf = conf;
}
回到之前的调用Invoker 另一个构造方法,但是入参会变.
/**
* This constructor takes a connectionId, instead of creating a new one.
*/
private Invoker(Class<?> protocol, Client.ConnectionId connId,
Configuration conf, SocketFactory factory) {
this.remoteId = connId;
this.client = CLIENTS.getClient(conf, factory, RpcWritable.Buffer.class);
this.protocolName = RPC.getProtocolName(protocol);
this.clientProtocolVersion = RPC
.getProtocolVersion(protocol);
}这个是Invoker真正的构建方法,这里面会将刚刚构建好的ConnectionId 赋值给remoteId 字段.
并且创建一个Client 对象.
// 获取/创建 客户端
this.client = CLIENTS.getClient(conf, factory, RpcWritable.Buffer.class);
我们看下getClient 方法. 这里面会先尝试从缓存中获取client对象, 如果没有的话,在自己创建一个,并且加到缓存中.
为什么会放到缓存中呢??
当client和server再次通讯的时候,可以复用这个client .
/**
* 如果没有缓存的client存在的话
* 根据用户提供的SocketFactory 构造 或者 缓存一个IPC 客户端
*
* Construct & cache an IPC client with the user-provided SocketFactory
* if no cached client exists.
*
* @param conf Configuration
* @param factory SocketFactory for client socket
* @param valueClass Class of the expected response
* @return an IPC client
*/
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class<? extends Writable> valueClass) {
// Construct & cache client.
//
// The configuration is only used for timeout,
// and Clients have connection pools. So we can either
// (a) lose some connection pooling and leak sockets, or
// (b) use the same timeout for all configurations.
//
// Since the IPC is usually intended globally, notper-job, we choose (a).
//从缓存中获取Client
Client client = clients.get(factory);
if (client == null) {
//client在缓存中不存在, 创建一个.
client = new Client(valueClass, conf, factory);
//缓存创建的client
clients.put(factory, client);
} else {
//client的引用计数+1
client.incCount();
}
if (Client.LOG.isDebugEnabled()) {
Client.LOG.debug("getting client out of cache: " + client);
}
// 返回client
return client;
}
到这里 Invoker 对象就创建完了.
回到 ProtobufRpcEngine 的getProxy 方法 .
//然后调用Proxy.newProxylnstance()获取动态代理对象，并通过ProtocolProxy返回
return new ProtocolProxy<T>(protocol, (T) Proxy.newProxyInstance(
protocol.getClassLoader(), new Class[]{protocol}, invoker), false);构建一个ProtocolProxy 对象返回
/**
* Constructor
*
* @param protocol protocol class
* @param proxy its proxy
* @param supportServerMethodCheck If false proxy will never fetch server
* methods and isMethodSupported will always return true. If true,
* server methods will be fetched for the first call to
* isMethodSupported.
*/
public ProtocolProxy(Class<T> protocol, T proxy,
boolean supportServerMethodCheck) {
this.protocol = protocol;
this.proxy = proxy;
this.supportServerMethodCheck = supportServerMethodCheck;
}
然后我们看Client端的第四步 , 根据proto协议,构建一个请求对象.
这个没啥可说的.是proto自动生成的,我们只是创建了一下而已.
//4. 构建发送请求对象
CustomProtos.GetMetaInfoRequestProto obj = CustomProtos.GetMetaInfoRequestProto.newBuilder().setPath("/meta").build();
然后就是Client端的第5步了将请求对象传入, 获取响应信息
//5. 将请求对象传入, 获取响应信息
CustomProtos.GetMetaInfoResponseProto metaData = proxy.getMetaInfo(null, obj);
Client端的最后一步输出响应信息.
//6. 输出数据
System.out.println(metaData.getInfo());------------------华丽的分割线-------------------------------------------------------------------
咦,到这里有点懵, 请求server端的代码呢??? 请求怎么发出去的??? 怎么拿到响应信息的呢????
嗯嗯,是动态代理. ProtobufRpcEngine 的getProxy 方法 .
return new ProtocolProxy<T>(protocol, (T) Proxy.newProxyInstance(
protocol.getClassLoader(), new Class[]{protocol}, invoker), false);主要是这个
(T) Proxy.newProxyInstance(
protocol.getClassLoader(), new Class[]{protocol}, invoker)public static Object newProxyInstance(ClassLoader loader,
Class<?>[] interfaces,
InvocationHandler h) {........................
}
这个方法的作用就是创建一个代理类对象，
它接收三个参数，我们来看下几个参数的含义：
loader ： 一个classloader对象，定义了由哪个classloader对象对生成的代理类进行加载
interfaces： 一个interface对象数组，表示我们将要给我们的代理对象提供一组什么样的接口，如果我们提供了这样一个接口对象数组，那么也就是声明了代理类实现了这些接口，代理类就可以调用接口中声明的所有方法。
h： 一个InvocationHandler对象，表示的是当动态代理对象调用方法的时候会关联到哪一个InvocationHandler对象上，并最终由其调用。
我们直接看 ProtobufRpcEngine#Invoker中的 invoke方法
/**
*
* ProtobufRpcEngine.Invoker.invoker() 方法主要做了三件事情:
* 1.构造请求头域，
* 使用protobuf将请求头序列化，这个请求头域 记录了当前RPC调用是什么接口的什么方法上的调用;
* 2.通过RPC.Client类发送请求头以 及序列化好的请求参数。
* 请求参数是在ClientNamenodeProtocolPB调用时就已经序列化好的，
* 调用Client.call()方法时，
* 需要将请求头以及请求参数使用一个RpcRequestWrapper对象封装;
* 3.获取响应信息，序列化响应信息并返回。
*
*
* This is the client side invoker of RPC method. It only throws
* ServiceException, since the invocation proxy expects only
* ServiceException to be thrown by the method in case protobuf service.
*
* ServiceException has the following causes:
* <ol>
* <li>Exceptions encountered on the client side in this method are
* set as cause in ServiceException as is.</li>
* <li>Exceptions from the server are wrapped in RemoteException and are
* set as cause in ServiceException</li>
* </ol>
*
* Note that the client calling protobuf RPC methods, must handle
* ServiceException by getting the cause from the ServiceException. If the
* cause is RemoteException, then unwrap it to get the exception thrown by
* the server.
*/
@Override
public Message invoke(Object proxy, final Method method, Object[] args)
throws ServiceException {
long startTime = 0;
if (LOG.isDebugEnabled()) {
startTime = Time.now();
}
// pb接口的参数只有两个，即RpcController + Message
if (args.length != 2) { // RpcController + Message
throw new ServiceException(
"Too many or few parameters for request. Method: ["
+ method.getName() + "]" + ", Expected: 2, Actual: "
+ args.length);
}
if (args[1] == null) {
throw new ServiceException("null param while calling Method: ["
+ method.getName() + "]");
}
// if Tracing is on then start a new span for this rpc.
// guard it in the if statement to make sure there isn't
// any extra string manipulation.
// todo 这个是啥
Tracer tracer = Tracer.curThreadTracer();
TraceScope traceScope = null;
if (tracer != null) {
traceScope = tracer.newScope(RpcClientUtil.methodToTraceString(method));
}
//构造请求头域，标明在什么接口上调用什么方法
RequestHeaderProto rpcRequestHeader = constructRpcRequestHeader(method);
if (LOG.isTraceEnabled()) {
LOG.trace(Thread.currentThread().getId() + ": Call -> " +
remoteId + ": " + method.getName() +
" {" + TextFormat.shortDebugString((Message) args[1]) + "}");
}
//获取请求调用的参数，例如RenameRequestProto
final Message theRequest = (Message) args[1];
final RpcWritable.Buffer val;
try {
//调用RPC.Client发送请求
val = (RpcWritable.Buffer) client.call(RPC.RpcKind.RPC_PROTOCOL_BUFFER,
new RpcProtobufRequest(rpcRequestHeader, theRequest), remoteId,
fallbackToSimpleAuth, alignmentContext);
} catch (Throwable e) {
if (LOG.isTraceEnabled()) {
LOG.trace(Thread.currentThread().getId() + ": Exception <- " +
remoteId + ": " + method.getName() +
" {" + e + "}");
}
if (traceScope != null) {
traceScope.addTimelineAnnotation("Call got exception: " +
e.toString());
}
throw new ServiceException(e);
} finally {
if (traceScope != null) traceScope.close();
}
if (LOG.isDebugEnabled()) {
long callTime = Time.now() - startTime;
LOG.debug("Call: " + method.getName() + " took " + callTime + "ms");
}
if (Client.isAsynchronousMode()) {
final AsyncGet<RpcWritable.Buffer, IOException> arr
= Client.getAsyncRpcResponse();
final AsyncGet<Message, Exception> asyncGet
= new AsyncGet<Message, Exception>() {
@Override
public Message get(long timeout, TimeUnit unit) throws Exception {
return getReturnMessage(method, arr.get(timeout, unit));
}
@Override
public boolean isDone() {
return arr.isDone();
}
};
ASYNC_RETURN_MESSAGE.set(asyncGet);
return null;
} else {
return getReturnMessage(method, val);
}
}
艾玛,这个有点长啊.
挑几点重要的说.
- 构造请求头域，标明在什么接口上调用什么方法
- 获取请求调用的参数
- 调用RPC.Client发送请求
- 获取响应信息
下面分别来说:
1.构造请求头域，标明在什么接口上调用什么方法
//构造请求头域，标明在什么接口上调用什么方法
RequestHeaderProto rpcRequestHeader = constructRpcRequestHeader(method);
这里很简单,就是根据协议定义,将 协议名称, 调用的方法名称, 版本号, 三个值传入,构建一个消息头.
private RequestHeaderProto constructRpcRequestHeader(Method method) {
RequestHeaderProto.Builder builder = RequestHeaderProto
.newBuilder();
builder.setMethodName(method.getName());
builder.setDeclaringClassProtocolName(protocolName);
builder.setClientProtocolVersion(clientProtocolVersion);
return builder.build();
}
2.获取请求调用的参数
获取请求调用的参数，这个是才client端代码就创建好的,通过参数传进来的.
// 获取请求调用的参数，这个是才client端代码就创建好的,通过参数传进来的.
// 例如 GetMetaInfoRequestProto
final Message theRequest = (Message) args[1];
3.调用RPC.Client发送请求
这个是核心,毕竟是发送请求,我们来重点关注一下.
//调用RPC.Client发送请求
val = (RpcWritable.Buffer) client.call(RPC.RpcKind.RPC_PROTOCOL_BUFFER,
new RpcProtobufRequest(rpcRequestHeader, theRequest), remoteId,
fallbackToSimpleAuth, alignmentContext);Client这个我就不说了,ProtobufRpcEngine#Invoker构造方法里面创建的.
/**
* This constructor takes a connectionId, instead of creating a new one.
*/
private Invoker(Class<?> protocol, Client.ConnectionId connId,
Configuration conf, SocketFactory factory) {
// 设置ConnectionId , ConnectionId里面保存着Client连接Server的信息
this.remoteId = connId;
// 获取/创建 客户端
this.client = CLIENTS.getClient(conf, factory, RpcWritable.Buffer.class);
// 获取协议的名称
this.protocolName = RPC.getProtocolName(protocol);
// 获取协议的版本 version
this.clientProtocolVersion = RPC
.getProtocolVersion(protocol);
}我们直接看Client#call 方法.
方法有点长. 摘取主要的说.
3.1.创建 Call 对象
3.2.获取&建立连接
3.3.发送RPC请求
3.4.获取响应
/**
*
* Make a call, passing <code>rpcRequest</code>, to the IPC server defined by
* <code>remoteId</code>, returning the rpc response.
*
* @param rpcKind
* @param rpcRequest - contains serialized method and method parameters
* @param remoteId - the target rpc server
* @param serviceClass - service class for RPC
* @param fallbackToSimpleAuth - set to true or false during this method to
* indicate if a secure client falls back to simple auth
* @param alignmentContext - state alignment context
* @return the rpc response
* Throws exceptions if there are network problems or if the remote code
* threw an exception.
*/
Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
ConnectionId remoteId, int serviceClass,
AtomicBoolean fallbackToSimpleAuth, AlignmentContext alignmentContext)
throws IOException {
/**
* 创建 Call 对象
* Client.call()方法将RPC请求封装成一个Call对象，
* Call对象中保存了RPC调用的完 成标志、返回值信息以及异常信息;
* 随后，Client.cal()方法会创建一个 Connection对象，
* Connection对象用于管理Client与Server的Socket连接。
*/
final Call call = createCall(rpcKind, rpcRequest);
call.setAlignmentContext(alignmentContext);
//获取&建立连接
//用ConnectionId作为key，
// 将新建的Connection对象放入Client.connections字段中保 存
// (
// 对于Connection对象，
// 由于涉及了与Server建立Socket连接，会比较耗费资 源，
// 所以Client类使用一个HashTable对象connections保存那些没有过期的 Connection，
// 如果可以复用，则复用这些Connection对象
// );
// 以callId作为key，将 构造的Call对象放入Connection.calls字段中保存。
final Connection connection = getConnection(remoteId, call, serviceClass,
fallbackToSimpleAuth);
try {
//检测是否是异步请求.
checkAsyncCall();
try {
//发送RPC请求
connection.sendRpcRequest(call); // send the rpc request
} catch (RejectedExecutionException e) {
throw new IOException("connection has been closed", e);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
IOException ioe = new InterruptedIOException(
"Interrupted waiting to send RPC request to server");
ioe.initCause(ie);
throw ioe;
}
} catch(Exception e) {
if (isAsynchronousMode()) {
releaseAsyncCall();
}
throw e;
}
if (isAsynchronousMode()) {
final AsyncGet<Writable, IOException> asyncGet
= new AsyncGet<Writable, IOException>() {
@Override
public Writable get(long timeout, TimeUnit unit)
throws IOException, TimeoutException{
boolean done = true;
try {
final Writable w = getRpcResponse(call, connection, timeout, unit);
if (w == null) {
done = false;
throw new TimeoutException(call + " timed out "
+ timeout + " " + unit);
}
return w;
} finally {
if (done) {
releaseAsyncCall();
}
}
}
@Override
public boolean isDone() {
synchronized (call) {
return call.done;
}
}
};
ASYNC_RPC_RESPONSE.set(asyncGet);
return null;
} else {
//服务器成功发回响应信息，返回RPC响应
return getRpcResponse(call, connection, -1, null);
}
}
3.1.创建 Call 对象
/**
* 创建 Call 对象
* Client.call()方法将RPC请求封装成一个Call对象，
* Call对象中保存了RPC调用的完 成标志、返回值信息以及异常信息;
* 随后，Client.cal()方法会创建一个 Connection对象，
* Connection对象用于管理Client与Server的Socket连接。
*/
final Call call = createCall(rpcKind, rpcRequest);
/**
*
* @param rpcKind rpcKind参数用于描述RPC请求的序列化工具类型
* @param rpcRequest rpcRequest参数则用于记录序列化后的RPC请求
* @return
*/
Call createCall(RPC.RpcKind rpcKind, Writable rpcRequest) {
return new Call(rpcKind, rpcRequest);
}这里就是构造方法, 生成一个唯一的callId ,进行初始化 .
private Call(RPC.RpcKind rpcKind, Writable param) {
this.rpcKind = rpcKind;
this.rpcRequest = param;
//获取 callId
final Integer id = callId.get();
if (id == null) {
// AtomicInteger callIdCounter 自增
this.id = nextCallId();
} else {
callId.set(null);
this.id = id;
}
final Integer rc = retryCount.get();
if (rc == null) {
this.retry = 0;
} else {
this.retry = rc;
}
// 设置异常处理类
this.externalHandler = EXTERNAL_CALL_HANDLER.get();
}
3.2.获取&建立连接
用ConnectionId作为key，
将新建的Connection对象放入Client.connections字段中保 存
(
对于Connection对象，
由于涉及了与Server建立Socket连接，会比较耗费资 源，
所以Client类使用一个ConcurrentMap对象connections保存那些没有过期的 Connection，
如果可以复用，则复用这些Connection对象
);
以callId作为key，将 构造的Call对象放入Connection.calls字段中保存。
final Connection connection = getConnection(remoteId, call, serviceClass,
fallbackToSimpleAuth);
/** Get a connection from the pool, or create a new one and add it to the
* pool. Connections to a given ConnectionId are reused.
* 获取连接
*
* */
private Connection getConnection(ConnectionId remoteId,
Call call, int serviceClass, AtomicBoolean fallbackToSimpleAuth)
throws IOException {
if (!running.get()) {
// the client is stopped
throw new IOException("The client is stopped");
}
Connection connection;
/* we could avoid this allocation for each RPC by having a
* connectionsId object and with set() method. We need to manage the
* refs for keys in HashMap properly. For now its ok.
*/
while (true) {
// These lines below can be shorten with computeIfAbsent in Java8
//首先尝试从Client.connections队列中获取Connection对象
connection = connections.get(remoteId);
if (connection == null) {
//如果connections队列中没有保存，则构造新的对象
connection = new Connection(remoteId, serviceClass);
// putIfAbsent 如果对应的 key 已经有值了,
// 则忽略本次操作,直接返回旧值
Connection existing = connections.putIfAbsent(remoteId, connection);
if (existing != null) {
connection = existing;
}
}
//将待发送请求对应的Call对象放入Connection.calls队列
if (connection.addCall(call)) {
break;
} else {
// This connection is closed, should be removed. But other thread could
// have already known this closedConnection, and replace it with a new
// connection. So we should call conditional remove to make sure we only
// remove this closedConnection.
connections.remove(remoteId, connection);
}
}
// If the server happens to be slow, the method below will take longer to
// establish a connection.
//调用setupIOstreams()方法，初始化Connection对象并获取IO流
connection.setupIOstreams(fallbackToSimpleAuth);
return connection;
}里面创建connection就不细说了, 其实就是创建connection对象,然后做一下请求地址和请求参数的设置. 并没有和server端进行请求.
//如果connections队列中没有保存，则构造新的对象
connection = new Connection(remoteId, serviceClass);
connection.setupIOstreams 这个才是建立连接的关键方法.
//调用setupIOstreams()方法，初始化Connection对象并获取IO流
connection.setupIOstreams(fallbackToSimpleAuth);
/**
*
*
* Connect to the server and set up the I/O streams. It then sends
* a header to the server and starts
* the connection thread that waits for responses.
*
* Client.call()方法调用 Connection.setupIOstreams() 方法建立与Server的Socket连接。
* setupIOstreams() 方法还会启动Connection线程，
* Connection线程会监听Socket并读取Server发回的响应信息。
*
*/
private synchronized void setupIOstreams( AtomicBoolean fallbackToSimpleAuth) {
if (socket != null || shouldCloseConnection.get()) {
return;
}
UserGroupInformation ticket = remoteId.getTicket();
if (ticket != null) {
final UserGroupInformation realUser = ticket.getRealUser();
if (realUser != null) {
ticket = realUser;
}
}
try {
connectingThread.set(Thread.currentThread());
if (LOG.isDebugEnabled()) {
LOG.debug("Connecting to "+server);
}
Span span = Tracer.getCurrentSpan();
if (span != null) {
span.addTimelineAnnotation("IPC client connecting to " + server);
}
short numRetries = 0;
Random rand = null;
while (true) {
//建立到Server的Socket连接，
// 并且在这个Socket连接上 获得InputStream和OutputStream对象。
setupConnection(ticket);
ipcStreams = new IpcStreams(socket, maxResponseLength);
// 调用writeConnectionHeader()方法在连接建立时发送连接头域。
writeConnectionHeader(ipcStreams);
if (authProtocol == AuthProtocol.SASL) {
try {
authMethod = ticket
.doAs(new PrivilegedExceptionAction<AuthMethod>() {
@Override
public AuthMethod run()
throws IOException, InterruptedException {
return setupSaslConnection(ipcStreams);
}
});
} catch (IOException ex) {
if (saslRpcClient == null) {
// whatever happened -it can't be handled, so rethrow
throw ex;
}
// otherwise, assume a connection problem
authMethod = saslRpcClient.getAuthMethod();
if (rand == null) {
rand = new Random();
}
handleSaslConnectionFailure(numRetries++, maxRetriesOnSasl, ex,
rand, ticket);
continue;
}
if (authMethod != AuthMethod.SIMPLE) {
// Sasl connect is successful. Let's set up Sasl i/o streams.
ipcStreams.setSaslClient(saslRpcClient);
// for testing
remoteId.saslQop =
(String)saslRpcClient.getNegotiatedProperty(Sasl.QOP);
LOG.debug("Negotiated QOP is :" + remoteId.saslQop);
if (fallbackToSimpleAuth != null) {
fallbackToSimpleAuth.set(false);
}
} else if (UserGroupInformation.isSecurityEnabled()) {
if (!fallbackAllowed) {
throw new IOException("Server asks us to fall back to SIMPLE " +
"auth, but this client is configured to only allow secure " +
"connections.");
}
if (fallbackToSimpleAuth != null) {
fallbackToSimpleAuth.set(true);
}
}
}
if (doPing) {
ipcStreams.setInputStream(new PingInputStream(ipcStreams.in));
}
//调用writeConnectionContext()方法写入连接上下文。
writeConnectionContext(remoteId, authMethod);
// update last activity time
//调用touch()方法更新上次活跃时间。
touch();
span = Tracer.getCurrentSpan();
if (span != null) {
span.addTimelineAnnotation("IPC client connected to " + server);
}
// 开启接收线程
// 调用start()方法启动Connection线程监听并接收Server发回的响应信息。
// start the receiver thread after the socket connection has been set up
start();
return;
}
} catch (Throwable t) {
if (t instanceof IOException) {
markClosed((IOException)t);
} else {
markClosed(new IOException("Couldn't set up IO streams: " + t, t));
}
close();
} finally {
connectingThread.set(null);
}
}在这个方法中通过setupConnection(ticket); 方法与server端建立连接.
//建立到Server的Socket连接，
// 并且在这个Socket连接上 获得InputStream和OutputStream对象。
setupConnection(ticket);
调用writeConnectionHeader()方法在连接建立时发送连接头信息。
writeConnectionHeader(ipcStreams); /* Write the connection context header for each connection
* Out is not synchronized because only the first thread does this.
*/
private void writeConnectionContext(ConnectionId remoteId,
AuthMethod authMethod)
throws IOException {
// Write out the ConnectionHeader
IpcConnectionContextProto message = ProtoUtil.makeIpcConnectionContext(
RPC.getProtocolName(remoteId.getProtocol()),
remoteId.getTicket(),
authMethod);
RpcRequestHeaderProto connectionContextHeader = ProtoUtil
.makeRpcRequestHeader(RpcKind.RPC_PROTOCOL_BUFFER,
OperationProto.RPC_FINAL_PACKET, CONNECTION_CONTEXT_CALL_ID,
RpcConstants.INVALID_RETRY_COUNT, clientId);
// do not flush. the context and first ipc call request must be sent
// together to avoid possibility of broken pipes upon authz failure.
// see writeConnectionHeader
final ResponseBuffer buf = new ResponseBuffer();
connectionContextHeader.writeDelimitedTo(buf);
message.writeDelimitedTo(buf);
synchronized (ipcStreams.out) {
ipcStreams.sendRequest(buf.toByteArray());
}
}
开启接收线程 调用start()方法启动Connection线程监听并接收Server发回的响应信息。
// 开启接收线程
// 调用start()方法启动Connection线程监听并接收Server发回的响应信息。
// start the receiver thread after the socket connection has been set up
start();
在这里会调用Connection里面的run()方法. 接收并处理返回消息.
@Override
public void run() {
if (LOG.isDebugEnabled())
LOG.debug(getName() + ": starting, having connections "
+ connections.size());
try {
while (waitForWork()) {//wait here for work - read or close connection
//接收到返回信息
receiveRpcResponse();
}
} catch (Throwable t) {
// This truly is unexpected, since we catch IOException in receiveResponse
// -- this is only to be really sure that we don't leave a client hanging
// forever.
LOG.warn("Unexpected error reading responses on connection " + this, t);
markClosed(new IOException("Error reading responses", t));
}
close();
if (LOG.isDebugEnabled())
LOG.debug(getName() + ": stopped, remaining connections "
+ connections.size());
}/**
接收到返回信息
Receive a response.
* Because only one receiver, so no synchronization on in.
* receiveRpcResponse()方法接收RPC响应。
* receiveRpcResponse()方法 会从输入流中读取序列化对象RpcResponseHeaderProto，
* 然后根据RpcResponseHeaderProto 中记录的callid字段获取对应的Call的对象。
*
* 接下来receiveRpcResponse()方法会从输入流中 读取响应消息，
* 然后调用Call.setRpcResponse()将响应消息保存在Call对象中。
* 如果服务器 在处理RPC请求时抛出异常，
* 则receiveRpcResponse()会从输入流中读取异常信息，并构造 异常对象，
* 然后调用Call.setException()将异常保存在Call对象中。
*
*/
private void receiveRpcResponse() {
if (shouldCloseConnection.get()) {
return;
}
//更新请求时间的
touch();
try {
//通过ipcStreams 获取响应对象
ByteBuffer bb = ipcStreams.readResponse();
//将响应对象,采用RpcWritable 进行包装.
RpcWritable.Buffer packet = RpcWritable.Buffer.wrap(bb);
// 获取响应的响应头.
RpcResponseHeaderProto header = packet.getValue(RpcResponseHeaderProto.getDefaultInstance());
// 检测头信息.
checkResponse(header);
// 获取call唯一标识callId
int callId = header.getCallId();
if (LOG.isDebugEnabled())
LOG.debug(getName() + " got value #" + callId);
// 获取头信息里面的响应状态.
RpcStatusProto status = header.getStatus();
//如果调用成功，则读取响应消息，在call实例中设置
if (status == RpcStatusProto.SUCCESS) {
//构建实例
Writable value = packet.newInstance(valueClass, conf);
//将Call 从 正在执行的calls缓存队列中移除.
final Call call = calls.remove(callId);
//将Call 设置请求信息
call.setRpcResponse(value);
if (call.alignmentContext != null) {
call.alignmentContext.receiveResponseState(header);
}
}
// verify that packet length was correct
if (packet.remaining() > 0) {
throw new RpcClientException("RPC response length mismatch");
}
//RPC调用失败
if (status != RpcStatusProto.SUCCESS) { // Rpc Request failed
//取出响应中的异常消息
final String exceptionClassName = header.hasExceptionClassName() ?
header.getExceptionClassName() :
"ServerDidNotSetExceptionClassName";
final String errorMsg = header.hasErrorMsg() ?
header.getErrorMsg() : "ServerDidNotSetErrorMsg" ;
final RpcErrorCodeProto erCode =
(header.hasErrorDetail() ? header.getErrorDetail() : null);
if (erCode == null) {
LOG.warn("Detailed error code not set by server on rpc error");
}
RemoteException re = new RemoteException(exceptionClassName, errorMsg, erCode);
//在Call对象中设置异常
if (status == RpcStatusProto.ERROR) {
final Call call = calls.remove(callId);
call.setException(re);
} else if (status == RpcStatusProto.FATAL) {
// Close the connection
markClosed(re);
}
}
} catch (IOException e) {
markClosed(e);
}
}
3.3.发送RPC请求
先构造RPC请求头. 将请求头和RPC请求 ,在 另外一个线程池 sendParamsExecutor的run方法中使用ipcStreams.sendRequest(buf.toByteArray()); 发送出去. 里面写的太麻烦了, 自己看吧.
//发送RPC请求
connection.sendRpcRequest(call); // send the rpc request
/**
*
* 发送RPC请求到Server。
*
* Initiates a rpc call by sending the rpc request to the remote server.
* Note: this is not called from the Connection thread, but by other
* threads.
* @param call - the rpc request
*
* RPC发送请求线程会调用Connection.sendRpcRequest()方法发送RPC请求到Server，
* 这 里要特别注意，这个方法不是由Connection线程调用的，
* 而是由发起RPC请求的线程调用 的。
*
*/
public void sendRpcRequest(final Call call)
throws InterruptedException, IOException {
if (shouldCloseConnection.get()) {
return;
}
// Serialize the call to be sent. This is done from the actual
// caller thread, rather than the sendParamsExecutor thread,
// so that if the serialization throws an error, it is reported
// properly. This also parallelizes the serialization.
//
// Format of a call on the wire:
// 0) Length of rest below (1 + 2)
// 1) RpcRequestHeader - is serialized Delimited hence contains length
// 2) RpcRequest
//
// Items '1' and '2' are prepared here.
//先构造RPC请求头
RpcRequestHeaderProto header = ProtoUtil.makeRpcRequestHeader(
call.rpcKind, OperationProto.RPC_FINAL_PACKET, call.id, call.retry,
clientId, call.alignmentContext);
final ResponseBuffer buf = new ResponseBuffer();
//将RPC请求头写入 输出流 ResponseBuffer
header.writeDelimitedTo(buf);
//将RPC请求(包括请求元数据和请求参数)封装成ProtobufWrapper ==> 写入输出流
RpcWritable.wrap(call.rpcRequest).writeTo(buf);
//这里使用 线程池 将请求发送出去，
// 请求包括三个部分:
// 1 长度;
// 2 RPC请求头;
// 3 RPC请求(包括 请求元数据以及请求参数)
synchronized (sendRpcRequestLock) {
Future<?> senderFuture = sendParamsExecutor.submit(new Runnable() {
@Override
public void run() {
try {
synchronized (ipcStreams.out) {
if (shouldCloseConnection.get()) {
return;
}
if (LOG.isDebugEnabled()) {
LOG.debug(getName() + " sending #" + call.id
+ " " + call.rpcRequest);
}
// RpcRequestHeader + RpcRequest
ipcStreams.sendRequest(buf.toByteArray());
ipcStreams.flush();
}
} catch (IOException e) {
// exception at this point would leave the connection in an
// unrecoverable state (eg half a call left on the wire).
// So, close the connection, killing any outstanding calls
//如果发生发送异常，则直接关闭连接
markClosed(e);
} finally {
//the buffer is just an in-memory buffer, but it is still polite to
// close early
//之前申请的buffer给关闭了，比较优雅
IOUtils.closeStream(buf);
}
}
});
try {
senderFuture.get();
} catch (ExecutionException e) {
Throwable cause = e.getCause();
// cause should only be a RuntimeException as the Runnable above
// catches IOException
if (cause instanceof RuntimeException) {
throw (RuntimeException) cause;
} else {
throw new RuntimeException("unexpected checked exception", cause);
}
}
}
}
3.4.获取响应
//服务器成功发回响应信息，返回RPC响应
return getRpcResponse(call, connection, -1, null);在这里,会不断轮训call的状态, 如果状态为done则代表数据已经处理完,并且已经获取到响应信息.
响应信息由connection中的run方法进行处理.
/** @return the rpc response or, in case of timeout, null. */
private Writable getRpcResponse(final Call call, final Connection connection,
final long timeout, final TimeUnit unit) throws IOException {
synchronized (call) {
while (!call.done) {
try {
//等待RPC响应
AsyncGet.Util.wait(call, timeout, unit);
if (timeout >= 0 && !call.done) {
return null;
}
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
throw new InterruptedIOException("Call interrupted");
}
}
if (call.error != null) {
if (call.error instanceof RemoteException) {
call.error.fillInStackTrace();
throw call.error;
} else { // local exception
InetSocketAddress address = connection.getRemoteAddress();
throw NetUtils.wrapException(address.getHostName(),
address.getPort(),
NetUtils.getHostname(),
0,
call.error);
}
} else {
return call.getRpcResponse();
}
}
4. 处理Client 端的响应信息
Client 获取响应信息, 不过是同步还是异步获取响应信息,都会调用这个方法: getReturnMessage(method, val);
getReturnMessage(method, val);
private Message getReturnMessage(final Method method,
final RpcWritable.Buffer buf) throws ServiceException {
Message prototype = null;
try {
//获取返回参数
prototype = getReturnProtoType(method);
} catch (Exception e) {
throw new ServiceException(e);
}
Message returnMessage;
try {
// 序列化响应信息
returnMessage = buf.getValue(prototype.getDefaultInstanceForType());
if (LOG.isTraceEnabled()) {
LOG.trace(Thread.currentThread().getId() + ": Response <- " +
remoteId + ": " + method.getName() +
" {" + TextFormat.shortDebugString(returnMessage) + "}");
}
} catch (Throwable e) {
throw new ServiceException(e);
}
//返回结果
return returnMessage;
}
最后我们看client端最后的步骤. 输出返回的信息.
640
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
