上一篇文章说到,现在这种每发起请求一次就新建一个client链接,但是如果在并发比较高的情况下,就会造成资源浪费,如果通过client和server进行长期链接,把需要处理的请求存到client里面,并且通过异步的形式返回,便会减少资源浪费。
这里有两个主要的问题,1、如何实现异步返回?2、如何把client和server对应起来?
先看第一个问题,所谓的异步返回,可以以点奶茶为例,当顾客(消费方)向奶茶店(服务端)点了一杯奶茶(发起请求),奶茶店给他一个号码(凭证),然后客户拿着号就去逛街干其他事情,过了一段时间回来问奶茶店有没有好了,此时奶茶店根据他的号码来给他对应的奶茶。
这里就是为什么上篇文章说的请求要带上requestID,这就相当于凭证的标识,而客户端链接可以保存这些请求并且返回一个凭证,当有返回时,把对应的结果塞到凭证里面,客户端可以做自己的事情或者一直等待凭证有结果,这时候结合线程池去做这个处理
第二个问题,需要一个链接池来管理链接和对应的远端关系
看看编码实现
第一个问题最重要的就是如何设计一个凭证,这时候需要用到Future接口来实现自己的凭证,通过一个内部同步器的管理,这个同步器主要是起到自旋改变状态的作用,同步器因为只有一个线程节点,会不断自旋尝试获取锁,获取锁的条件是远端有返回并且改变了同步器的值。
/**
* @author: lele
* @date: 2019/11/21 上午10:54
* 实现异步返回
*/
public class RpcFuture implements Future<Object> {
private RpcResponse rpcResponse;
private RpcRequest rpcRequest;
/**
* 自定义同步器,这里只是用来通过自选改变状态
*/
private Sync sync;
public RpcFuture(RpcRequest rpcRequest) {
this.rpcRequest = rpcRequest;
this.sync = new Sync();
}
@Override
public boolean cancel(boolean mayInterruptIfRunning) {
throw new UnsupportedOperationException();
}
@Override
public boolean isCancelled() {
throw new UnsupportedOperationException();
}
/**
* 返回状态是否改变了
* @return
*/
@Override
public boolean isDone() {
return sync.isDone();
}
/**
* 赋值并设置同步器锁状态为1
* @param response
*/
public void done(RpcResponse response) {
this.rpcResponse = response;
sync.tryRelease(1);
}
/**
* 自选等待结果,这里一直执行acquire
* 直到tryacquire方法return true即state为1
* @return
* @throws InterruptedException
* @throws ExecutionException
*/
@Override
public Object get() throws InterruptedException, ExecutionException {
sync.acquire(-1);
return this.rpcResponse;
}
/**
* 超时抛异常
* @param timeout
* @param unit
* @return
* @throws InterruptedException
* @throws ExecutionException
* @throws TimeoutException
*/
@Override
public Object get(long timeout, TimeUnit unit) throws InterruptedException, ExecutionException, TimeoutException {
//超时获取
boolean success = sync.tryAcquireNanos(-1, unit.toNanos(timeout));
return get(success);
}
private Object get(boolean success) {
if (success) {
return this.rpcResponse;
} else {
throw new RuntimeException("超时:requestID" + rpcRequest.getRequestId() +
" method:" + rpcRequest.getMethodName() + " interface:" + rpcRequest.getInterfaceName()
);
}
}
/**
* 继承同步器,这里只是用来自旋改变状态,根据state来实现,state初始为0
*/
static class Sync extends AbstractQueuedSynchronizer {
/**
* 尝试获取锁,如果获取不了,加入同步队列,阻塞自己,只由同步队列的头自旋获取锁
* 当状态为1,即有结果返回时可以获取锁进行后续操作,设置result
*这里只有一个节点,会不断自选尝试获取锁
* @param arg
* @return
*/
@Override
protected boolean tryAcquire(int arg) {
return getState() == 1;
}
/**
* 用于远端有返回时,设置状态变更
* 从头唤醒同步队列的队头下一个等待的节点,如果下一个节点为空,则从队尾唤醒
* @param arg
* @return
*/
@Override
protected boolean tryRelease(int arg) {
//把状态设置为1,给tryAcquire获取锁进行操作
return getState() == 0 ? compareAndSetState(0, 1) : true;
}
public boolean isDone() {
return getState() == 1;
}
}
}
凭证写好了,接下来就写处理凭证的handler类,这个类有一个发送请求的方法,主要是把凭证和对应的标识存储起来,并在有结果返回时设置返回状态
package com.gdut.rpcstudy.demo.framework.protocol.netty.asyn;
import com.gdut.rpcstudy.demo.framework.URL;
import com.gdut.rpcstudy.demo.framework.protocol.netty.asyn.RpcFuture;
import com.gdut.rpcstudy.demo.framework.serialize.tranobject.RpcRequest;
import com.gdut.rpcstudy.demo.framework.serialize.tranobject.RpcResponse;
import io.netty.buffer.Unpooled;
import io.netty.channel.*;
import lombok.Data;
import lombok.EqualsAndHashCode;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CountDownLatch;
/**
* @author: lele
* @date: 2019/11/21 下午4:07
* 异步模式下的处理
*/
@Data
@EqualsAndHashCode(callSuper = false)
public class NettyAsynHandler extends SimpleChannelInboundHandler<RpcResponse> {
//key:requestId,value自定义future
private ConcurrentHashMap<String, RpcFuture> resultMap = new ConcurrentHashMap<>();
private volatile Channel channel;
//对应的远端URL
private final URL url;
@Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
this.channel = ctx.channel();
}
public void close() {
this.channel.close();
}
@Override
protected void channelRead0(ChannelHandlerContext channelHandlerContext, RpcResponse s) throws Exception {
System.out.println("收到结果:" + s);
String requestId = s.getRequestId();
//设置完成并移除future
RpcFuture future = resultMap.get(requestId);
if (s != null) {
future.done(s);
resultMap.remove(requestId);
}
}
public RpcFuture sendRequest(RpcRequest rpcRequest) {
final CountDownLatch latch = new CountDownLatch(1);
RpcFuture future = new RpcFuture(rpcRequest);
//放到请求列表里面
resultMap.put(rpcRequest.getRequestId(), future);
//发送请求
channel.writeAndFlush(rpcRequest).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture channelFuture) throws Exception {
System.out.println("发送了消息" + rpcRequest.toString());
latch.countDown();
}
});
try {
//等待结果
latch.await();
} catch (InterruptedException e) {
e.printStackTrace();
}
return future;
}
}
第一个问题的两个比较核心的类就编写到这里。
然后到第二个问题,如何实现一个链接管理池?要为每个服务下的每个server建立一个客户端,则需要先通过注册中心获取地址,再进行后续操作,这里获取每个服务下的server地址
public static Map<String,List<URL>> getAllURL(){
Map<String,List<URL>> mapList=null;
try {
List<String> servcieList=client.getChildren().forPath("/");
mapList=new HashMap<>(servcieList.size());
for (String s : servcieList) {
mapList.put(s,getService(s));
}
} catch (Exception e) {
e.printStackTrace();
}
return mapList;
}
获取地址后,我们可以建立关系了,采用{serviceName:{serverAddress:clienthandler}}的形式进行存储,从client handler存放的就是上面说的客户端处理器,通过这里我们可以通过不同的选择方式挑一个出来发送请求,这里采用的选择方式是轮询。
大概是这样的形式去获取凭证,管理池根据服务名轮询地选取一个客户端,并且把凭证让他处理,然后返回凭证,可这里有个隐患,客户端连接到服务端是需要时间的,再没有可用的客户端情况下,会报空指针异常
public RpcFuture sendFuture(String serviceName, RpcRequest request) {
NettyAsynHandler handler=ConnectManager.getInstance().getConnectionWithPolling(serviceName);
RpcFuture future = handler.sendRequest(request);
return future;
}
这时候我们需要为加入锁和条件队列(Condition),当满足条件时,才可以链接,不然就自旋等待直到条件满足,当客户端链接上u对应的server,我们才可以获取对应的客户端处理器,但是服务有这么多,用一个锁是可以,但是调用不相干的服务也要被锁吗?这时候可以细粒度化我们的锁,为每个服务建立一个锁及其相应的条件队列。这是连接池需要注意的点,下面看看代码
/**
* @author: lele
* @date: 2019/11/21 上午11:58
* 管理连接池
*/
public class ConnectManager {
private Boolean isShutDown = false;
/**
* 客户端链接服务端超时时间
*/
private long connectTimeoutMillis = 5000;
/**
* 自定义6个线程组用于客户端服务
*/
private EventLoopGroup eventLoopGroup = new NioEventLoopGroup(6);
/**
* 存放服务对应的访问数,用于轮询
*/
private Map<String, AtomicInteger> pollingMap = new ConcurrentHashMap<>();
/**
* 对于每个服务都有一个锁,每个锁都有个条件队列,用于控制链接获取
*/
private Map<String, Object[]> serviceCondition = new ConcurrentHashMap<>();
/**
* 存放服务端地址和handler的关系
*/
private Map<String, Map<URL, NettyAsynHandler>> serverClientMap = new ConcurrentHashMap<>();
/**
* 用来初始化客户端
*/
private ThreadPoolExecutor clientBooter = new ThreadPoolExecutor(
16, 16, 600, TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(1024)
, new BooterThreadFactory(), new ThreadPoolExecutor.AbortPolicy());
private static class Holder {
private static final ConnectManager j = new ConnectManager();
}
private ConnectManager() {
//初始化时把所有的url加进去,这里可能没有可用链接,所以需要添加对节点的监听
Map<String, List<URL>> allURL = ZkRegister.getAllURL();
for (String s : allURL.keySet()) {
//为每个服务添加锁和条件队列,通过条件队列控制客户端链接获取
ReentrantLock lock =new ReentrantLock();
Condition condition = lock.newCondition();
serviceCondition.put(s,new Object[]{lock,condition});
}
addServer(allURL);
}
public static ConnectManager getInstance() {
return Holder.j;
}
/**
* 添加该服务对应的链接和handler
* @param serviceName
* @param url
* @param handler
*/
public void addConnection(String serviceName, URL url, NettyAsynHandler handler) {
Map<URL, NettyAsynHandler> handlerMap;
if (!serverClientMap.containsKey(serviceName)) {
handlerMap = new HashMap<>();
} else {
handlerMap = serverClientMap.get(serviceName);
}
handlerMap.put(url, handler);
//添加服务名和对应的url:客户端链接
serverClientMap.put(serviceName, handlerMap);
//唤醒等待客户端链接的线程
signalAvailableHandler(serviceName);
}
/**
* 获取对应服务下的handler,通过轮询获取
* @param servicName
* @return
*/
public NettyAsynHandler getConnectionWithPolling(String servicName) {
Map<URL, NettyAsynHandler> urlNettyAsynHandlerMap = serverClientMap.get(servicName);
int size = 0;
//先尝试获取
if (urlNettyAsynHandlerMap != null) {
size = urlNettyAsynHandlerMap.size();
}
//不行就自选等待
while (!isShutDown && size <= 0) {
try {
//自旋等待可用服务出现,因为客户端与服务链接需要一定的时间,如果直接返回会出现空指针异常
boolean available = waitingForHandler(servicName);
if (available) {
urlNettyAsynHandlerMap = serverClientMap.get(servicName);
size = urlNettyAsynHandlerMap.size();
}
} catch (InterruptedException e) {
throw new RuntimeException("出错", e);
}
}
//获取对应的访问次数
AtomicInteger count = pollingMap.get(servicName);
int index = (count.getAndAdd(1) + size) % size;
Iterator<Map.Entry<URL, NettyAsynHandler>> iterator = urlNettyAsynHandlerMap.entrySet().iterator();
//取出相应的handler
NettyAsynHandler nettyAsynHandler = null;
for (int i = 0; i <= index; i++) {
nettyAsynHandler = iterator.next().getValue();
}
return nettyAsynHandler;
}
/**
* 等待一定时间,等handler和相应的server建立建立链接,用条件队列控制
* @param serviceName
* @return
* @throws InterruptedException
*/
private boolean waitingForHandler(String serviceName) throws InterruptedException {
Object[] objects = serviceCondition.get(serviceName);
ReentrantLock lock = (ReentrantLock) objects[0];
lock.lock();
Condition condition= (Condition) objects[1];
try {
return condition.await(this.connectTimeoutMillis, TimeUnit.MILLISECONDS);
} finally {
lock.unlock();
}
}
public void removeURL(URL url) {
List<String> list = new ArrayList<>();
for (Map.Entry<String, Map<URL, NettyAsynHandler>> map : serverClientMap.entrySet()) {
for (Map.Entry<URL, NettyAsynHandler> urlNettyAsynHandlerEntry : map.getValue().entrySet()) {
if (urlNettyAsynHandlerEntry.getKey().equals(url)) {
urlNettyAsynHandlerEntry.getValue().close();
list.add(map.getKey() + "@" + urlNettyAsynHandlerEntry.getKey());
}
}
}
for (String s : list) {
String[] split = s.split("@");
serverClientMap.get(split[0]).remove(split[1]);
}
}
/**
* 释放对应服务的条件队列,代表有客户端链接可用了
* @param serviceName
*/
private void signalAvailableHandler(String serviceName) {
Object[] objects = serviceCondition.get(serviceName);
ReentrantLock lock = (ReentrantLock) objects[0];
lock.lock();
Condition condition= (Condition) objects[1];
try {
condition.signalAll();
} finally {
lock.unlock();
}
}
/**
* 添加server,并启动对应的服务器
* @param allURL
*/
public void addServer(Map<String, List<URL>> allURL) {
for (String s : allURL.keySet()) {
pollingMap.put(s, new AtomicInteger(0));
List<URL> urls = allURL.get(s);
for (URL url : urls) {
//提交创建任务
clientBooter.submit(new Runnable() {
@Override
public void run() {
createClient(s, eventLoopGroup, url);
}
});
}
}
System.out.println("初始化客户端ing");
}
/**
* 创建客户端,持久化链接
* @param serviceName
* @param eventLoopGroup
* @param url
*/
public void createClient(String serviceName, EventLoopGroup eventLoopGroup, URL url) {
Bootstrap b = new Bootstrap();
b.group(eventLoopGroup)
.channel(NioSocketChannel.class)
.handler((new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ch.pipeline()
//把request实体变为字节
.addLast(new RpcEncoder(RpcRequest.class))
//把返回的response字节变为对象
.addLast(new RpcDecoder(RpcResponse.class))
.addLast(new NettyAsynHandler(url));
}
}));
ChannelFuture channelFuture = b.connect(url.getHostname(), url.getPort());
channelFuture.addListener(new ChannelFutureListener() {
@Override
public void operationComplete(final ChannelFuture channelFuture) throws Exception {
//链接成功后的操作,把相应的url地址和客户端链接存入
if (channelFuture.isSuccess()) {
NettyAsynHandler handler = channelFuture.channel().pipeline().get(NettyAsynHandler.class);
addConnection(serviceName, url, handler);
}
}
});
}
/**
* 关闭方法,关闭每个客户端链接,释放所有锁,关掉创建链接的线程池,和客户端的处理器
*/
public void stop() {
isShutDown = true;
for (Map<URL, NettyAsynHandler> urlNettyAsynHandlerMap : serverClientMap.values()) {
urlNettyAsynHandlerMap.values().forEach(e -> e.close());
}
for(String s:serviceCondition.keySet()){
signalAvailableHandler(s);
}
clientBooter.shutdown();
eventLoopGroup.shutdownGracefully();
}
/**
* 启动客户端链接的自定义线程工厂
*/
static class BooterThreadFactory implements ThreadFactory {
private static final AtomicInteger poolNumber = new AtomicInteger(1);
private final ThreadGroup group;
private final AtomicInteger threadNumber = new AtomicInteger(1);
private final String namePrefix;
BooterThreadFactory() {
group = new ThreadGroup("connectManger");
group.setDaemon(false);
group.setMaxPriority(5);
namePrefix = "clientBooter-" +
poolNumber.getAndIncrement() +
"-thread-";
}
@Override
public Thread newThread(Runnable r) {
Thread t = new Thread(group, r,
namePrefix + threadNumber.getAndIncrement(),
0);
return t;
}
}
}
这两个问题解决好后,接下来对整体框架做修改,首先是接口注解,增加了异步模式,对象工厂根据注解来选择不同的代理方式,
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
//用于接口上,name为服务名,zk则在注册服务改为 服务名/ip,服务端通过传来的接口名通过反射获取类,或者通过给spring托管获取其class
public @interface RpcStudyClient {
String name();
//结果返回是异步还是同步模式
int mode() default sync;
int sync=0;
int asyn=1;
}
@Data
@EqualsAndHashCode(callSuper = false)
public class RpcStudyClientFactoryBean implements FactoryBean<Object> {
private Class<?> type;
@Override
public Object getObject() throws Exception {
//根据RpcStudeClient的mode字段选择以哪种方式代理
RpcStudyClient annotation = type.getAnnotation(RpcStudyClient.class);
int mode = annotation.mode();
return mode==RpcStudyClient.asyn?ProxyFactory.getAsyncProxy(this.type):ProxyFactory.getProxy(this.type);
}
@Override
public Class<?> getObjectType() {
return this.type;
}
}
代理工厂的新增异步代理,这里需要自旋获取结果,不然会导致结果获取不到。
public static <T> T getAsyncProxy(Class interfaceClass) {
return (T) Proxy.newProxyInstance(interfaceClass.getClassLoader(), new Class[]{interfaceClass}, new InvocationHandler() {
@Override
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
//指定所用协议
Protocol protocol = ProtocolFactory.netty();
RpcStudyClient annotation = (RpcStudyClient) interfaceClass.getAnnotation(RpcStudyClient.class);
String requestId = UUID.randomUUID().toString().replace("-", "");
//封装方法参数
RpcRequest rpcRequest = new RpcRequest(requestId, interfaceClass.getName(), method.getName(), args, method.getParameterTypes(), annotation.mode());
Future<RpcFuture> res = futureTask.submit(new Callable<RpcFuture>() {
@Override
public RpcFuture call() throws Exception {
RpcFuture res = protocol.sendFuture(annotation.fetch(), annotation.name(), rpcRequest);
//先尝试一次
if (res.isDone()) {
return res;
}
//不行就自旋等待
while (!res.isDone()) {
}
return res;
}
;
});
//发送请求
//这里的管理连接池通过服务名去访问zk,获取可用的url
return returnResult(res.get());
}
});
}
/**
* 具体的处理异步返回的方法
* @param res
* @return
* @throws ExecutionException
* @throws InterruptedException
*/
public static Object returnResult(RpcFuture res) throws ExecutionException, InterruptedException {
RpcResponse response = (RpcResponse) res.get();
if (response.getError() != null) {
throw new RuntimeException(response.getError());
} else {
return response.getResult();
}
}
请求实体类也增加多一个属性mode,标识是同步还是异步模式,给serverhandler做处理,如果是异步,则不关闭链接,不然会导致客户端发的请求无法接受,同步则关闭。
@Data
@AllArgsConstructor
public class RpcRequest {
private String requestId;
private String interfaceName;
private String methodName;
private Object[] params;
//防止重载
private Class[] paramsTypes;
//是否异步
private int mode;
}
ctx.writeAndFlush(response).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture channelFuture) throws Exception {
System.out.println("发送了结果"+response);
//如果是mode为同步,就直接关闭连接
if(rpcRequest.getMode()==RpcStudyClient.sync){
ctx.channel().close();
}
// 当异步模式时,不关闭链接
}
});
这里还可以为异步请求添加回调处理,慢请求日志记录等等,都可以在自定义的凭证里面下文章。异步请求和连接池大概就到这里,具体代码https://github.com/97lele/rpcstudy/tree/withconcurrent