本文详细介绍了Thrift服务端的各类模型，包括TSimpleServer的单线程阻塞模型，适用于测试；TThreadPoolServer的多线程阻塞模型，处理并发能力提升；以及非阻塞式服务模型如TNonblockingServer、THsHaServer和TThreadedSelectorServer，深入解析了它们的工作原理和各自特点。
文章目录
服务层是我们介绍thrift的最后一层，之前也介绍过，服务层的功能是将其他几个层进行聚合，以实现几层之间的功能上的耦合。我们从客户端和服务端两个方面来介绍服务层，二者都有一个核心的类：
- 客户端：TServiceClient
- 服务端：TServer
1.服务端
首先看下Tserver这个类的源码:
/**
* Generic interface for a Thrift server.
*
*/
public abstract class TServer {
public static class Args extends AbstractServerArgs<Args> {
public Args(TServerTransport transport) {
super(transport);
}
}
public static abstract class AbstractServerArgs<T extends AbstractServerArgs<T>> {
final TServerTransport serverTransport;
TProcessorFactory processorFactory;
TTransportFactory inputTransportFactory = new TTransportFactory();
TTransportFactory outputTransportFactory = new TTransportFactory();
TProtocolFactory inputProtocolFactory = new TBinaryProtocol.Factory();
TProtocolFactory outputProtocolFactory = new TBinaryProtocol.Factory();
...
private boolean isServing;
protected TServer(AbstractServerArgs args) {
processorFactory_ = args.processorFactory;
serverTransport_ = args.serverTransport;
inputTransportFactory_ = args.inputTransportFactory;
outputTransportFactory_ = args.outputTransportFactory;
inputProtocolFactory_ = args.inputProtocolFactory;
outputProtocolFactory_ = args.outputProtocolFactory;
}
/**
* The run method fires up the server and gets things going.
*/
public abstract void serve();
/**
* Stop the server. This is optional on a per-implementation basis. Not
* all servers are required to be cleanly stoppable.
*/
public void stop() {}
public boolean isServing() {
return isServing;
}
protected void setServing(boolean serving) {
isServing = serving;
}
}
TServer持有了上面几层的工厂对象，并暴露了服务器一般行为接口：启动和关闭服务器。thrift根据服务模型的不同，提供了几种不同的Tserver的实现类：
我们可以将其大致分为两类：
阻塞式服务模型：TSimpleServer、TThreadPoolServer
非阻塞式服务模型：TNonblockingServer、THsHaServer、TThreadedSelectorServer
- TSimpleServer：单线程阻塞模型，该模型仅能同时处理一个socket链接，主要用于测试
- TThreadPoolServer：多线程阻塞模型，该模型基于java的阻塞式IO，主线程循环接受socket链接，接收后将请求交给提供的线程池处理请求
- AbstractNonblockingServer：基于jdk提供的非阻塞IO模型建立的非阻塞抽象模型
- TNonblockingServer：AbstractNonblockingServer实现类，单线程非阻塞模型，提供一个线程作为selector来接受事件，并使用该线程同步阻塞处理IO事件
- THsHaServer：TNonblockingServer实现类，提供半同步半异步处理方式。半同步是由于依然使用单线程作为selector去接收IO事件，半异步则是提供了一个invoker线程池用来异步处理IO事件
- TThreadedSelectorServer：AbstractNonblockingServer实现类，多线程非阻塞IO模型，提供了一组selector线程，里面有多个线程作为selector去接收IO事件，提供了一个线程AcceptThread专门处理socket连接事件，同时与THsHaServer一样，提供了一个线程池去处理IO读写事件
接下来我们分别看下这个几个服务模型的工作原理：
1.1 TSimpleServer
public void serve() {
stopped_ = false;
try {
// 开启监听
serverTransport_.listen();
} catch (TTransportException ttx) {
LOGGER.error("Error occurred during listening.", ttx);
return;
}
setServing(true);
// 循环阻塞等待接收IO事件
while (!stopped_) {
TTransport client = null;
TProcessor processor = null;
TTransport inputTransport = null;
TTransport outputTransport = null;
TProtocol inputProtocol = null;
TProtocol outputProtocol = null;
try {
// 建立socket连接
client = serverTransport_.accept();
// 同步处理读写事件
if (client != null) {
processor = processorFactory_.getProcessor(client);
inputTransport = inputTransportFactory_.getTransport(client);
outputTransport = outputTransportFactory_.getTransport(client);
inputProtocol = inputProtocolFactory_.getProtocol(inputTransport);
outputProtocol = outputProtocolFactory_.getProtocol(outputTransport);
// 循环阻塞等待底层真实服务的业务逻辑完成，完成后该线程可以循环等待下次socket请求
while (processor.process(inputProtocol, outputProtocol)) {}
}
} catch (TTransportException ttx) {
// Client died, just move on
} catch (TException tx) {
if (!stopped_) {
LOGGER.error("Thrift error occurred during processing of message.", tx);
}
} catch (Exception x) {
if (!stopped_) {
LOGGER.error("Error occurred during processing of message.", x);
}
}
if (inputTransport != null) {
inputTransport.close();
}
if (outputTransport != null) {
outputTransport.close();
}
}
setServing(false);
}
很显然，该服务模型将所有的服务处理逻辑全部放在了一个线程中，其吞吐量可想而知，因此该模型我们应该仅用于测试
1.2 TThreadPoolServer
public void serve() {
try {
// 开启监听
serverTransport_.listen();
} catch (TTransportException ttx) {
LOGGER.error("Error occurred during listening.", ttx);
return;
}
stopped_ = false;
setServing(true);
while (!stopped_) {
int failureCount = 0;
try {
// 循环阻塞等待socket连接
TTransport client = serverTransport_.accept();
// socket请求到来，创建一个WorkerProcess线程去处理该请求
WorkerProcess wp = new WorkerProcess(client);
executorService_.execute(wp);
} catch (TTransportException ttx) {
if (!stopped_) {
++failureCount;
LOGGER.warn("Transport error occurred during acceptance of message.", ttx);
}
}
}
我们来看下WorkerProcess线程做了什么：
public void run() {
TProcessor processor = null;
TTransport inputTransport = null;
TTransport outputTransport = null;
TProtocol inputProtocol = null;
TProtocol outputProtocol = null;
try {
processor = processorFactory_.getProcessor(client_);
inputTransport = inputTransportFactory_.getTransport(client_);
outputTransport = outputTransportFactory_.getTransport(client_);
inputProtocol = inputProtocolFactory_.getProtocol(inputTransport);
outputProtocol = outputProtocolFactory_.getProtocol(outputTransport);
// 处理IO事件以及完成业务逻辑
while (!stopped_ && processor.process(inputProtocol, outputProtocol)) {}
} catch (TTransportException ttx) {
// Assume the client died and continue silently
} catch (TException tx) {
LOGGER.error("Thrift error occurred during processing of message.", tx);
} catch (Exception x) {
LOGGER.error("Error occurred during processing of message.", x);
}
if (inputTransport != null) {
inputTransport.close();
}
if (outputTransport != null) {
outputTransport.close();
}
}
与TSimpleServer相比，TThreadPoolServer有了明显的提升，至少IO事件和业务逻辑处理不用让主线程阻塞了，这就让服务器拥有了同时接受多个socket连接的能力，虽然如此，但是当并发数量大于最大处理线程数时，服务器依然无法接受请求
1.3 AbstractNonblockingServer
在介绍具体的非阻塞线程模型之前，我们先看下AbstractNonblockingServer这个抽象类，了解下thrift的非阻塞IO处理框架：
public void serve() {
// 开启IO处理线程
if (!startThreads()) {
return;
}
// 开启监听
if (!startListening()) {
return;
}
// 服务器运行状态置为true
setServing(true);
// 阻塞等待服务器关闭信号
waitForShutdown();
// 服务器运行状态置为false
setServing(false);
// 停止监听，做一些clean工作
stopListening();
}
startListening:
protected boolean startListening() {
try {
// 开启监听
serverTransport_.listen();
return true;
} catch (TTransportException ttx) {
LOGGER.error("Failed to start listening on server socket!", ttx);
return false;
}
}
AbstractNonBlockingServer提供了一个selector线程抽象模型来实现thrift中selector的功能：
protected abstract class AbstractSelectThread extends Thread {
protected final Selector selector;
// 一组FrameBuffer，这组FrameBuffer可以改变它们感兴趣的事件，如读转写，写转读等
protected final Set<FrameBuffer> selectInterestChanges = new HashSet<FrameBuffer>();
public AbstractSelectThread() throws IOException {
this.selector = SelectorProvider.provider().openSelector();
}
/**
* 用于唤醒阻塞的selector线程
*/
public void wakeupSelector() {
selector.wakeup();
}
/**
* 将framebuffer添加到上面的set中，并唤醒阻塞的selector，如果selector的select方法执行完毕，framebuffe就有机会改变其感兴趣的事件种类
*/
public void requestSelectInterestChange(FrameBuffer frameBuffer) {
synchronized (selectInterestChanges) {
selectInterestChanges.add(frameBuffer);
}
selector.wakeup();
}
/**
* 对所有要转变感兴趣事件的framebuffer进行兴趣事件转变，从读转到写或者从写转到读
*/
protected void processInterestChanges() {
synchronized (selectInterestChanges) {
for (FrameBuffer fb : selectInterestChanges) {
fb.changeSelectInterests();
}
selectInterestChanges.clear();
}
}
/**
* 如果framebuffer是可读的，执行读操作并进行接下来的业务逻辑
*/
protected void handleRead(SelectionKey key) {
FrameBuffer buffer = (FrameBuffer) key.attachment();
if (!buffer.read()) {
cleanupSelectionKey(key);
return;
}
// 如果完成读操作，则执行framebuffer的业务逻辑
if (buffer.isFrameFullyRead()) {
if (!requestInvoke(buffer)) {
cleanupSelectionKey(key);
}
}
}
/**
* 执行写操作
*/
protected void handleWrite(SelectionKey key) {
FrameBuffer buffer = (FrameBuffer) key.attachment();
if (!buffer.write()) {
cleanupSelectionKey(key);
}
}
/**
* 对指定的selectionKey（指代某个socket连接）进行关闭和清理操作
*/
protected void cleanupSelectionKey(SelectionKey key) {
// remove the records from the two maps
FrameBuffer buffer = (FrameBuffer) key.attachment();
if (buffer != null) {
// close the buffer
buffer.close();
}
// cancel the selection key
key.cancel();
}
} // SelectThread
我们在上面可以看到一个FrameBuffer对象，很显然，在该线程模型中，FrameBuffer对真正的IO操作和真实服务逻辑调用进行了封装：
/**
* FrameBuffer是一个有限状态机，其在网络IO读写的各类状态之间进行切换，提供了针对thrift的结构化协议的读写功能和真实服务逻辑的调用，下面介绍下几个主要方法
*/
protected class FrameBuffer {
/**
* 当读事件到来时，进行读操作
*/
public boolean read() {
// 如果状态为读取消息头中，首先读取frame消息头，即消息大小数据
if (state_ == FrameBufferState.READING_FRAME_SIZE) {
if (!internalRead()) {
return false;
}
// 读完消息头后，读取真实消息数据
if (buffer_.remaining() == 0) {
// pull out the frame size as an integer.
int frameSize = buffer_.getInt(0);
if (frameSize <= 0) {
LOGGER.error("Read an invalid frame size of " + frameSize
+ ". Are you using TFramedTransport on the client side?");
return false;
}
// 消息大小超过了可读的最大阈值，关闭连接打日志
if (frameSize > MAX_READ_BUFFER_BYTES) {
LOGGER.error("Read a frame size of " + frameSize
+ ", which is bigger than the maximum allowable buffer size for ALL connections.");
return false;
}
// 如果目前的内存不足以读取这个消息，先返回，等待重新分配更多内存
if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
return true;
}
// 将该消息的大小添加到已分配内存中，用于统计
readBufferBytesAllocated.addAndGet(frameSize);
// 分配该消息大小的空间
buffer_ = ByteBuffer.allocate(frameSize);
// 状态置为读取消息中
state_ = FrameBufferState.READING_FRAME;
} else {
// this skips the check of READING_FRAME state below, since we can't
// possibly go on to that state if there's data left to be read at
// this one.
return true;
}
}
// 消息头读取完毕，现在读取消息数据
if (state_ == FrameBufferState.READING_FRAME) {
if (!internalRead()) {
return false;
}
// since we're already in the select loop here for sure, we can just
// modify our selection key directly.
if (buffer_.remaining() == 0) {
// get rid of the read select interests
selectionKey_.interestOps(0);
state_ = FrameBufferState.READ_FRAME_COMPLETE;
}
return true;
}
// if we fall through to this point, then the state must be invalid.
LOGGER.error("Read was called but state is invalid (" + state_ + ")");
return false;
}
/**
* 写操作
*/
public boolean write() {
if (state_ == FrameBufferState.WRITING) {
try {
if (trans_.write(buffer_) < 0) {
return false;
}
} catch (IOException e) {
LOGGER.warn("Got an IOException during write!", e);
return false;
}
// 写操作完成，转换为读状态
if (buffer_.remaining() == 0) {
prepareRead();
}
return true;
}
LOGGER.error("Write was called, but state is invalid (" + state_ + ")");
return false;
}
/**
* 切换兴趣事件
*/
public void changeSelectInterests() {
if (state_ == FrameBufferState.AWAITING_REGISTER_WRITE) {
// set the OP_WRITE interest
selectionKey_.interestOps(SelectionKey.OP_WRITE);
state_ = FrameBufferState.WRITING;
} else if (state_ == FrameBufferState.AWAITING_REGISTER_READ) {
prepareRead();
} else if (state_ == FrameBufferState.AWAITING_CLOSE) {
close();
selectionKey_.cancel();
} else {
LOGGER.error("changeSelectInterest was called, but state is invalid (" + state_ + ")");
}
}
/**
* 关闭连接
*/
public void close() {
// if we're being closed due to an error, we might have allocated a
// buffer that we need to subtract for our memory accounting.
if (state_ == FrameBufferState.READING_FRAME || state_ == FrameBufferState.READ_FRAME_COMPLETE) {
readBufferBytesAllocated.addAndGet(-buffer_.array().length);
}
trans_.close();
}
/**
* Check if this FrameBuffer has a full frame read.
*/
public boolean isFrameFullyRead() {
return state_ == FrameBufferState.READ_FRAME_COMPLETE;
}
/**
* 当处理层处理完业务逻辑后，都必须调用该方法，
*/
public void responseReady() {
// 读取操作已经结束，释放分配的读取空间，准备下一轮的操作
readBufferBytesAllocated.addAndGet(-buffer_.array().length);
if (response_.len() == 0) {
// 如果是单工通讯方式，可以直接转换到：等待被转换为读状态
state_ = FrameBufferState.AWAITING_REGISTER_READ;
buffer_ = null;
} else {
buffer_ = ByteBuffer.wrap(response_.get(), 0, response_.len());
// 将状态切换为：等待被转换为写状态
state_ = FrameBufferState.AWAITING_REGISTER_WRITE;
}
// 状态转换
requestSelectInterestChange();
}
/**
* 调用真实的服务实现
*/
public void invoke() {
TTransport inTrans = getInputTransport();
TProtocol inProt = inputProtocolFactory_.getProtocol(inTrans);
TProtocol outProt = outputProtocolFactory_.getProtocol(getOutputTransport());
try {
processorFactory_.getProcessor(inTrans).process(inProt, outProt);
responseReady();
return;
} catch (TException te) {
LOGGER.warn("Exception while invoking!", te);
} catch (Throwable t) {
LOGGER.error("Unexpected throwable while invoking!", t);
}
// This will only be reached when there is a throwable.
state_ = FrameBufferState.AWAITING_CLOSE;
requestSelectInterestChange();
}
/**
* 从传输层读取数据
*/
private boolean internalRead() {
try {
if (trans_.read(buffer_) < 0) {
return false;
}
return true;
} catch (IOException e) {
LOGGER.warn("Got an IOException in internalRead!", e);
return false;
}
}
/**
* 写结束了，状态切换为读取状态
*/
private void prepareRead() {
// 我们不能直接修改selectionKey的状态，因为目前在select线程中
selectionKey_.interestOps(SelectionKey.OP_READ);
// 预先分配4字节消息头空间
buffer_ = ByteBuffer.allocate(4);
// 状态置为读取消息头，说明该FrameBuffer可以读取新消息了
state_ = FrameBufferState.READING_FRAME_SIZE;
}
通过上面的状态转换，一个FrameBuffer可以在一次调用中反复被复用。
1.4 TNonBlockingServer
protected boolean startThreads() {
// start the selector
try {
// 开启一个selector线程用于进行IO事件接收
selectAcceptThread_ = new SelectAcceptThread((TNonblockingServerTransport)serverTransport_);
stopped_ = false;
selectAcceptThread_.start();
return true;
} catch (IOException e) {
LOGGER.error("Failed to start selector thread!", e);
return false;
}
}
我们看下select线程的工作原理
/**
* 循环调用select
*/
public void run() {
try {
while (!stopped_) {
select();
processInterestChanges();
}
for (SelectionKey selectionKey : selector.keys()) {
cleanupSelectionKey(selectionKey);
}
} catch (Throwable t) {
LOGGER.error("run() exiting due to uncaught error", t);
} finally {
stopped_ = true;
}
}
继续看select方法：
private void select() {
try {
// 阻塞等待IO事件
selector.select();
// 当IO事件到来时，处理IO事件
Iterator<SelectionKey> selectedKeys = selector.selectedKeys().iterator();
while (!stopped_ && selectedKeys.hasNext()) {
SelectionKey key = selectedKeys.next();
selectedKeys.remove();
// skip if not valid
if (!key.isValid()) {
cleanupSelectionKey(key);
continue;
}
// 接收新连接
if (key.isAcceptable()) {
handleAccept();
} else if (key.isReadable()) {
// 处理读事件
handleRead(key);
} else if (key.isWritable()) {
// 处理写事件
handleWrite(key);
} else {
LOGGER.warn("Unexpected state in select! " + key.interestOps());
}
}
} catch (IOException e) {
LOGGER.warn("Got an IOException while selecting!", e);
}
}
private void handleAccept() throws IOException {
SelectionKey clientKey = null;
TNonblockingTransport client = null;
try {
// 接收连接
client = (TNonblockingTransport)serverTransport.accept();
在selector中注册client
clientKey = client.registerSelector(selector, SelectionKey.OP_READ);
// 创建FrameBuffer并注册
FrameBuffer frameBuffer = new FrameBuffer(client, clientKey,
SelectAcceptThread.this);
clientKey.attach(frameBuffer);
} catch (TTransportException tte) {
// something went wrong accepting.
LOGGER.warn("Exception trying to accept!", tte);
tte.printStackTrace();
if (clientKey != null) cleanupSelectionKey(clientKey);
if (client != null) client.close();
}
}
1.5 THsHaServer
THsHaServer其他原理与NonBlockingServer相同，仅仅是提供了一个新的线程池来处理业务逻辑：
/**
* 重写该方法，创建一个新的线程来执行真实服务的调用并交给线程池管理
*/
@Override
protected boolean requestInvoke(FrameBuffer frameBuffer) {
try {
Runnable invocation = getRunnable(frameBuffer);
invoker.execute(invocation);
return true;
} catch (RejectedExecutionException rx) {
LOGGER.warn("ExecutorService rejected execution!", rx);
return false;
}
}
1.6 TThreadedSelectorServer
/**
* 创建接收线程和selector线程组
*/
@Override
protected boolean startThreads() {
try {
// 循环创建selector线程
for (int i = 0; i < args.selectorThreads; ++i) {
selectorThreads.add(new SelectorThread(args.acceptQueueSizePerThread));
}
// 创建接收线程
acceptThread = new AcceptThread((TNonblockingServerTransport) serverTransport_,
createSelectorThreadLoadBalancer(selectorThreads));
stopped_ = false;
// 开启selector线程
for (SelectorThread thread : selectorThreads) {
thread.start();
}
// 开启接收线程
acceptThread.start();
return true;
} catch (IOException e) {
LOGGER.error("Failed to start threads!", e);
return false;
}
}
TThreadedSelectorServer与THsHaServer一样，同样提供了一个线程池来执行业务逻辑：
@Override
protected boolean requestInvoke(FrameBuffer frameBuffer) {
Runnable invocation = getRunnable(frameBuffer);
if (invoker != null) {
try {
invoker.execute(invocation);
return true;
} catch (RejectedExecutionException rx) {
LOGGER.warn("ExecutorService rejected execution!", rx);
return false;
}
} else {
// Invoke on the caller's thread
invocation.run();
return true;
}
}
selector线程与TNonBlockingServer相比仅缺少了socket连接的逻辑，其余实现相同，在此不再赘述。
我们看下接收线程的run方法：
public void run() {
try {
// 循环调用select方法
while (!stopped_) {
select();
}
} catch (Throwable t) {
LOGGER.error("run() exiting due to uncaught error", t);
} finally {
// This will wake up the selector threads
TThreadedSelectorServer.this.stop();
}
}
select方法：
private void select() {
try {
// 阻塞等待socket连接
acceptSelector.select();
// 处理连接事件
Iterator<SelectionKey> selectedKeys = acceptSelector.selectedKeys().iterator();
while (!stopped_ && selectedKeys.hasNext()) {
SelectionKey key = selectedKeys.next();
selectedKeys.remove();
// skip if not valid
if (!key.isValid()) {
continue;
}
if (key.isAcceptable()) {
// 连接
handleAccept();
} else {
LOGGER.warn("Unexpected state in select! " + key.interestOps());
}
}
} catch (IOException e) {
LOGGER.warn("Got an IOException while selecting!", e);
}
}
handleAccept方法：
/**
* Accept a new connection.
*/
private void handleAccept() {
final TNonblockingTransport client = doAccept();
if (client != null) {
// Pass this connection to a selector thread
final SelectorThread targetThread = threadChooser.nextThread();
// 如果接收策略是迅速接收，直接使用该线程去添加到队列中
if (args.acceptPolicy == Args.AcceptPolicy.FAST_ACCEPT || invoker == null) {
doAddAccept(targetThread, client);
} else {
// 如果接收策略是公平接收，则交给一个线程去执行连接接收工作
try {
invoker.submit(new Runnable() {
public void run() {
doAddAccept(targetThread, client);
}
});
} catch (RejectedExecutionException rx) {
LOGGER.warn("ExecutorService rejected accept registration!", rx);
// close immediately
client.close();
}
}
}
}
2. 客户端
客户端的代码我们之前在客户端的处理层中介绍过，详细请看这篇文章：https://blog.csdn.net/qq_21399231/article/details/106224758
3. 总结
本篇文章介绍了thrift的服务层，重点介绍了服务端中thrift提供的各个服务模型。到此为止，thrift的源码学习介绍完了，希望对大家有所帮助
扫一扫
4936
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
