上一节我们分析到Acceptor把接收到的请求给了不同的Processor,然而Processor只是把这些请求给缓存起来了。这些请求终归是需要处理的,接下来我们就看Processor是如何处理这些请求的?,首先从run()方法开始:
override def run() {
startupComplete()
while (isRunning) {
try {
// setup any new connections that have been queued up
//TODO 注册 OP_READ事件
configureNewConnections()
// register any new responses for writing
processNewResponses()
//TODO poll 处理读请求,把请求存入了stagedReceives,completedReceives
poll()
//TODO 处理已经接收到的请求
processCompletedReceives()
processCompletedSends()
processDisconnected()
} catch {
// We catch all the throwables here to prevent the processor thread from exiting. We do this because
// letting a processor exit might cause a bigger impact on the broker. Usually the exceptions thrown would
// be either associated with a specific socket channel or a bad request. We just ignore the bad socket channel
// or request. This behavior might need to be reviewed if we see an exception that need the entire broker to stop.
case e: ControlThrowable => throw e
case e: Throwable =>
error("Processor got uncaught exception.", e)
}
}
debug("Closing selector - processor " + id)
swallowError(closeAll())
shutdownComplete()
}
我们接着看 configureNewConnections() 方法是怎么注册OP_READ事件的:
/**
* Register any new connections that have been queued up
*/
private def configureNewConnections() {
//上一节我们将客户端过来的连接加入到队列newConnections中
//队列newConnections不为空的时候,不断迭代队列里面的连接
while (!newConnections.isEmpty) {
//TODO 从队列里面获取channel
val channel = newConnections.poll()
try {
debug(s"Processor $id listening to new connection from ${channel.socket.getRemoteSocketAddress}")
val localHost = channel.socket().getLocalAddress.getHostAddress
val localPort = channel.socket().getLocalPort
val remoteHost = channel.socket().getInetAddress.getHostAddress
val remotePort = channel.socket().getPort
val connectionId = ConnectionId(localHost, localPort, remoteHost, remotePort).toString
//TODO 把channel注册到Selector上,注册监听OP_READ事件
selector.register(connectionId, channel)
} catch {
// We explicitly catch all non fatal exceptions and close the socket to avoid a socket leak. The other
// throwables will be caught in processor and logged as uncaught exceptions.
case NonFatal(e) =>
val remoteAddress = channel.getRemoteAddress
// need to close the channel here to avoid a socket leak.
close(channel)
error(s"Processor $id closed connection from $remoteAddress", e)
}
}
}
接着看 selector.register(connectionId, channel),这里才是真正实现读取事件的注册,如下:
/**
* Register the nioSelector with an existing channel
* Use this on server-side, when a connection is accepted by a different thread but processed by the Selector
* Note that we are not checking if the connection id is valid - since the connection already exists
*/
public void register(String id, SocketChannel socketChannel) throws ClosedChannelException {
//TODO 真正的注册操作,注册了OP_READ事件,这样就可以读取发送过来的信息了。
SelectionKey key = socketChannel.register(nioSelector, SelectionKey.OP_READ);
KafkaChannel channel = channelBuilder.buildChannel(id, key, maxReceiveSize);
key.attach(channel);
//把channel放到channels
this.channels.put(id, channel);
}
我再看poll方法是怎么将连接请求存入了stagedReceives的?
private def poll() {
//TODO 调用Selector的poll方法
try selector.poll(300)
catch {
case e @ (_: IllegalStateException | _: IOException) =>
error(s"Closing processor $id due to illegal state or IO exception")
swallow(closeAll())
shutdownComplete()
throw e
}
}
点击selector.poll(300):
@Override
public void poll(long timeout) throws IOException {
if (timeout < 0)
throw new IllegalArgumentException("timeout should be >= 0");
clear();
if (hasStagedReceives() || !immediatelyConnectedKeys.isEmpty())
timeout = 0;
/* check ready keys */
long startSelect = time.nanoseconds();
//TODO 执行select 操作,返回key
int readyKeys = select(timeout);
long endSelect = time.nanoseconds();
this.sensors.selectTime.record(endSelect - startSelect, time.milliseconds());
if (readyKeys > 0 || !immediatelyConnectedKeys.isEmpty()) {
//TODO 对所有绑定在Selector上的key进行处理 (一个broker连接就对应一个key)
pollSelectionKeys(this.nioSelector.selectedKeys(), false, endSelect);
pollSelectionKeys(immediatelyConnectedKeys, true, endSelect);
}
addToCompletedReceives();
long endIo = time.nanoseconds();
this.sensors.ioTime.record(endIo - endSelect, time.milliseconds());
// we use the time at the end of select to ensure that we don't close any connections that
// have just been processed in pollSelectionKeys
maybeCloseOldestConnection(endSelect);
}
我们主要关注这个方法 pollSelectionKeys(),该方法在生产者源码中已经有详细的说明,我这里只看部分关键代码,如下:
private void pollSelectionKeys(Iterable<SelectionKey> selectionKeys,
boolean isImmediatelyConnected,
long currentTimeNanos) {
//TODO 遍历每一个SelectionKey
Iterator<SelectionKey> iterator = selectionKeys.iterator();
while (iterator.hasNext()) {
SelectionKey key = iterator.next();
iterator.remove();
//TODO 根据key获取到对应的KafkaChannel
KafkaChannel channel = channel(key);
// register all per-connection metrics at once
sensors.maybeRegisterConnectionMetrics(channel.id());
if (idleExpiryManager != null)
idleExpiryManager.update(channel.id(), currentTimeNanos);
try {
略 . . . . . .
/* if channel is ready read from any connections that have readable data */
//TODO 重要代码
//这儿处理的是NIO里读请求的事件,那很明显
//这个方法会在broker返回了一些服务端的响应的时候执行的。
//只有key绑定了READ事件才能执行这个方法
//但是目前key没有绑定READ事件
if (channel.ready() && key.isReadable() && !hasStagedReceive(channel)) {
NetworkReceive networkReceive;
//TODO 读取数据
// 这个地方读取响应的时候可以会读取到多个响应。所以里面的代码需要处理一个粘包的问题。
while ((networkReceive = channel.read()) != null)
addToStagedReceives(channel, networkReceive);
}
略 . . . . . .
} catch (Exception e) {
String desc = channel.socketDescription();
if (e instanceof IOException)
log.debug("Connection with {} disconnected", desc, e);
else
log.warn("Unexpected error from {}; closing connection", desc, e);
close(channel);
this.disconnected.add(channel.id());
}
}
}
继续点击 addToStagedReceives,发现方法将连接请求加入到 stagedReceives里面:
/**
* adds a receive to staged receives
*/
private void addToStagedReceives(KafkaChannel channel, NetworkReceive receive) {
if (!stagedReceives.containsKey(channel))
stagedReceives.put(channel, new ArrayDeque<NetworkReceive>());
Deque<NetworkReceive> deque = stagedReceives.get(channel);
deque.add(receive);
}
我们再回到Selector的poll方法,代码继续往下执行到 addToCompletedReceives,点击 addToCompletedReceives,该方法就是将请求加入到已完成队列,如下:
/**
* checks if there are any staged receives and adds to completedReceives
*/
private void addToCompletedReceives() {
//如果里面有数据
if (!this.stagedReceives.isEmpty()) {
//获取所有数据
Iterator<Map.Entry<KafkaChannel, Deque<NetworkReceive>>> iter = this.stagedReceives.entrySet().iterator();
//对数据进行遍历
while (iter.hasNext()) {
//每次遍历的是对应一个连接的数据。一个连接对应一个Deque
Map.Entry<KafkaChannel, Deque<NetworkReceive>> entry = iter.next();
KafkaChannel channel = entry.getKey();
if (!channel.isMute()) {
Deque<NetworkReceive> deque = entry.getValue();
//获取一个连接的一个Deque里面的第一条消息
NetworkReceive networkReceive = deque.poll();
//把消息加入到completedReceives集合里面
this.completedReceives.add(networkReceive);
this.sensors.recordBytesReceived(channel.id(), networkReceive.payload().limit());
//队列读完了以后就移除
if (deque.isEmpty())
iter.remove();
}
}
}
}
总结:我们发现Processor线程并没有直接处理请求,而是直接将newConnections队列的请求加入到stagedReceives队列再加入到completedReceives队列。