网络层
《kafka源码刨析》读书笔记
客户端一般不会碰到大量连接请求,但是broker会遇到,因为它不止收到提供者/消费者的连接请求,还需要收到集群中其他broker的请求,所以对于服务端来说,面临的是高并发场景,客户端使用NetworkClient来管理连接就足够了,server是不行的,它采用reactor模式实现网络层。
1.Reactor模式
reactor模式,基于事件驱动的模式,java nio也是提供了reactor模式
常见的单线程java nio模式
为了提高效率,kafka 采用多个线程来处理请求,同时也提高pc机cpu的效率。此处把read和write改为了线程池来处理,提高处理速度。
但是,当一个时刻出现大量的i/o实践,那么单线程的selector就无法处理,所以后期还进行独写分离,使用多个selector来接收事件请求。
2.SocketServer
kafka采用多线程模式,使用多个selector的,核心类是SockerServer,其中包含一个Acceptor用于接收并处理所有新的连接,每个Acceptor对应多个Processor线程,每个Processor线程拥有自己的selector,主要用于从连接中读取/写回响应。而processor对应多个Handler,用来处理request返回response,processor和handler使用requestChannel来连接,
我们看一下socketServer的核心字段:注意scala的变量写法,变量名在前,变量类型在后面。
private val endpoints = config.listeners.map(l => l.listenerName -> l).toMap //Endpoint集合
private val numProcessorThreads = config.numNetworkThreads //processor线程的个数
private val maxQueuedRequests = config.queuedMaxRequests //RequestChannel的requestQueue中缓冲的最大请求个数
private val totalProcessorThreads = numProcessorThreads * endpoints.size //processor线程的总个数 也就是:numProcessortThreads*endpoints.size
private val maxConnectionsPerIp = config.maxConnectionsPerIp //每个ip上能创建的最大连接数
private val maxConnectionsPerIpOverrides = config.maxConnectionsPerIpOverrides //具体指定某ip上最大的连接数
private val logContext = new LogContext(s"[SocketServer brokerId=${config.brokerId}] ")
this.logIdent = logContext.logPrefix
private val memoryPoolSensor = metrics.sensor("MemoryPoolUtilization")
private val memoryPoolDepletedPercentMetricName = metrics.metricName("MemoryPoolAvgDepletedPercent", "socket-server-metrics")
private val memoryPoolDepletedTimeMetricName = metrics.metricName("MemoryPoolDepletedTimeTotal", "socket-server-metrics")
memoryPoolSensor.add(new Meter(TimeUnit.MILLISECONDS, memoryPoolDepletedPercentMetricName, memoryPoolDepletedTimeMetricName))
private val memoryPool = if (config.queuedMaxBytes > 0) new SimpleMemoryPool(config.queuedMaxBytes, config.socketRequestMaxBytes, false, memoryPoolSensor) else MemoryPool.NONE
val requestChannel = new RequestChannel(totalProcessorThreads, maxQueuedRequests) //processor线程与handler线程之间交换数据的队列
private val processors = new Array[Processor](totalProcessorThreads) //processor线程集合,包含所有endpoint对应的processor线程
private[network] val acceptors = mutable.Map[EndPoint, Acceptor]() //Acceptor对象集合,每个endpoint对应一个Acceptor对象
private var connectionQuotas: ConnectionQuotas = _ //提供控制每个ip上的最大连接数的功能,底层是一个map,记录每个ip的连接数
3.AbstractServerThread
acceptor和processor都继承了AbstractServerThread类,它的方法,主要用来操作上面的字段,控制线程的阻塞和唤醒。
private val startupLatch = new CountDownLatch(1) //标识当前线程的startup操作是否完成
@volatile private var shutdownLatch = new CountDownLatch(0) //标识当前线程的shutdown操作是否完成
private val alive = new AtomicBoolean(true) //标识当前线程是否存活,初始值为true
4.Acceptor
主要功能是接收客户端建立连接的请求,创建socket连接并分配给processor处理,有两个重要字段,一个是java nio selector,第二个是接收客户端请求的serverSocketChannel对象。
private val nioSelector = NSelector.open()
val serverChannel = openServerSocket(endPoint.host, endPoint.port)
this.synchronized { //为每个processor都创建对应的线程,并启动
processors.foreach { processor =>
KafkaThread.nonDaemon(s"kafka-network-thread-$brokerId-${endPoint.listenerName}-${endPoint.securityProtocol}-${processor.id}",
processor).start()
}
}
它的核心逻辑,处理accept事件
def run() {
serverChannel.register(nioSelector, SelectionKey.OP_ACCEPT) //注册accept事件
startupComplete() //标识当前线程启动操作已经完成
try {
var currentProcessor = 0
while (isRunning) { //检测线程运行状态
try {
val ready = nioSelector.select(500) //等待关注的事件(accept事件)
if (ready > 0) {
val keys = nioSelector.selectedKeys()
val iter = keys.iterator()
while (iter.hasNext && isRunning) {
try {
val key = iter.next
iter.remove()
if (key.isAcceptable)
accept(key, processors(currentProcessor)) //调用accept()方法处理accept事件
else //如果不是accept事件,则报错
throw new IllegalStateException("Unrecognized key state for acceptor thread.")
// round robin to the next processor thread
//更新currentProcessor,使用round-robin方式选择
currentProcessor = (currentProcessor + 1) % processors.length
...
}
深入来到accept()
def accept(key: SelectionKey, processor: Processor) {
val serverSocketChannel = key.channel().asInstanceOf[ServerSocketChannel]
val socketChannel = serverSocketChannel.accept()
try { //增加ConnectionQuotas中记录的连接数
connectionQuotas.inc(socketChannel.socket().getInetAddress)
//配置socketChannel的相关属性
socketChannel.configureBlocking(false)
socketChannel.socket().setTcpNoDelay(true)
socketChannel.socket().setKeepAlive(true)
if (sendBufferSize != Selectable.USE_DEFAULT_BUFFER_SIZE)
socketChannel.socket().setSendBufferSize(sendBufferSize)
//将socketChannel交给processor处理
processor.accept(socketChannel)
}...
}
这里一共三个功能,增加ConnectionQuotas中的连接数,第二个配置socketChannel,第三个就是交给processor,我们在下一小节中处理。
5.Processor
主要用来完成读取请求和写回响应的操作,processor不参与具体业务逻辑的处理。
主要字段:
private val newConnections = new ConcurrentLinkedQueue[SocketChannel]() //保存了由此processor处理的socketChannel
private val inflightResponses = mutable.Map[String, RequestChannel.Response]() //保存待发送的响应
//kafka selector 复杂管理网络连接,
private val selector = createSelector(
ChannelBuilders.serverChannelBuilder(listenerName, securityProtocol, config, credentialProvider.credentialCache))
requestChannel: RequestChannel, //processor与handler线程之间传递数据的队列
processor收到acceptor发出的accept()方法,来到processor处理
def accept(socketChannel: SocketChannel) {
newConnections.add(socketChannel) //保存socketChannel
wakeup()
}
processor 的run()方法:【核心】
override def run() {
startupComplete() //标识当前线程初始化流程已经结束,唤醒等待此processor初始化的线程
try {
while (isRunning) {
configureNewConnections() //1.配置socketchannel
processNewResponses() //2.处理response
poll() //3.缓冲下面三项内容
processCompletedReceives() //4.处理 request处理后的channel
processCompletedSends() //5.处理response完成的channel
processDisconnected() //6.处理断开连接的channel
...
}
5.1 configureNewConnections()
获取刚刚的socketchannel,将其注册到selector,这里有一个connectionId,就是根据address,port,等信息组成的字符串!
private def configureNewConnections() {
while (!newConnections.isEmpty) {
val channel = newConnections.poll() //获取新放入的socketChannel
selector.register(connectionId(channel.socket), channel) //每个socketchannel都要在nioSelector上注册read事件
}
connectionId(){
val localHost = socket.getLocalAddress.getHostAddress
val localPort = socket.getLocalPort
val remoteHost = socket.getInetAddress.getHostAddress
val remotePort = socket.getPort
val connId = ConnectionId(localHost, localPort, remoteHost, remotePort, nextConnectionIndex).toString
}
注册事件,默认就是read事件
public void register(String id, SocketChannel socketChannel) throws IOException {
SelectionKey key = socketChannel.register(nioSelector, SelectionKey.OP_READ); //read事件
buildChannel(socketChannel, id, key);
}
5.2 processNewResponses()
处理requestChannel对应的responseQueue队列,并处理其中缓冲的Response,注意这里是一个循环!
private def processNewResponses() {
var curr: RequestChannel.Response = null
while ({curr = requestChannel.receiveResponse(id); curr != null}) { //从responseQueues获取id对应的response响应
val channelId = curr.request.context.connectionId //找到对应的channel id,用来找kafkaChannel
try {
curr.responseAction match {
case RequestChannel.NoOpAction =>
// There is no response to send to the client, we need to read more pipelined requests
// that are sitting in the server's socket buffer
//如果是NOopAction类型的response,说明不需要返回client,直接给channel注册read事件继续工作
updateRequestMetrics(curr)
trace("Socket server received empty response to send, registering for read: " + curr)
openOrClosingChannel(channelId).foreach(c => selector.unmute(c.id))
case RequestChannel.SendAction =>
//如果是SendAction,需要返回到client,注册write事件,
val responseSend = curr.responseSend.getOrElse(
throw new IllegalStateException(s"responseSend must be defined for SendAction, response: $curr"))
sendResponse(curr, responseSend) //【入】 就在下面第一个方法
case RequestChannel.CloseConnectionAction =>
updateRequestMetrics(curr)
trace("Closing socket connection actively according to the response code.")
close(channelId) //关闭channel
}
}...
}
}
5.3 poll()
调用selector.poll()方法,功能:读取请求,发送响应,将请求,以及断开的连接放入其completeReceives,completedSends,disconnected队列中等待处理,
5.4 processCompletedReceives()
处理刚才缓冲的队列,
private def processCompletedReceives() {
//遍历 completedReceives
selector.completedReceives.asScala.foreach { receive =>
try {
openOrClosingChannel(receive.source) match {
case Some(channel) =>
val header = RequestHeader.parse(receive.payload)
val context = new RequestContext(header, receive.source, channel.socketAddress,
channel.principal, listenerName, securityProtocol)
//将 上方读取的内容,封装成request
val req = new RequestChannel.Request(processor = id, context = context,
startTimeNanos = time.nanoseconds, memoryPool, receive.payload, requestChannel.metrics)
//放入requestQueue中,等待handler队列处理
requestChannel.sendRequest(req)
//取消注册的read 事件,表示在发送响应之前,此连接不能再读取任何请求。
selector.mute(receive.source)
...
}
}
5.5 processCompletedSends()
private def processCompletedSends() {
selector.completedSends.asScala.foreach { send =>
try {
//首先将inflightResponse中保存的对应response删除,
val resp = inflightResponses.remove(send.destination)
//重新注册read事件,运行此连接继续读取数据
updateRequestMetrics(resp)
selector.unmute(send.destination)
...
}
5.6 processDisconnected()
处理断开连接的channel
private def processDisconnected() {
selector.disconnected.keySet.asScala.foreach { connectionId =>
try {
val remoteHost = ConnectionId.fromString(connectionId)
//删除对应的Response
inflightResponses.remove(connectionId).foreach(updateRequestMetrics)
// the channel has been closed by the selector but the quotas still need to be updated
//减少ConnectionQuotas中对应记录的连接数
connectionQuotas.dec(InetAddress.getByName(remoteHost))
...
}
//processor的线程个数 缓冲请求的最大个数,即requestQueues数组的长度
class RequestChannel(val numProcessors: Int, val queueSize: Int) extends KafkaMetricsGroup {
val metrics = new RequestChannel.Metrics
private var responseListeners: List[(Int) => Unit] = Nil //监听器列表,handler向responseQueue存放响应时唤醒对应的processor线程
private val requestQueue = new ArrayBlockingQueue[BaseRequest](queueSize) //存放processor发来的请求
private val responseQueues = new Array[BlockingQueue[RequestChannel.Response]](numProcessors) //存放handler返回结果的队列
requestChannel提供了增删requestQueue,responseQueues集合以及Listener列表的方法
6.1 sendResponse()
添加的response类型,对应processor处理的三种类型。
def sendResponse(response: RequestChannel.Response) {
if (isTraceEnabled) {
val requestHeader = response.request.header
val message = response.responseAction match {
case SendAction =>
s"Sending ${requestHeader.apiKey} response to client ${requestHeader.clientId} of ${response.responseSend.get.size} bytes."
case NoOpAction =>
s"Not sending ${requestHeader.apiKey} response to client ${requestHeader.clientId} as it's not required."
case CloseConnectionAction =>
s"Closing connection for client ${requestHeader.clientId} due to error during ${requestHeader.apiKey}."
}
trace(message)
}
responseQueues(response.processor).put(response)
for(onResponse <- responseListeners) //调用监听器
onResponse(response.processor)
}