kafka源码---server之网络层

                                                           网络层

《kafka源码刨析》读书笔记   

   客户端一般不会碰到大量连接请求，但是broker会遇到，因为它不止收到提供者/消费者的连接请求，还需要收到集群中其他broker的请求，所以对于服务端来说，面临的是高并发场景，客户端使用NetworkClient来管理连接就足够了，server是不行的，它采用reactor模式实现网络层。

1.Reactor模式

    reactor模式，基于事件驱动的模式，java nio也是提供了reactor模式

常见的单线程java nio模式

 

为了提高效率，kafka 采用多个线程来处理请求，同时也提高pc机cpu的效率。此处把read和write改为了线程池来处理，提高处理速度。

 

但是，当一个时刻出现大量的i/o实践，那么单线程的selector就无法处理，所以后期还进行独写分离，使用多个selector来接收事件请求。

 

2.SocketServer

    kafka采用多线程模式，使用多个selector的，核心类是SockerServer,其中包含一个Acceptor用于接收并处理所有新的连接，每个Acceptor对应多个Processor线程，每个Processor线程拥有自己的selector，主要用于从连接中读取/写回响应。而processor对应多个Handler，用来处理request返回response，processor和handler使用requestChannel来连接，

我们看一下socketServer的核心字段：注意scala的变量写法，变量名在前，变量类型在后面。

 

private val endpoints = config.listeners.map(l => l.listenerName -> l).toMap  //Endpoint集合
private val numProcessorThreads = config.numNetworkThreads                   //processor线程的个数
private val maxQueuedRequests = config.queuedMaxRequests                    //RequestChannel的requestQueue中缓冲的最大请求个数
private val totalProcessorThreads = numProcessorThreads * endpoints.size    //processor线程的总个数 也就是：numProcessortThreads*endpoints.size

private val maxConnectionsPerIp = config.maxConnectionsPerIp                //每个ip上能创建的最大连接数
private val maxConnectionsPerIpOverrides = config.maxConnectionsPerIpOverrides   //具体指定某ip上最大的连接数

private val logContext = new LogContext(s"[SocketServer brokerId=${config.brokerId}] ")
this.logIdent = logContext.logPrefix

private val memoryPoolSensor = metrics.sensor("MemoryPoolUtilization")
private val memoryPoolDepletedPercentMetricName = metrics.metricName("MemoryPoolAvgDepletedPercent", "socket-server-metrics")
private val memoryPoolDepletedTimeMetricName = metrics.metricName("MemoryPoolDepletedTimeTotal", "socket-server-metrics")
memoryPoolSensor.add(new Meter(TimeUnit.MILLISECONDS, memoryPoolDepletedPercentMetricName, memoryPoolDepletedTimeMetricName))
private val memoryPool = if (config.queuedMaxBytes > 0) new SimpleMemoryPool(config.queuedMaxBytes, config.socketRequestMaxBytes, false, memoryPoolSensor) else MemoryPool.NONE
val requestChannel = new RequestChannel(totalProcessorThreads, maxQueuedRequests)     //processor线程与handler线程之间交换数据的队列
private val processors = new Array[Processor](totalProcessorThreads)          //processor线程集合，包含所有endpoint对应的processor线程

private[network] val acceptors = mutable.Map[EndPoint, Acceptor]()            //Acceptor对象集合，每个endpoint对应一个Acceptor对象
private var connectionQuotas: ConnectionQuotas = _                          //提供控制每个ip上的最大连接数的功能，底层是一个map，记录每个ip的连接数

3.AbstractServerThread

acceptor和processor都继承了AbstractServerThread类，它的方法，主要用来操作上面的字段，控制线程的阻塞和唤醒。

private val startupLatch = new CountDownLatch(1)   //标识当前线程的startup操作是否完成
@volatile private var shutdownLatch = new CountDownLatch(0)   //标识当前线程的shutdown操作是否完成
private val alive = new AtomicBoolean(true)   //标识当前线程是否存活，初始值为true

4.Acceptor

    主要功能是接收客户端建立连接的请求，创建socket连接并分配给processor处理，有两个重要字段，一个是java nio selector，第二个是接收客户端请求的serverSocketChannel对象。

private val nioSelector = NSelector.open()
val serverChannel = openServerSocket(endPoint.host, endPoint.port)

this.synchronized { //为每个processor都创建对应的线程，并启动
  processors.foreach { processor =>
    KafkaThread.nonDaemon(s"kafka-network-thread-$brokerId-${endPoint.listenerName}-${endPoint.securityProtocol}-${processor.id}",
      processor).start()
  }
}

它的核心逻辑，处理accept事件

def run() {
  serverChannel.register(nioSelector, SelectionKey.OP_ACCEPT) //注册accept事件
  startupComplete()  //标识当前线程启动操作已经完成
  try {
    var currentProcessor = 0
    while (isRunning) {  //检测线程运行状态
      try {
        val ready = nioSelector.select(500)  //等待关注的事件（accept事件）
        if (ready > 0) {
          val keys = nioSelector.selectedKeys()
          val iter = keys.iterator()
          while (iter.hasNext && isRunning) {
            try {
              val key = iter.next
              iter.remove()
              if (key.isAcceptable)
                accept(key, processors(currentProcessor)) //调用accept()方法处理accept事件
              else  //如果不是accept事件，则报错
                throw new IllegalStateException("Unrecognized key state for acceptor thread.")

              // round robin to the next processor thread
              //更新currentProcessor,使用round-robin方式选择
              currentProcessor = (currentProcessor + 1) % processors.length
     ...
}

深入来到accept()

def accept(key: SelectionKey, processor: Processor) {
  val serverSocketChannel = key.channel().asInstanceOf[ServerSocketChannel]
  val socketChannel = serverSocketChannel.accept()
  try { //增加ConnectionQuotas中记录的连接数
    connectionQuotas.inc(socketChannel.socket().getInetAddress)
    //配置socketChannel的相关属性
    socketChannel.configureBlocking(false)
    socketChannel.socket().setTcpNoDelay(true)
    socketChannel.socket().setKeepAlive(true)
    if (sendBufferSize != Selectable.USE_DEFAULT_BUFFER_SIZE)
      socketChannel.socket().setSendBufferSize(sendBufferSize)
    //将socketChannel交给processor处理
    processor.accept(socketChannel)
  }...
}

  这里一共三个功能，增加ConnectionQuotas中的连接数，第二个配置socketChannel，第三个就是交给processor，我们在下一小节中处理。

5.Processor

    主要用来完成读取请求和写回响应的操作，processor不参与具体业务逻辑的处理。

主要字段：

private val newConnections = new ConcurrentLinkedQueue[SocketChannel]()    //保存了由此processor处理的socketChannel
private val inflightResponses = mutable.Map[String, RequestChannel.Response]()  //保存待发送的响应
//kafka selector 复杂管理网络连接，
private val selector = createSelector(
    ChannelBuilders.serverChannelBuilder(listenerName, securityProtocol, config, credentialProvider.credentialCache))
requestChannel: RequestChannel,  //processor与handler线程之间传递数据的队列

processor收到acceptor发出的accept()方法，来到processor处理

def accept(socketChannel: SocketChannel) {
  newConnections.add(socketChannel)   //保存socketChannel
  wakeup()
}

processor 的run()方法：【核心】

override def run() {
  startupComplete()  //标识当前线程初始化流程已经结束，唤醒等待此processor初始化的线程
  try {
    while (isRunning) {
        configureNewConnections()  //1.配置socketchannel
        processNewResponses()      //2.处理response
        poll()                     //3.缓冲下面三项内容
        processCompletedReceives() //4.处理 request处理后的channel
        processCompletedSends()    //5.处理response完成的channel
        processDisconnected()      //6.处理断开连接的channel
    ...
}

5.1 configureNewConnections()

获取刚刚的socketchannel,将其注册到selector，这里有一个connectionId，就是根据address,port,等信息组成的字符串！

private def configureNewConnections() {
  while (!newConnections.isEmpty) {
    val channel = newConnections.poll()  //获取新放入的socketChannel
      selector.register(connectionId(channel.socket), channel) //每个socketchannel都要在nioSelector上注册read事件
}
connectionId(){
val localHost = socket.getLocalAddress.getHostAddress
val localPort = socket.getLocalPort
val remoteHost = socket.getInetAddress.getHostAddress
val remotePort = socket.getPort
val connId = ConnectionId(localHost, localPort, remoteHost, remotePort, nextConnectionIndex).toString
}

注册事件，默认就是read事件

public void register(String id, SocketChannel socketChannel) throws IOException {
    SelectionKey key = socketChannel.register(nioSelector, SelectionKey.OP_READ); //read事件
    buildChannel(socketChannel, id, key);
}

5.2 processNewResponses()

处理requestChannel对应的responseQueue队列，并处理其中缓冲的Response,注意这里是一个循环！

private def processNewResponses() {
  var curr: RequestChannel.Response = null
  while ({curr = requestChannel.receiveResponse(id); curr != null}) { //从responseQueues获取id对应的response响应
    val channelId = curr.request.context.connectionId   //找到对应的channel id,用来找kafkaChannel
    try {
      curr.responseAction match {
        case RequestChannel.NoOpAction =>    
          // There is no response to send to the client, we need to read more pipelined requests
          // that are sitting in the server's socket buffer
          //如果是NOopAction类型的response，说明不需要返回client，直接给channel注册read事件继续工作
          updateRequestMetrics(curr)
          trace("Socket server received empty response to send, registering for read: " + curr)
          openOrClosingChannel(channelId).foreach(c => selector.unmute(c.id))
        case RequestChannel.SendAction =>
          //如果是SendAction，需要返回到client,注册write事件，
          val responseSend = curr.responseSend.getOrElse(
            throw new IllegalStateException(s"responseSend must be defined for SendAction, response: $curr"))
          sendResponse(curr, responseSend) //【入】 就在下面第一个方法
        case RequestChannel.CloseConnectionAction =>
          updateRequestMetrics(curr)
          trace("Closing socket connection actively according to the response code.")
          close(channelId) //关闭channel
      }
    }...
  }
}

5.3 poll()

调用selector.poll()方法，功能：读取请求，发送响应，将请求，以及断开的连接放入其completeReceives,completedSends,disconnected队列中等待处理，

5.4 processCompletedReceives()

    处理刚才缓冲的队列，

private def processCompletedReceives() {
  //遍历 completedReceives
  selector.completedReceives.asScala.foreach { receive =>
    try {
      openOrClosingChannel(receive.source) match {
        case Some(channel) =>
          val header = RequestHeader.parse(receive.payload)
          val context = new RequestContext(header, receive.source, channel.socketAddress,
            channel.principal, listenerName, securityProtocol)
          //将 上方读取的内容，封装成request
          val req = new RequestChannel.Request(processor = id, context = context,
            startTimeNanos = time.nanoseconds, memoryPool, receive.payload, requestChannel.metrics)
          //放入requestQueue中，等待handler队列处理
          requestChannel.sendRequest(req)
          //取消注册的read 事件，表示在发送响应之前，此连接不能再读取任何请求。
          selector.mute(receive.source)
        ...
  }
}

5.5 processCompletedSends()

private def processCompletedSends() {
  selector.completedSends.asScala.foreach { send =>
    try {
      //首先将inflightResponse中保存的对应response删除，
      val resp = inflightResponses.remove(send.destination)
      //重新注册read事件，运行此连接继续读取数据
      updateRequestMetrics(resp)
      selector.unmute(send.destination)
   ...
}

5.6 processDisconnected()

处理断开连接的channel

private def processDisconnected() {
  selector.disconnected.keySet.asScala.foreach { connectionId =>
    try {
      val remoteHost = ConnectionId.fromString(connectionId)
      //删除对应的Response
      inflightResponses.remove(connectionId).foreach(updateRequestMetrics)
      // the channel has been closed by the selector but the quotas still need to be updated
      //减少ConnectionQuotas中对应记录的连接数
      connectionQuotas.dec(InetAddress.getByName(remoteHost))
    ...
}

6.RequestChannel

    processor线程与handler线程通过requestchannel完成数据传输，requestChannel中包含了一个requestQueue队列和多个responseQueues队列，每个processor对应一个responseQueue队列。processor将读取到的request放入requestQueue中，handler线程从requestQueue中取出请求进行处理，把结果放入processor对应的responseQueue中。processor线程从其对应的responseQueue中取出响应并发送给客户端。

              //processor的线程个数              缓冲请求的最大个数，即requestQueues数组的长度
class RequestChannel(val numProcessors: Int, val queueSize: Int) extends KafkaMetricsGroup {
  val metrics = new RequestChannel.Metrics
  private var responseListeners: List[(Int) => Unit] = Nil        //监听器列表，handler向responseQueue存放响应时唤醒对应的processor线程
  private val requestQueue = new ArrayBlockingQueue[BaseRequest](queueSize)         //存放processor发来的请求
  private val responseQueues = new Array[BlockingQueue[RequestChannel.Response]](numProcessors) //存放handler返回结果的队列

requestChannel提供了增删requestQueue,responseQueues集合以及Listener列表的方法

6.1 sendResponse()

添加的response类型，对应processor处理的三种类型。

def sendResponse(response: RequestChannel.Response) {
  if (isTraceEnabled) {
    val requestHeader = response.request.header
    val message = response.responseAction match { 
      case SendAction =>
        s"Sending ${requestHeader.apiKey} response to client ${requestHeader.clientId} of ${response.responseSend.get.size} bytes."
      case NoOpAction =>
        s"Not sending ${requestHeader.apiKey} response to client ${requestHeader.clientId} as it's not required."
      case CloseConnectionAction =>
        s"Closing connection for client ${requestHeader.clientId} due to error during ${requestHeader.apiKey}."
    }
    trace(message)
  }

  responseQueues(response.processor).put(response)
  for(onResponse <- responseListeners)  //调用监听器
    onResponse(response.processor)
}