阻塞IO下的echo实验

代码分析

echo代码位置：recipes-master/tpc/echo.cc echo_client.cc
服务端代码

int main(int argc, char* argv[])
{
  InetAddress listenAddr(3007);
  Acceptor acceptor(listenAddr);
  printf("Accepting... Ctrl-C to exit\n");
  int count = 0;
  bool nodelay = argc > 1 && strcmp(argv[1], "-D") == 0;
  while (true)
  {
    TcpStreamPtr tcpStream = acceptor.accept();
    printf("accepted no. %d client\n", ++count);
    if (nodelay)
      tcpStream->setTcpNoDelay(true);
	// 一个线程处理一个客户端连接
    // C++11 doesn't allow capturing unique_ptr in lambda, C++14 allows.
    std::thread thr([count] (TcpStreamPtr stream) {
      printf("thread for no. %d client started.\n", count);
      char buf[4096];
      int nr = 0;
      while ( (nr = stream->receiveSome(buf, sizeof(buf))) > 0)
      {
        int nw = stream->sendAll(buf, nr);
        if (nw < nr)
        {
          break;
        }
      }
      printf("thread for no. %d client ended.\n", count);
    }, std::move(tcpStream));
    thr.detach();
  }
}

服务端的逻辑主要就是，建立连接，每次读取4096个字节，然后这些数据再返回给客户端。

客户端

int main(int argc, const char* argv[])
{
  if (argc < 3)
  {
    printf("Usage: %s hostname message_length [scp]\n", argv[0]);
    return 0;
  }

  const int len = atoi(argv[2]);

  InetAddress addr(3007);
  if (!InetAddress::resolve(argv[1], &addr))
  {
    printf("Unable to resolve %s\n", argv[1]);
    return 0;
  }

  printf("connecting to %s\n", addr.toIpPort().c_str());
  TcpStreamPtr stream(TcpStream::connect(addr));
  if (!stream)
  {
    printf("Unable to connect %s\n", addr.toIpPort().c_str());
    perror("");
    return 0;
  }

  printf("connected, sending %d bytes\n", len);
	// 发送数据
  std::string message(len, 'S');
  int nw = stream->sendAll(message.c_str(), message.size());
  printf("sent %d bytes\n", nw);

  if (argc > 3)
  {
    for (char cmd : std::string(argv[3]))
    {
      if (cmd == 's')  // shutdown
      {
        printf("shutdown write\n");
        stream->shutdownWrite();
      }
      else if (cmd == 'p') // pause
      {
        printf("sleeping for 10 seconds\n");
        ::sleep(10);
        printf("done\n");
      }
      else if (cmd == 'c') // close
      {
        printf("close without reading response\n");
        return 0;
      }
      else
      {
        printf("unknown command '%c'\n", cmd);
      }
    }
  }
	// 返回数据
  std::vector<char> receive(len);
  int nr = stream->receiveAll(receive.data(), receive.size());
  printf("received %d bytes\n", nr);
  if (nr != nw)
  {
    printf("!!! Incomplete response !!!\n");
  }
}

客户端的逻辑主要就是，建立连接，发送数据，数据发送完再接收读取返回的数据。

测试

客户端在测试20m的数据时出现了阻塞，此时服务端也阻塞当中

[wang@localhost tpc]$ ./echo_client ip 2048000
connecting to 192.168.1.104:3007
connected, sending 2048000 bytes
# ... 客户端阻塞

[wang@localhost tpc]$ ./echo
Accepting... Ctrl-C to exit
accepted no. 1 client
thread for no. 1 client started.
# ... 服务端阻塞

排查问题

查看Recv-Q和Send-Q

服务端

[wang@localhost ~]$ netstat -tpn | grep '3007\|^[AP]'
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp   536080 260640 192.168.1.104:3007      192.168.1.105:56412     ESTABLISHED 5985/./ech

客户端

[wang@localhost ~]$ netstat -tpn | grep '3007\|^[AP]'
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp   261264 307992 192.168.1.105:56412     192.168.1.104:3007      ESTABLISHED 12526/./echo_client 

可以看见不管客户端还是服务端，Send-Q上都有大量的缓存数据，可以看出send发生阻塞。

分析原因

服务端：由于服务端在每次recv4K字节后需要send数据给客户端，但此时客户端正在send数据的过程中，没有recv数据，导致服务端send阻塞
客户端：由于服务端阻塞在send中，没有继续recv数据，导致客户端也陷入send阻塞当中

解决方案

我们在设计协议的时候，可以先让客户端发一个header，告诉服务端数据的长度，然后服务端在接收到header后，准备好一个符合大小的buffer，将数据全部读取到buffer当中，之后再返回数据给客户端。