谈谈遇到的IncompleteRead

起因：最近通过HTTP的rest API 获取数据，数据格式为JSON字符串（大小在1MB ~ 10 MB 左右）

调用过程中，有时会出现如下的异常

IncompleteRead(57821 bytes read, 829570 more expected)
IncompleteRead(57821 bytes read, 582761 more expected)
IncompleteRead(57821 bytes read, 731578 more expected)

这个异常表示client端期待从HTTP body 中读取A 字节，但实际能够读取的字节数小于A个字节

HTTP Response 的Header 大致如下:

{'connection': 'keep-alive', 'content-type': 'application/json', 'content-length': '44668', 'content-encoding': 'gzip'}

可以判断Response body完整传输(safe transport)的要素是 content-length，也就是实际读到的body体的长度小于content_length

读到这里惊讶的发现

1. httplib 支持gzip （压缩格式文本的处理)，但是在httplib 中没有找到数据是在何处被unzip 的,找到的朋友告诉我一声，不甚感激。

2. IncompleteRead 异常都是在读到大概57821个字节时发生的，也找到少量是57820个字节。

3. IncompleteRead 不可重现，导数任务结束后2天，重新执行此脚本，发现跑多少遍，都没有再抛出异常

阅读httplib代码后，我推断出现抛出异常的代码片段如下

  def _safe_read(self, amt):
        """Read the number of bytes requested, compensating for partial reads.

        Normally, we have a blocking socket, but a read() can be interrupted
        by a signal (resulting in a partial read).

        Note that we cannot distinguish between EOF and an interrupt when zero
        bytes have been read. IncompleteRead() will be raised in this
        situation.

        This function should be used when <amt> bytes "should" be present for
        reading. If the bytes are truly not available (due to EOF), then the
        IncompleteRead exception can be used to detect the problem.
        """
        # NOTE(gps): As of svn r74426 socket._fileobject.read(x) will never
        # return less than x bytes unless EOF is encountered.  It now handles
        # signal interruptions (socket.error EINTR) internally.  This code
        # never caught that exception anyways.  It seems largely pointless.
        # self.fp.read(amt) will work fine.
        s = []
        while amt > 0:
            chunk = self.fp.read(min(amt, MAXAMOUNT))
            if not chunk:
                raise IncompleteRead(''.join(s), amt)
            s.append(chunk)
            amt -= len(chunk)
        return ''.join(s)

        Note that we cannot distinguish between EOF and an interrupt when zero
        bytes have been read. IncompleteRead() will be raised in this
        situation.

标红的注释提到如果收到EOF标示，或者中断都会导致IncompleteRead。

结论:

重试，异常没有重现这推断我们的代码应该是没有问题的。所能读取的字节长度基本固定，我推断有可能是某个硬件设备，网卡或者交换机路由器达到了服务能力上限，导致异常的中断，从而抛出IncompleteRead。只要在代码逻辑中加入重试机制即可。