本文用python在TCP的基础上实现一个HTTP客户端, 该客户端能够复用TCP连接, 使用HTTP1.1协议.
一. 创建HTTP请求
HTTP是基于TCP连接的, 它的请求报文格式如下:
因此, 我们只需要创建一个到服务器的TCP连接, 然后按照上面的格式写好报文并发给服务器, 就实现了一个HTTP请求.
在2018年发布的HTTP3.0版本中, TCP协议已经被QUIC代替. QUIC全称为Quick UDP Internet Connect, 是基于UDP实现的.
补充说明
1. HTTPConnection类
基于以上的分析, 我们首先定义一个HTTPConnection类来管理连接和请求内容:
classHTTPConnection:
default_port= 80_http_vsn= 11_http_vsn_str= 'HTTP/1.1'
def __init__(self, host: str, port: int = None) ->None:
self.sock=None
self._buffer=[]
self.host=host
self.port= port if port is not None elseself.default_port
self._state=_CS_IDLE
self._response=None
self._method=None
self.block_size= 8192
def _output(self, s: Union[str, bytes]) ->None:if hasattr(s, 'encode'):
s= s.encode('latin-1')
self._buffer.append(s)defconnect(self) -> None:
self.sock= socket.create_connection((self.host, self.port))
对于这个HTTPConnection对象, 我们只需要创建TCP连接, 然后按照HTTP协议的格式把请求数据写入buffer中, 最后把buffer中的数据发送出去就行了.
2. 编写请求行
请求行的内容比较简单, 就是说明请求方法, 请求路径和HTTP协议. 使用下面的方法来编写一个请求行:
def put_request(self, method: str, url: str) ->None:
self._method=method
url= url or '/'request= f'{method} {url} {self._http_vsn_str}'self._output(request)
3. 添加请求头
HTTP请求头和python的字典类似, 每行都是一个字段名与值的映射关系. HTTP协议并不要求设置所有合法的请求头的值, 我们只需要按照需要, 设置特定的请求头即可. 使用如下代码添加请求头:
def put_header(self, header: Union[bytes, str], value: Union[bytes, str, int]) ->None:if hasattr(header, 'encode'):
header= header.encode('ascii')if hasattr(value, 'encode'):
value= value.encode('latin-1')elifisinstance(value, int):
value= str(value).encode('ascii')
header= header + b':' +value
self._output(header)
此外, 在HTTP请求中, Host请求头字段是必须的, 否则网站可能会拒绝响应. 因此, 如果用户没有设置这个字段, 这里就应该主动把它加上去:
def _add_host(self, url: str) ->None:#所有HTTP / 1.1请求报文中必须包含一个Host头字段
#如果用户没给,就调用这个函数来生成
netloc = ''
if url.startswith('http'):
nil, netloc, nil, nil, nil=urllib.parse.urlsplit(url)ifnetloc:try:
netloc_enc= netloc.encode('ascii')exceptUnicodeEncodeError:
netloc_enc= netloc.encode('idna')
self.put_header('Host', netloc_enc)else:
host=self.host
port=self.porttry:
host_enc= host.encode('ascii')exceptUnicodeEncodeError:
host_enc= host.encode('idna')#对IPv6的地址进行额外处理
if host.find(':') >=0:
host_enc= b'[' + host_enc + b']'
if port ==self.default_port:
self.put_header('Host', host_enc)else:
host_enc= host_enc.decode('ascii')
self.put_header('Host', f'{host_enc}:{port}')
4. 发送请求正文
我们接受两种形式的body数据: 一个基于io.IOBase的可读文件对象, 或者是一个能通过迭代得到数据的对象. 在传输数据之前, 我们首先要确定数据是否采用分块传输:
def request(self, method: str, url: str, headers: dict = None, body: Union[io.IOBase, Iterable] =None,
encode_chunked: bool= False) ->None:
...if 'content-length' not inheader_names:if 'transfer-encoding' not inheader_names:
encode_chunked=False
content_length=self._get_content_length(body, method)if content_length isNone:if body is notNone:#在这种情况下, body一般是个生成器或者可读文件之类的东西,应该分块传输
encode_chunked =True
self.put_header('Transfer-Encoding', 'chunked')else:
self.put_header('Content-Length', str(content_length))else:#如果设置了transfer-encoding,则根据用户给的encode_chunked参数决定是否分块
pass
else:#只要给了content-length,那么一定不是分块传输
encode_chunked =False
...
@staticmethoddef _get_content_length(body: Union[str, bytes, bytearray, Iterable, io.IOBase], method: str) ->Optional[int]:if body isNone:#PUT,POST,PATCH三个方法默认是有body的
if method.upper() in_METHODS_EXPECTING_BODY:return0else:returnNoneif hasattr(body, 'read'):returnNonetry:#对于bytes或者bytearray格式的数据,通过memoryview获取它的长度
returnmemoryview(body).nbytesexceptTypeError:pass
ifisinstance(body, str):returnlen(body)return None
在确定了是否分块之后, 就可以把正文发出去了. 如果body是一个可读文件的话, 就调用_read_readable方法把它封装为一个生成器:
def _send_body(self, message_body: Union[str, bytes, bytearray, Iterable, io.IOBase], encode_chunked: bool) ->None:if hasattr(message_body, 'read'):
chunks=self._read_readable(message_body)else:try:
memoryview(message_body)exceptTypeError:try:
chunks=iter(message_body)exceptTypeError:raiseTypeError(
f'message_body should be a bytes-like object or an iterable, got {repr(type(message_body))}')else:#如果是字节类型的,通过一次迭代把它发出去
chunks =(message_body,)for chunk inchunks:if notchunk:continue
ifencode_chunked:
chunk= f'{len(chunk):X}\r\n'.encode('ascii') + chunk + b'\r\n'self.send(chunk)ifencode_chunked:
self.send(b'0\r\n\r\n')def _read_readable(self, readable: io.IOBase) ->Generator[bytes, None, None]:
need_encode=Falseifisinstance(readable, io.TextIOBase):
need_encode=TruewhileTrue:
data_block=readable.read(self.block_size)if notdata_block:break
ifneed_encode:
data_block= data_block.encode('utf-8')yield data_block
二. 获取响应数据
HTTP响应报文的格式与请求报文大同小异, 它大致是这样的:
因此, 我们只要用HTTPConnection的socket对象读取服务器发送的数据, 然后按照上面的格式对数据进行解析就行了.
1. HTTPResponse类
我们首先定义一个简单的HTTPResponse类. 它的属性大致上就是socket的文件对象以及一些请求的信息等等, 调用它的begin方法来解析响应行和响应头的数据, 然后调用read方法读取响应正文:
classHTTPResponse:def __init__(self, sock: socket.socket, method: str = None) ->None:
self.fp= sock.makefile('rb')
self._method=method
self.headers=None
self.version=_UNKNOWN
self.status=_UNKNOWN
self.reason=_UNKNOWN
self.chunked=_UNKNOWN
self.chunk_left=_UNKNOWN
self.length=_UNKNOWN
self.will_close=_UNKNOWNdef begin(self) ->None:
...def read(self, amount: int = None) ->bytes:
...
2. 解析状态行
状态行的解析比较简单, 我们只需要读取响应的第一行数据, 然后把它解析为HTTP协议版本,状态码和原因短语三部分就行了:
def _read_status(self) ->Tuple[str, int, str]:
line= str(self._read_line(), 'latin-1')if notline:raise RemoteDisconnected('Remote end closed connection without response')try:
version, status, reason= line.split(None, 2)exceptValueError:#reason只是给人看的, 一般和status对应, 所以它有可能不存在
try:
version, status= line.split(None, 1)
reason= ''
exceptValueError:
version, status, reason= '', '', ''
if not version.startswith('HTTP/'):
self._close_conn()raiseBadStatusLine(line)try:
status=int(status)if status < 100 or status > 999:raiseBadStatusLine(line)exceptValueError:raiseBadStatusLine(line)return version, status, reason.strip()
如果状态码为100, 则客户端需要解析多个响应状态行. 它的原理是这样的: 在请求数据过大的时候, 有的客户端会先不发送请求数据, 而是先在header中添加一个Expect: 100-continue, 如果服务器愿意接收数据, 会返回100的状态码, 这时候客户端再把数据发过去. 因此, 如果读取到100的状态码, 那么后面往往还会收到一个正式的响应数据, 应该继续读取响应头. 这部分的代码如下:
def begin(self) ->None:whileTrue:
version, status, reason=self._read_status()if status !=HTTPStatus.CONTINUE:break
#跳过100状态码部分的响应头
whileTrue:
skip=self._read_line().strip()if notskip:breakself.status =status
self.reason=reasonif version in ('HTTP/1.0', 'HTTP/0.9'):
self.version= 10
elif version.startswith('HTTP/1.'):
self.version= 11
else:#HTTP2还没研究, 这里就不写了
raiseUnknownProtocol(version)
...
3. 解析响应头
解析响应头比响应行还要简单. 因为每个header字段占一行, 我们只需要一直调用read_line方法读取字段, 直到读完header为止就行了.
def _parse_header(self) ->None:
headers={}whileTrue:
line=self._read_line()if len(headers) >_MAX_HEADERS:raise HTTPException('got more than %d headers' %_MAX_HEADERS)if line in_EMPTY_LINE:breakline= line.decode('latin-1')
i= line.find(':')if i == -1:raiseBadHeaderLine(line)#这里默认没有重名的情况
key, value = line[:i].lower(), line[i + 1:].strip()headers[key]=value
self.headers= headers
4. 接收响应正文
在接收响应正文之前, 首先要确定它的传输方式和长度:
def _set_chunk(self) ->None:
transfer_encoding= self.get_header('transfer-encoding')if transfer_encoding and transfer_encoding.lower() == 'chunked':
self.chunked=True
self.chunk_left=Noneelse:
self.chunked=Falsedef _set_length(self) ->None:#首先要知道数据是否是分块传输的
if self.chunked ==_UNKNOWN:
self._set_chunk()#如果状态码是1xx或者204(无响应内容)或者304(使用上次缓存的内容),则没有响应正文
#如果这是个HEAD请求,那么也不能有响应正文
if (self.status == HTTPStatus.NO_CONTENT orself.status== HTTPStatus.NOT_MODIFIED or
100 <= self.status < 200 orself._method== 'HEAD'):
self.length=0returnlength= self.get_header('content-length')if length and notself.chunked:try:
self.length=int(length)exceptValueError:
self.length=Noneelse:if self.length <0:
self.length=Noneelse:
self.length= None
然后, 我们实现一个read方法, 从body中读取指定大小的数据:
def read(self, amount: int = None) ->bytes:ifself.is_closed():return b''
if self._method == 'HEAD':
self.close()return b''
if amount isNone:returnself._read_all()return self._read_amount(amount)
如果没有指定需要的数据大小, 就默认读取所有数据:
def _read_all(self) ->bytes:ifself.chunked:returnself._read_all_chunk()if self.length isNone:
s=self.fp.read()else:try:
s=self._read_bytes(self.length)exceptIncompleteRead:
self.close()raiseself.length=0
self.close()returnsdef _read_all_chunk(self) ->bytes:assert self.chunked !=_UNKNOWN
value=[]try:whileTrue:
chunk=self._read_chunk()if chunk isNone:breakvalue.append(chunk)return b''.join(value)exceptIncompleteRead:raise IncompleteRead(b''.join(value))def _read_chunk(self) ->Optional[bytes]:try:
chunk_size=self._read_chunk_size()exceptValueError:raise IncompleteRead(b'')if chunk_size ==0:
self._read_and_discard_trailer()
self.close()returnNone
chunk=self._read_bytes(chunk_size)#每块的结尾会有一个\r\n,这里把它读掉
self._read_bytes(2)returnchunkdef _read_chunk_size(self) ->int:
line= self._read_line(error_message='chunk size')
i= line.find(b';')if i >=0:
line=line[:i]try:return int(line, 16)exceptValueError:
self.close()raise
def _read_and_discard_trailer(self) ->None:#chunk的尾部可能会挂一些额外的信息,比如MD5值,过期时间等等,一般会在header中用trailer字段说明
#当chunk读完之后调用这个函数, 这些信息就先舍弃掉得了
whileTrue:
line= self._read_line(error_message='chunk size')if line in_EMPTY_LINE:break
否则的话, 就读取部分数据, 如果正好是分块数据的话, 就比较复杂了. 简单来说, 就是用bytearray制造一个所需大小的数组, 然后依次读取chunk把数据往里面填, 直到填满或者没数据为止. 然后用chunk_left记录下当前块剩余的量, 以便下次读取.
def _read_amount(self, amount: int) ->bytes:ifself.chunked:returnself._read_amount_chunk(amount)if isinstance(self.length, int) and amount >self.length:
amount=self.length
container=bytearray(amount)
n=self.fp.readinto(container)if not n andcontainer:#如果读不到字节了,也就可以关了
self.close()elif self.length is notNone:
self.length-=nif notself.length:
self.close()returnmemoryview(container)[:n].tobytes()def _read_amount_chunk(self, amount: int) ->bytes:#调用这个方法,读取amount大小的chunk类型数据,不足就全部读取
assert self.chunked !=_UNKNOWN
total_bytes=0
container=bytearray(amount)
mvb=memoryview(container)try:whileTrue:#mvb可以理解为容器的空的那一部分
#这里一直调用_full_readinto把数据填进去,让mvb越来越小,同时记录填入的量
#等没数据或者当前数据足够把mvb填满之后,跳出循环
chunk_left =self._get_chunk_left()if chunk_left isNone:break
if len(mvb) <=chunk_left:
n=self._full_readinto(mvb)
self.chunk_left= chunk_left -n
total_bytes+=nbreaktemp_mvb=mvb[:chunk_left]
n=self._full_readinto(temp_mvb)
mvb=mvb[n:]
total_bytes+=n
self.chunk_left=0exceptIncompleteRead:raiseIncompleteRead(bytes(container[:total_bytes]))returnmemoryview(container)[:total_bytes].tobytes()def _full_readinto(self, container: memoryview) ->int:#返回读取的量.如果没能读满,这个方法会报警
amount =len(container)
n=self.fp.readinto(container)if n Optional[int]:#如果当前块读了一半,那么直接返回self.chunk_left就行了
#否则,有三种情况
#1). chunk_left为None,说明body压根没开始读,于是返回当前这一整块的长度
#2). chunk_left为0,说明这块读完了,于是返回下一块的长度
#3). body数据读完了,返回None,顺便做好善后工作
chunk_left =self.chunk_leftif notchunk_left:if chunk_left ==0:#如果剩余零,说明上一块已经读完了,这里把\r\n读掉
#如果是None,就说明chunk压根没开始读
self._read_bytes(2)try:
chunk_left=self._read_chunk_size()exceptValueError:raise IncompleteRead(b'')if chunk_left ==0:
self._read_and_discard_trailer()
self.close()
chunk_left=None
self.chunk_left=chunk_leftreturn chunk_left
三. 复用TCP连接
HTTP通信本质上是基于TCP连接发送和接收HTTP请求和响应, 因此, 只要TCP连接不断开, 我们就可以继续用它进行HTTP请求, 这样就避免了创建和销毁TCP连接产生的消耗.
1. 判断连接是否会断开
在下面几种情况中, 服务端会自动断开连接:
HTTP协议小于1.1且没有在头部设置了keep-alive
HTTP协议大于等于1.1但是在头部设置了connection: close
数据没有分块传输, 也没有说明数据的长度, 这种情况下, 服务器一般会在发送完成后断开连接, 让客户端知道数据发完了
根据上面列出来的几种情况, 通过下面的代码来判断连接是否会断开:
def _check_close(self) ->bool:
conn= self.get_header('connection')if not self.chunked and self.length isNone:returnTrueif self.version == 11:if conn and 'close' inconn.lower():returnTruereturnFalseelse:if self.headers.get('keep-alive'):returnFalseif conn and 'keep-alive' inconn.lower():returnFalsereturn True
2. 正确地关闭HTTPResponse对象
由于TCP连接的复用, 一个HTTPConnection可以产生多个HTTPResponse对象, 而这些对象在同一个TCP连接上, 会共用这个连接的读缓冲区. 这就导致, 如果上一个HTTPResponse对象没有把它的那部分数据读完, 就会对下一个响应产生影响.
另一方面来看, 我们也需要及时地关闭与这个TCP关联的文件对象来避免占用资源. 因此, 我们定义如下的close方法关闭一个HTTPResponse对象:
def close(self) ->None:ifself.is_closed():returnfp=self.fp
self.fp=None
fp.close()def is_closed(self) ->bool:return self.fp is None
用户调用HTTPResponse对象的read方法, 把缓冲区数据读完之后, 就会自动调用close方法(具体实现见上一章的第四节: 读取响应数据这部分). 因此, 在获取下一个响应数据之前, 我们只需要调用这个对象的is_closed方法, 就能判断读缓冲区是否已经读完, 能否继续接收响应了.
3. HTTP请求的生命周期
不使用管道机制的话, 不同的HTTP请求必须按次序进行, 相互之间不能重叠. 基于这个原因, 我们为HTTPConnection对象设置IDLE, REQ_STARTED和REQ_SENT三种状态, 一个完整的请求应该经历这几种状态:
根据上面的流程, 对HTTPConnection中对应的方法进行修改:
def get_response(self) ->HTTPResponse:if self._response andself._response.is_closed():
self._response=Noneif self._state != _CS_REQ_SENT orself._response:raiseResponseNotReady(self._state)
response= HTTPResponse(self.sock, method=self._method)try:try:
response.begin()exceptConnectionError:
self.close()raise
assert response.will_close !=_UNKNOWN
self._state=_CS_IDLEifresponse.will_close:
self.close()else:
self._response=responsereturnresponseexceptException as _:
response.close()raise
def put_request(self, method: str, url: str) ->None:#调用这个函数开始新一轮的请求,它负责写好请求行输出到缓存里面去
#调用它的前提是当前处于空闲状态
#如果之前的response还在并且已结束,会自动把它消除掉
if self._response andself._response.is_closed():
self._response=Noneif self._state ==_CS_IDLE:
self._state=_CS_REQ_STARTEDelse:raiseCannotSendRequest(self._state)
...def put_header(self, header: Union[bytes, str], value: Union[bytes, str, int]) ->None:if self._state !=_CS_REQ_STARTED:raiseCannotSendHeader()
...def end_headers(self, message_body=None, encode_chunked=False) ->None:if self._state ==_CS_REQ_STARTED:
self._state=_CS_REQ_SENTelse:raiseCannotSendHeader()
...
需要注意的是, 如果第二个请求已经进入到获取响应的阶段了, 而上一个请求的响应还没关闭, 那么就应该直接报错, 否则读取到的会是上一个请求剩余的响应部分数据, 导致解析响应出现问题.
事实上, HTTP1.1开始支持管道化技术, 也就是一次提交多个HTTP请求, 然后等待响应, 而不是在接收到上一个请求的响应后, 才发送后面的请求.
基于这种处理模式, 管道化技术理论上可以减少IO时间的损耗, 提升效率, 不过, 需要服务端的支持, 而且会增加程序的复杂程度, 这里就不实现了.
补充说明
四. 总结
1. 完整代码
HTTPConnection的完整代码如下:
classHTTPConnection:
default_port= 80_http_vsn= 11_http_vsn_str= 'HTTP/1.1'
def __init__(self, host: str, port: int = None) ->None:
self.sock=None
self._buffer=[]
self.host=host
self.port= port if port is not None elseself.default_port
self._state=_CS_IDLE
self._response=None
self._method=None
self.block_size= 8192
def request(self, method: str, url: str, headers: dict = None, body: Union[io.IOBase, Iterable] =None,
encode_chunked: bool= False) ->None:
self.put_request(method, url)
headers= headers or{}
header_names= frozenset(k.lower() for k inheaders.keys())if 'host' not inheader_names:
self._add_host(url)if 'content-length' not inheader_names:if 'transfer-encoding' not inheader_names:
encode_chunked=False
content_length=self._get_content_length(body, method)if content_length isNone:if body is notNone:
encode_chunked=True
self.put_header('Transfer-Encoding', 'chunked')else:
self.put_header('Content-Length', str(content_length))else:#如果设置了transfer-encoding,则根据用户给的encode_chunked参数决定是否分块
pass
else:#只要给了content-length,那么一定不是分块传输
encode_chunked =Falsefor hdr, value inheaders.items():
self.put_header(hdr, value)ifisinstance(body, str):
body=_encode(body)
self.end_headers(body, encode_chunked=encode_chunked)def send(self, data: bytes) ->None:if self.sock isNone:
self.connect()
self.sock.sendall(data)def get_response(self) ->HTTPResponse:if self._response andself._response.is_closed():
self._response=Noneif self._state != _CS_REQ_SENT orself._response:raiseResponseNotReady(self._state)
response= HTTPResponse(self.sock, method=self._method)try:try:
response.begin()exceptConnectionError:
self.close()raise
assert response.will_close !=_UNKNOWN
self._state=_CS_IDLEifresponse.will_close:
self.close()else:
self._response=responsereturnresponseexceptException as _:
response.close()raise
def connect(self) ->None:
self.sock=socket.create_connection((self.host, self.port))def close(self) ->None:
self._state=_CS_IDLEtry:
sock=self.sockifsock:
self.sock=None
sock.close()finally:
response=self._responseifresponse:
self._response=None
response.close()def put_request(self, method: str, url: str) ->None:#调用这个函数开始新一轮的请求,它负责写好请求行输出到缓存里面去
#调用它的前提是当前处于空闲状态
#如果之前的response还在并且已结束,会自动把它消除掉
if self._response andself._response.is_closed():
self._response=Noneif self._state ==_CS_IDLE:
self._state=_CS_REQ_STARTEDelse:raiseCannotSendRequest(self._state)
self._method=method
url= url or '/'request= f'{method} {url} {self._http_vsn_str}'self._output(request)def put_header(self, header: Union[bytes, str], value: Union[bytes, str, int]) ->None:if self._state !=_CS_REQ_STARTED:raiseCannotSendHeader()if hasattr(header, 'encode'):
header= header.encode('ascii')if hasattr(value, 'encode'):
value= value.encode('latin-1')elifisinstance(value, int):
value= str(value).encode('ascii')
header= header + b':' +value
self._output(header)def end_headers(self, message_body=None, encode_chunked=False) ->None:if self._state ==_CS_REQ_STARTED:
self._state=_CS_REQ_SENTelse:raiseCannotSendHeader()
self._send_output(message_body, encode_chunked=encode_chunked)def _add_host(self, url: str) ->None:#所有HTTP / 1.1请求报文中必须包含一个Host头字段
#如果用户没给,就调用这个函数来生成
netloc = ''
if url.startswith('http'):
nil, netloc, nil, nil, nil=urlsplit(url)ifnetloc:try:
netloc_enc= netloc.encode('ascii')exceptUnicodeEncodeError:
netloc_enc= netloc.encode('idna')
self.put_header('Host', netloc_enc)else:
host=self.host
port=self.porttry:
host_enc= host.encode('ascii')exceptUnicodeEncodeError:
host_enc= host.encode('idna')#对IPv6的地址进行额外处理
if host.find(':') >=0:
host_enc= b'[' + host_enc + b']'
if port ==self.default_port:
self.put_header('Host', host_enc)else:
host_enc= host_enc.decode('ascii')
self.put_header('Host', f'{host_enc}:{port}')def _output(self, s: Union[str, bytes]) ->None:#将数据添加到缓冲区
if hasattr(s, 'encode'):
s= s.encode('latin-1')
self._buffer.append(s)def _send_output(self, message_body=None, encode_chunked=False) ->None:#发送并清空缓冲数据.然后,如果有请求正文,就也顺便发送
self._buffer.extend((b'', b''))
msg= b'\r\n'.join(self._buffer)
self._buffer.clear()
self.send(msg)if message_body is notNone:
self._send_body(message_body, encode_chunked)def _send_body(self, message_body: Union[bytes, str, bytearray, Iterable, io.IOBase], encode_chunked: bool) ->None:if hasattr(message_body, 'read'):
chunks=self._read_readable(message_body)else:try:
memoryview(message_body)exceptTypeError:try:
chunks=iter(message_body)exceptTypeError:raiseTypeError(
f'message_body should be a bytes-like object or an iterable, got {repr(type(message_body))}')else:#如果是字节类型的,通过一次迭代把它发出去
chunks =(message_body,)for chunk inchunks:if notchunk:continue
ifencode_chunked:
chunk= f'{len(chunk):X}\r\n'.encode('ascii') + chunk + b'\r\n'self.send(chunk)ifencode_chunked:
self.send(b'0\r\n\r\n')def _read_readable(self, readable: io.IOBase) ->Generator[bytes, None, None]:
need_encode=Falseifisinstance(readable, io.TextIOBase):
need_encode=TruewhileTrue:
data_block=readable.read(self.block_size)if notdata_block:break
ifneed_encode:
data_block= data_block.encode('utf-8')yielddata_block
@staticmethoddef _get_content_length(body: Union[str, bytes, bytearray, Iterable, io.IOBase], method: str) ->Optional[int]:if body isNone:#PUT,POST,PATCH三个方法默认是有body的
if method.upper() in_METHODS_EXPECTING_BODY:return0else:returnNoneif hasattr(body, 'read'):returnNonetry:#对于bytes或者bytearray格式的数据,通过memoryview获取它的长度
returnmemoryview(body).nbytesexceptTypeError:pass
ifisinstance(body, str):returnlen(body)return None
HTTPConnection
HTTPResponse的完整代码如下:
classHTTPResponse:def __init__(self, sock: socket.socket, method: str = None) ->None:
self.fp= sock.makefile('rb')
self._method=method
self.headers=None
self.version=_UNKNOWN
self.status=_UNKNOWN
self.reason=_UNKNOWN
self.chunked=_UNKNOWN
self.chunk_left=_UNKNOWN
self.length=_UNKNOWN
self.will_close=_UNKNOWNdef begin(self) ->None:if self.headers is notNone:returnself._parse_status_line()
self._parse_header()
self._set_chunk()
self._set_length()
self.will_close=self._check_close()def _read_line(self, limit: int = _MAX_LINE + 1, error_message: str = '') ->bytes:#注意,这个方法默认不去除line尾部的\r\n
line =self.fp.readline(limit)if len(line) >_MAX_LINE:raiseLineTooLong(error_message)returnlinedef _read_bytes(self, amount: int) ->bytes:
data=self.fp.read(amount)if len(data) None:whileTrue:
version, status, reason=self._read_status()if status !=HTTPStatus.CONTINUE:break
whileTrue:
skip= self._read_line(error_message='header line').strip()if notskip:breakself.status=status
self.reason=reasonif version in ('HTTP/1.0', 'HTTP/0.9'):
self.version= 10
elif version.startswith('HTTP/1.'):
self.version= 11
else:raiseUnknownProtocol(version)def _read_status(self) ->Tuple[str, int, str]:
line= str(self._read_line(error_message='status line'), 'latin-1')if notline:raise RemoteDisconnected('Remote end closed connection without response')try:
version, status, reason= line.split(None, 2)exceptValueError:#reason只是给人看的, 和status对应, 所以它有可能不存在
try:
version, status= line.split(None, 1)
reason= ''
exceptValueError:
version, status, reason= '', '', ''
if not version.startswith('HTTP/'):
self.close()raiseBadStatusLine(line)try:
status=int(status)if status < 100 or status > 999:raiseBadStatusLine(line)exceptValueError:raiseBadStatusLine(line)returnversion, status, reason.strip()def _parse_header(self) ->None:
headers={}whileTrue:
line= self._read_line(error_message='header line')if len(headers) >_MAX_HEADERS:raise HTTPException('got more than %d headers' %_MAX_HEADERS)if line in_EMPTY_LINE:breakline= line.decode('latin-1')
i= line.find(':')if i == -1:raiseBadHeaderLine(line)#这里默认没有重名的情况
key, value = line[:i].lower(), line[i + 1:].strip()
headers[key]=value
self.headers=headersdef _set_chunk(self) ->None:
transfer_encoding= self.get_header('transfer-encoding')if transfer_encoding and transfer_encoding.lower() == 'chunked':
self.chunked=True
self.chunk_left=Noneelse:
self.chunked=Falsedef _set_length(self) ->None:#首先要知道数据是否是分块传输的
if self.chunked ==_UNKNOWN:
self._set_chunk()#如果状态码是1xx或者204(无响应内容)或者304(使用上次缓存的内容),则没有响应正文
#如果这是个HEAD请求,那么也不能有响应正文
assertisinstance(self.status, int)if (self.status == HTTPStatus.NO_CONTENT orself.status== HTTPStatus.NOT_MODIFIED or
100 <= self.status < 200 orself._method== 'HEAD'):
self.length=0returnlength= self.get_header('content-length')if length and notself.chunked:try:
self.length=int(length)exceptValueError:
self.length=Noneelse:if self.length <0:
self.length=Noneelse:
self.length=Nonedef _check_close(self) ->bool:
conn= self.get_header('connection')if not self.chunked and self.length isNone:returnTrueif self.version == 11:if conn and 'close' inconn.lower():returnTruereturnFalseelse:if self.headers.get('keep-alive'):returnFalseif conn and 'keep-alive' inconn.lower():returnFalsereturnTruedef close(self) ->None:ifself.is_closed():returnfp=self.fp
self.fp=None
fp.close()def is_closed(self) ->bool:return self.fp isNonedef read(self, amount: int = None) ->bytes:ifself.is_closed():return b''
if self._method == 'HEAD':
self.close()return b''
if amount isNone:returnself._read_all()print(amount, amount isNone)returnself._read_amount(amount)def _read_all(self) ->bytes:ifself.chunked:returnself._read_all_chunk()if self.length isNone:
s=self.fp.read()else:try:
s=self._read_bytes(self.length)exceptIncompleteRead:
self.close()raiseself.length=0
self.close()returnsdef _read_all_chunk(self) ->bytes:assert self.chunked !=_UNKNOWN
value=[]try:whileTrue:
chunk=self._read_chunk()if chunk isNone:breakvalue.append(chunk)return b''.join(value)exceptIncompleteRead:raise IncompleteRead(b''.join(value))def _read_chunk(self) ->Optional[bytes]:try:
chunk_size=self._read_chunk_size()exceptValueError:raise IncompleteRead(b'')if chunk_size ==0:
self._read_and_discard_trailer()
self.close()returnNone
chunk=self._read_bytes(chunk_size)#每块的结尾会有一个\r\n,这里把它读掉
self._read_bytes(2)returnchunkdef _read_chunk_size(self) ->int:
line= self._read_line(error_message='chunk size')
i= line.find(b';')if i >=0:
line=line[:i]try:return int(line, 16)exceptValueError:
self.close()raise
def _read_and_discard_trailer(self) ->None:#chunk的尾部可能会挂一些额外的信息,比如MD5值,过期时间等等,一般会在header中用trailer字段说明
#当chunk读完之后调用这个函数, 这些信息就先舍弃掉得了
whileTrue:
line= self._read_line(error_message='chunk size')if line in_EMPTY_LINE:break
def _read_amount(self, amount: int) ->bytes:ifself.chunked:returnself._read_amount_chunk(amount)if isinstance(self.length, int) and amount >self.length:
amount=self.length
container=bytearray(amount)
n=self.fp.readinto(container)if not n andcontainer:#如果读不到字节了,也就可以关了
self.close()elif self.length is notNone:
self.length-=nif notself.length:
self.close()returnmemoryview(container)[:n].tobytes()def _read_amount_chunk(self, amount: int) ->bytes:#调用这个方法,读取amount大小的chunk类型数据,不足就全部读取
assert self.chunked !=_UNKNOWN
total_bytes=0
container=bytearray(amount)
mvb=memoryview(container)try:whileTrue:#mvb可以理解为容器的空的那一部分
#这里一直调用_full_readinto把数据填进去,让mvb越来越小,同时记录填入的量
#等没数据或者当前数据足够把mvb填满之后,跳出循环
chunk_left =self._get_chunk_left()if chunk_left isNone:break
if len(mvb) <=chunk_left:
n=self._full_readinto(mvb)
self.chunk_left= chunk_left -n
total_bytes+=nbreaktemp_mvb=mvb[:chunk_left]
n=self._full_readinto(temp_mvb)
mvb=mvb[n:]
total_bytes+=n
self.chunk_left=0exceptIncompleteRead:raiseIncompleteRead(bytes(container[:total_bytes]))returnmemoryview(container)[:total_bytes].tobytes()def _full_readinto(self, container: memoryview) ->int:#返回读取的量.如果没能读满,这个方法会报警
amount =len(container)
n=self.fp.readinto(container)if n Optional[int]:#如果当前块读了一半,那么直接返回self.chunk_left就行了
#否则,有三种情况
#1). chunk_left为None,说明body压根没开始读,于是返回当前这一整块的长度
#2). chunk_left为0,说明这块读完了,于是返回下一块的长度
#3). body数据读完了,返回None,顺便做好善后工作
chunk_left =self.chunk_leftif notchunk_left:if chunk_left ==0:#如果剩余零,说明上一块已经读完了,这里把\r\n读掉
#如果是None,就说明chunk压根没开始读
self._read_bytes(2)try:
chunk_left=self._read_chunk_size()exceptValueError:raise IncompleteRead(b'')if chunk_left ==0:
self._read_and_discard_trailer()
self.close()
chunk_left=None
self.chunk_left=chunk_leftreturnchunk_leftdef get_header(self, name, default: str = None) ->Optional[str]:if self.headers isNone:raiseResponseNotReady()returnself.headers.get(name, default)
@propertydef info(self) ->str:return repr(self.headers)
HTTPResponse
这两个类应该放到同一个py文件中, 同时这个文件内还有其他一些辅助性质的代码:
importioimportsocketfrom typing importGenerator, Iterable, Optional, Tuple, Unionfrom urllib.parse importurlsplit
_CS_IDLE= 'Idle'_CS_REQ_STARTED= 'Request-started'_CS_REQ_SENT= 'Request-sent'_METHODS_EXPECTING_BODY= {'PATCH', 'POST', 'PUT'}
_UNKNOWN= 'UNKNOWN'_MAX_LINE= 65536_MAX_HEADERS= 100_EMPTY_LINE= (b'\r\n', b'\n', b'')classHTTPStatus:
CONTINUE= 100SWITCHING_PROTOCOLS= 101PROCESSING= 102OK= 200CREATED= 201ACCEPTED= 202NON_AUTHORITATIVE_INFORMATION= 203NO_CONTENT= 204RESET_CONTENT= 205PARTIAL_CONTENT= 206MULTI_STATUS= 207ALREADY_REPORTED= 208IM_USED= 226MULTIPLE_CHOICES= 300MOVED_PERMANENTLY= 301FOUND= 302SEE_OTHER= 303NOT_MODIFIED= 304USE_PROXY= 305TEMPORARY_REDIRECT= 307PERMANENT_REDIRECT= 308BAD_REQUEST= 400UNAUTHORIZED= 401PAYMENT_REQUIRED= 402FORBIDDEN= 403NOT_FOUND= 404METHOD_NOT_ALLOWED= 405NOT_ACCEPTABLE= 406PROXY_AUTHENTICATION_REQUIRED= 407REQUEST_TIMEOUT= 408CONFLICT= 409GONE= 410LENGTH_REQUIRED= 411PRECONDITION_FAILED= 412REQUEST_ENTITY_TOO_LARGE= 413REQUEST_URI_TOO_LONG= 414UNSUPPORTED_MEDIA_TYPE= 415REQUESTED_RANGE_NOT_SATISFIABLE= 416EXPECTATION_FAILED= 417MISDIRECTED_REQUEST= 421UNPROCESSABLE_ENTITY= 422LOCKED= 423FAILED_DEPENDENCY= 424UPGRADE_REQUIRED= 426PRECONDITION_REQUIRED= 428TOO_MANY_REQUESTS= 429REQUEST_HEADER_FIELDS_TOO_LARGE= 431UNAVAILABLE_FOR_LEGAL_REASONS= 451INTERNAL_SERVER_ERROR= 500NOT_IMPLEMENTED= 501BAD_GATEWAY= 502SERVICE_UNAVAILABLE= 503GATEWAY_TIMEOUT= 504HTTP_VERSION_NOT_SUPPORTED= 505VARIANT_ALSO_NEGOTIATES= 506INSUFFICIENT_STORAGE= 507LOOP_DETECTED= 508NOT_EXTENDED= 510NETWORK_AUTHENTICATION_REQUIRED= 511
classHTTPResponse:
...classHTTPConnection:
...def _encode(data: str, encoding: str = 'latin-1', name: str = 'data') ->bytes:#给请求正文等不知道能怎么转码的东西转码时用这个,默认使用latin-1编码
#它的好处是,转码失败后能抛出详细的错误信息,一目了然
try:returndata.encode(encoding)exceptUnicodeEncodeError as err:raiseUnicodeEncodeError(
err.encoding,
err.object,
err.start,
err.end,"{} ({:.20!r}) is not valid {}. Use {}.encode('utf-8') if you want to send it encoded in UTF-8.".format(
name.title(), data[err.start:err.end], encoding, name)
)fromNoneclassHTTPException(Exception):pass
classImproperConnectionState(HTTPException):pass
classCannotSendRequest(ImproperConnectionState):pass
classCannotSendHeader(ImproperConnectionState):pass
classCannotCloseStream(ImproperConnectionState):pass
classResponseNotReady(ImproperConnectionState):pass
classLineTooLong(HTTPException):def __init__(self, line_type):
HTTPException.__init__(self, 'got more than %d bytes when reading %s'
%(_MAX_LINE, line_type))classBadStatusLine(HTTPException):def __init__(self, line):if notline:
line=repr(line)
self.args=line,
self.line=lineclassBadHeaderLine(HTTPException):def __init__(self, line):if notline:
line=repr(line)
self.args=line,
self.line=lineclassRemoteDisconnected(ConnectionResetError, BadStatusLine):def __init__(self, *args, **kwargs):
BadStatusLine.__init__(self, '')
ConnectionResetError.__init__(self, *args, **kwargs)classUnknownProtocol(HTTPException):def __init__(self, version):
self.args=version,
self.version=versionclassUnknownTransferEncoding(HTTPException):pass
classIncompleteRead(HTTPException):def __init__(self, partial, expected=None):
self.args=partial,
self.partial=partial
self.expected=expecteddef __repr__(self):if self.expected is notNone:
e= f', {self.expected} more expected'
else:
e= ''
return f'{self.__class__.__name__}({len(self.partial)} bytes read{e})'
__str__ = object.__str__
client.py
2. 需要注意的点
总的来说, 本文的内容不算复杂, 毕竟HTTP属于不难理解, 但知识点很多很杂的类型. 这里把本文中一些需要注意的点总结一下:
请求和响应数据的结构大致相同, 都是状态行+头部+正文, 状态行和头部的每个字段都用一个\r\n分割, 与正文之间用两个分割;
状态行是必须的, 请求头则最少需要host这个字段, 同时为了大家的方便, 你最好也设置一下Accept-encoding和Accept来限制服务器返回给你的数据内容和格式;
正文不是必须的, 特别是对于除了3P(PATCH, POST, PUT)之外的方法来说. 如果你有正文, 你最好在header中使用Content-Length说明正文的长度, 如果是分块发送, 则使用Transfer-Encoding字段说明;
如果对正文使用分块传输, 每块的格式是: 16进制的数据长度+\r\n+数据+\r\n, 使用0\r\n\r\n来收尾. 收尾之后, 你还可以放一个trailer, 里面放数据的MD5值或者过期时间什么的, 这时候最好在header中设置trailer字段;
在一个请求的生命周期完成后, TCP连接是否会断开取决于三点: 响应数据的HTTP版本, 响应头中的Connection和Keep-Alive字段, 是否知道响应正文的长度;
最最重要的一点, HTTP协议只是一个约定而非限制, 这就和矿泉水的建议零售价差不多, 你可以选择遵守, 也可以不遵守, 后果自负.
3. 结果测试
首先, 我们用tornado写一个简单的服务器, 它会显示客户端的地址和接口;
importtornado.webimporttornado.ioloopclassIndexHandler(tornado.web.RequestHandler):def get(self) ->None:print(f'new connection from {self.request.connection.context.address}')
self.write('hello world')
app= tornado.web.Application([(r'/', IndexHandler)])
app.listen(8888)
tornado.ioloop.IOLoop.current().start()
然后, 使用我们刚写好的客户端进行测试:
from client importHTTPConnectiondef fetch(conn: HTTPConnection, url: str = '') ->None:
conn.request('GET', url)
res=conn.get_response()print(res.read())
connection= HTTPConnection('127.0.0.1', 8888)for i in range(10):
fetch(connection)
结果如下:
3万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
