Python requests、grequests库

1. requests

requests 是一个 Python HTTP 库，通过它可以方便地发起 HTTP 请求。

请求方法

import requests

r = requests.get('http://httpbin.org/get')
r = requests.post('http://httpbin.org/post', data = {'key':'value'})
r = requests.put('http://httpbin.org/put', data = {'key':'value'})
r = requests.delete('http://httpbin.org/delete')
r = requests.head('http://httpbin.org/get')
r = requests.options('http://httpbin.org/get')

URL参数

payload = {'key1': 'value1', 'key2': ['value2', 'value3']}
r = requests.get("http://httpbin.org/get", params=payload)

响应内容

• 文本形式

  r = requests.get('http://httpbin.org/get')
  r.encoding = 'utf-8'		# 设置编码
  print(r.text)	
  

• 字节形式

  r = requests.get('http://httpbin.org/get')
  print(r.content)
  

  b'{\n  "args": {}, \n  "headers": {\n    "Accept": "*/*",...
  

• JSON 形式

  r = requests.get('http://httpbin.org/get')
  print(r.json())
  

请求头

headers = {'user-agent': 'my-app/0.0.1'}
r = requests.get('http://httpbin.org/get', headers=headers)

响应头

r = requests.get('http://httpbin.org/get')
print(r.headers['Content-Type'])

POST请求

• POST表单

  payload = {'some': 'data'}
  r = requests.post('http://httpbin.org/post', data=payload)
  

• POST JSON

  payload = {'some': 'data'}
  r = requests.post('http://httpbin.org/post', json=payload)
  

超时

在经过指定秒数之后停止等待。

r = requests.get('http://github.com', timeout=0.001)

会话

会话对象让你能够跨请求保持某些参数。它也会在同一个 Session 实例发出的所有请求之间保持 cookie， 期间使用 urllib3 的 connection pooling 功能。所以如果你向同一主机发送多个请求，底层的 TCP 连接将会被重用，从而带来显著的性能提升。

s = requests.Session()

s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get("http://httpbin.org/cookies")

s.close()

会话也可为请求方法提供缺省数据。这是通过为会话对象的属性提供数据来实现的：

s = requests.Session()
s.auth = ('user', 'pass')
s.headers.update({'x-test': 'true'})

SSL证书验证

Requests 可以为 HTTPS 请求验证 SSL 证书，就像 web 浏览器一样。SSL 验证默认是开启的。

r = requests.get('https://github.com', verify='/path/to/certfile')

或：

s = requests.Session()
s.verify = '/path/to/certfile'

忽略证书验证：

r = requests.get('https://kennethreitz.org', verify=False)

---

2. grequests

grequest 是基于 gevent 和 requests 实现的，可以实现并发发送 HTTP 请求。

grequests 支持 get、post、put、delete 等HTTP请求方法，使用方法和 requests 一致。

import grequests


def err_handler(request: grequests.AsyncRequest, exception):
    print(f'{request.url}: {exception}')


req_list = [
    grequests.get('http://httpbin.org/get'),
    grequests.post('http://httpbin.org/post', data={'a': 1, 'b': 2}),
    grequests.get('http://httpbin.org/delay/1', timeout=0.001),
]

# 并发发送，等待所有请求响应后返回
res_list = grequests.map(req_list, exception_handler=err_handler)
print(res_list)

http://httpbin.org/delay/1: HTTPConnectionPool(host='httpbin.org', port=80): Max retries exceeded with url: /delay/1 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x000001FDC0034250>, 'Connection to httpbin.org timed out. (connect timeout=0.001)'))
[<Response [200]>, <Response [200]>, None]