Python Spider：requests库使用

requests库详解：
基于urllib开发的，采用Apache2 Licensed开源协议的HTTp库

安装：pip3 install requests
response = requests.get(url)

response.text 请求文本内容

带参数的get请求
data = {
key:value
}
response = requests.get(url, params=data)
解析json:
response = requests.get(url)
response.json() 返回json格式数据
获取二进制数据
response.content
with open(‘图片名’, ‘wb’) as f:
f.write(response.content)
添加headers
requests.get(url, headers=headers)

基本的POST请求
data ={}
response = requests.get(url, data=data)
response.text

响应:
response属性
response.status_code
response.headers
response.url
response.cookies

会话维持:模拟登陆
s = requests.Session()
s.get(‘http://httpbin.org/cookies/set/number/1231231’)
response = s.get(‘http://httpbin.org/cookies’)

证书验证:
response = requests.get(url, verify=False)
代理设置:
proxies = {
‘http’:‘http://127.0.0.1:9743’
#有有户名密码
‘http’:‘uname：password@http://127.0.0.1:9743’
}
requests.get(url, proxies=proxies)

超时设置：
requests.get(url,timeout=)
超时抛出异常
认证设置:在访问网站时需要登陆才能浏览网页
from requests.auth import HTTPBasicAuth
requests.get(url, auth=HTTPBasicAth(‘uname’,‘upwd’)
或者
requests.get(url, auth=(‘uname’, ‘upwd’)