爬虫（十） requests库

requests库和urllib库一样可以用来获取网页内容，但requests使用起来比起urllib库要方便许多，requests不需要想urllib那样先获取请求，再通过请求获取响应，只需要直接通过requests使用get/post等直接获取响应，并且requests库可以通过创建会话轻松实现cookie、代理等高级功能。

安装requests

pip install requests

官方使用文档：

http://www.python-request.org/

获取响应：

requests.get(url, headers=headers, params=data)

requests.post()....

响应对象（假设r为响应对象）：

r.text  字符串形式查看响应内容
r.content   字节类型查看响应内容
r.encoding   查看或者设置编码类型
r.status_code    查看状态码
r.headers      查看响应头部
r.url     查看所请求的url
r.json        查看json格式数据

get请求代码示例：

import requests

url = 'http://www.baidu.com/'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) '
                  'AppleWebKit/537.36 (KHTML, like Gecko)'
                  ' Chrome/63.0.3239.132 Safari/537.36'
}
response = requests.get(url, headers=headers)
# print(response)
response.encoding = 'utf-8'
with open(r'C:\Users\m1552\PycharmProjects\newWork\pa_chong\requests.txt', 'w', encoding = 'utf-8') as fp:
	fp.write(response.text)

带参数的get请求代码示例：

url = 'https://www.baidu.com/s?wd=%E4%B8%AD%E5%9B%BD'
data = {
	'wd':'%E4%B8%AD%E5%9B%BD',
}

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) '
                  'AppleWebKit/537.36 (KHTML, like Gecko)'
                  ' Chrome/63.0.3239.132 Safari/537.36'
}

# parameters参数
response = requests.get(url, headers=headers, params=data)
response.encoding = 'utf-8'
print(response.content)

# # 把结果写入到文件
# with open(r'C:\Users\m1552\PycharmProjects\newWork\pa_chong\baidu.html', 'wb') as fp:
# 	fp.write(response.content)

post请求代码示例：

import json
url = 'https://cn.bing.com/ttranslationlookup?&IG=DF5310683E534917A8E04ECAF4BE95D7&IID=translator.5038.4'
data = {
	'from':'en',
	'to':'zh-CHS',
	'text':'computer',
}

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) '
                  'AppleWebKit/537.36 (KHTML, like Gecko)'
                  ' Chrome/63.0.3239.132 Safari/537.36'
}

response = requests.post(url=url, headers=headers, data=data)

with open(r'C:\Users\m1552\PycharmProjects\newWork\pa_chong\computer.txt', 'w', encoding='utf8') as fp:
	fp.write(json.dumps(response.json()))

cookie的使用：

先创建一个会话Session，并且后面都用这个会话获取响应。

下面是代码示例：

import requests

# 如果碰到会话相关的问题，要首先创建一个会话。
# 往下所有的操作都通过s进行访问 s.get() s.post()
s = requests.Session()

post_url = 'http://www.renren.com/ajaxLogin/login?1=1&uniqueTimestamp=20191009575'

fromdata = {
	'email':'15521093428',
	'icode':'',
	'origURL':'http://www.renren.com/home',
	'domain':'renren.com',
	'key_id':'1',
	'captcha_type':'web_login',
	'password':'3c493a9b58fc672420b168275078df37bd0b51e3c424cec1e5928d37d3005a4d',
	'rkey':'2de811fa2fc149ce4704d89fcdb84777',
	'f':'https%3A%2F%2Fwww.baidu.com%2Flink%3Furl%3D1pi5EUYGZKK9a8cdrLgTUtcEDhxgrPP8IhzTf_vmIsy%26wd%3D%26eqid%3Dd710580c0002a94a000000035c5efa51',
}

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) '
                  'AppleWebKit/537.36 (KHTML, like Gecko)'
                  ' Chrome/63.0.3239.132 Safari/537.36'
}

response = s.post(url=post_url, headers=headers, data = fromdata)

# print(response.text)

get_url = 'http://www.renren.com/969499615/profile'
response2 = s.get(url = get_url, headers=headers)
print(response2.text)

代理使用：

r = requests.post(url=url, headers=headers, proxies=proxies)
proxies = {
   'http': 'http://代理号'
}