Python爬虫(二)：Requests详解

最新推荐文章于 2024-02-21 16:31:23 发布

清紫悠悠

最新推荐文章于 2024-02-21 16:31:23 发布

阅读量1k

点赞数

文章标签： Python 爬虫

本文链接：https://blog.csdn.net/qq_36833171/article/details/97395531

版权

Requests库的使用

整体演示

import requests

response = requests.get("https://www.baidu.com")
print(response)     # <Response [200]>
print(type(response))   # <class 'requests.models.Response'>
print(response.status_code) # 200
# print(response.text)  # <!DOCTYPE html> ...
print(response.cookies) # <RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>]>
print(response.content) # <!DOCTYPE html> ...   返回二进制格式，建议不用response.text
print(response.content.decode("utf-8")) # <!DOCTYPE html> ...  转码为的UTF-8

response.text会自动检测目标网页的编码格式，可能会造成乱码问题
可以通过response.encoding手动设置编码集

response =requests.get("http://www.baidu.com")
response.encoding="utf-8"
print(response.text)

通过response.encoding="utf-8"和response.content.decode("utf-8")都可以解决乱码问题

各种请求方式

request提供了各种请求的方式

import requests
print(requests.post("http://httpbin.org/post").text)
print(requests.put("http://httpbin.org/post").text)
print(requests.delete("http://httpbin.org/post").text)
print(requests.head("http://httpbin.org/post").text)
print(requests.options("http://httpbin.org/post").text)

向URL传递查询数据

requests可以向params传递字典的方式向URL传递查询数据

import requests
data = {
    "name":"zhaofan",
    "age":22
}
response = requests.get("http://httpbin.org/get",params=data)
print(response.url)
print(response.text`)

解析JSON

import requests
import json

response = requests.get("http://httpbin.org/get")
print(response.text)
print(response.json())
print(json.loads(response.text))
print(type(response.json()))

result:

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.22.0"
  }, 
  "origin": "111.227.172.92, 111.227.172.92", 
  "url": "https://httpbin.org/get"
}

{'args': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.22.0'}, 'origin': '111.227.172.92, 111.227.172.92', 'url': 'https://httpbin.org/get'}
{'args': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.22.0'}, 'origin': '111.227.172.92, 111.227.172.92', 'url': 'https://httpbin.org/get'}
<class 'dict'>

可以看出json.loads(response.text)等同于json()

获取二进制数据

之前用到的response.content获取的是二进制数据，也可以用来获取视频，音乐等资源

添加headers

类似urllib一样，request也可以定制headers的信息，如下，直接访问知乎网站是无法访问的

import requests
response= requests.get("http://www.zhihu.com")
print(response)
print(response.text)

result:

<Response [400]>
<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>openresty</center>
</body>
</html>

因为访问知乎需要头文件，我们在Chrome种输入chrome://version/即可获取Headers信息

import requests
headers = {
    "User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36"
}

response= requests.get("http://www.zhihu.com",headers=headers)
print(response)

result:

<Response [200]>

发送POST请求

使用字典设置发送POST请求的参数

import requests

data = {
    "name":"zhaofan",
    "age":23
}

reponse = requests.post("http://httpbin.org/post",data=data)
print(reponse.text)

响应

通过response可以得到很多属性

import requests

response = requests.get("http://www.baidu.com")     
print(response.status_code)     # 200
# print(response.headers)       
print(response.cookies) # <RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>]>

print(response.url)     # http://www.baidu.com/
print(response.history)    # []

状态码判断

Requests还附带了一个内置的状态码查询对象

100: ('continue',),
101: ('switching_protocols',),
102: ('processing',),
103: ('checkpoint',),
122: ('uri_too_long', 'request_uri_too_long'),
200: ('ok', 'okay', 'all_ok', 'all_okay', 'all_good', '\o/', '✓'),
201: ('created',),
202: ('accepted',),
203: ('non_authoritative_info', 'non_authoritative_information'),
204: ('no_content',),
205: ('reset_content', 'reset'),
206: ('partial_content', 'partial'),
207: ('multi_status', 'multiple_status', 'multi_stati', 'multiple_stati'),
208: ('already_reported',),
226: ('im_used',),

Redirection.
300: ('multiple_choices',),
301: ('moved_permanently', 'moved', '\o-'),
302: ('found',),
303: ('see_other', 'other'),
304: ('not_modified',),
305: ('use_proxy',),
306: ('switch_proxy',),
307: ('temporary_redirect', 'temporary_moved', 'temporary'),
308: ('permanent_redirect',
'resume_incomplete', 'resume',), # These 2 to be removed in 3.0

Client Error.
400: ('bad_request', 'bad'),
401: ('unauthorized',),
402: ('payment_required', 'payment'),
403: ('forbidden',),
404: ('not_found', '-o-'),
405: ('method_not_allowed', 'not_allowed'),
406: ('not_acceptable',),
407: ('proxy_authentication_required', 'proxy_auth', 'proxy_authentication'),
408: ('request_timeout', 'timeout'),
409: ('conflict',),
410: ('gone',),
411: ('length_required',),
412: ('precondition_failed', 'precondition'),
413: ('request_entity_too_large',),
414: ('request_uri_too_large',),
415: ('unsupported_media_type', 'unsupported_media', 'media_type'),
416: ('requested_range_not_satisfiable', 'requested_range', 'range_not_satisfiable'),
417: ('expectation_failed',),
418: ('im_a_teapot', 'teapot', 'i_am_a_teapot'),
421: ('misdirected_request',),
422: ('unprocessable_entity', 'unprocessable'),
423: ('locked',),
424: ('failed_dependency', 'dependency'),
425: ('unordered_collection', 'unordered'),
426: ('upgrade_required', 'upgrade'),
428: ('precondition_required', 'precondition'),
429: ('too_many_requests', 'too_many'),
431: ('header_fields_too_large', 'fields_too_large'),
444: ('no_response', 'none'),
449: ('retry_with', 'retry'),
450: ('blocked_by_windows_parental_controls', 'parental_controls'),
451: ('unavailable_for_legal_reasons', 'legal_reasons'),
499: ('client_closed_request',),

Server Error.
500: ('internal_server_error', 'server_error', '/o\', '✗'),
501: ('not_implemented',),
502: ('bad_gateway',),
503: ('service_unavailable', 'unavailable'),
504: ('gateway_timeout',),
505: ('http_version_not_supported', 'http_version'),
506: ('variant_also_negotiates',),
507: ('insufficient_storage',),
509: ('bandwidth_limit_exceeded', 'bandwidth'),
510: ('not_extended',),
511: ('network_authentication_required', 'network_auth', 'network_authentication'),

也可以通过以下方式进行测试

import requests

response= requests.get("http://www.baidu.com")
if response.status_code == requests.codes.ok:
    print("访问成功")

response高级用法

文件上传

file类似其他参数，以字典方式上传

import requests
files= {"files":open("git.jpeg","rb")}
response = requests.post("http://httpbin.org/post",files=files)
print(response.text)

获取Cookie

import requests

response = requests.get("http://www.baidu.com")
print(response.cookies)

for key,value in response.cookies.items():
    print(key+"="+value)

会话维持

import requests
# 创建Session监控对象
s = requests.Session()

# 发送个Get请求，并产生Cookie
s.get("http://httpbin.org/cookies/set/number/123456")

# 获取Cookie
response = s.get("http://httpbin.org/cookies")
print(response.text)

证书验证

现在的很多网站都是https的方式访问，所以这个时候就涉及到证书的问题

import requests

response = requests.get("https:/www.12306.cn")
print(response.status_code)

默认12306的证书是不合法的

解决办法

import requests
from requests.packages import urllib3
urllib3.disable_warnings()
response = requests.get("https://www.12306.cn",verify=False)
print(response.status_code)

代理设置

import requests

proxies= {
    "http":"http://127.0.0.1:9999",
    "https":"http://127.0.0.1:8888"
}
response  = requests.get("https://www.baidu.com",proxies=proxies)
print(response.text)

如果代理需要设置账户名和密码,只需要将字典更改为如下：

proxies = {
"http":"http://user:password@127.0.0.1:9999"
}

如果你的代理是通过sokces这种方式则需要
pip install "requests[socks]"

proxies= {
"http":"socks5://127.0.0.1:9999",
"https":"sockes5://127.0.0.1:8888"
}

异常处理

import requests

from requests.exceptions import ReadTimeout,ConnectionError,RequestException

try:
    response = requests.get("http://httpbin.org/get",timeout=0.1)
    print(requests.status_codes)
except ReadTimeout:
    print("time out")
except ConnectionError:
    print('connection Error')
except RequestException:
    print("error")

result: connection Error

清紫悠悠

关注

0
点赞
踩
4

收藏

觉得还不错? 一键收藏
0
评论
Python爬虫(二)：Requests详解

Requests库的使用整体演示import requestsresponse = requests.get("https://www.baidu.com")print(response) # <Response [200]>print(type(response)) # <class 'requests.models.Response'>prin...
复制链接

扫一扫