python爬虫的requests库详解

一行玩python

于 2019-05-29 16:23:54 发布

阅读量1.9k

点赞数 1

文章标签： python 爬虫 requests库详解

更多资料、源码请关注公众号「一行玩python」即可领取

本文链接：https://blog.csdn.net/qq_42554007/article/details/90673269

版权

本文详细介绍了Python的requests库，包括其作为HTTP客户端的便捷特性，如基本GET和POST请求、响应处理、高级操作如文件上传、会话维持及异常处理等，是进行网络数据抓取和HTTP交互的利器。

摘要由CSDN通过智能技术生成

1.requests是什么

Requests 是⽤Python语⾔编写，基于urllib，采⽤Apache2 Licensed 开源协议的 HTTP库。
它⽐ urllib 更加⽅便，可以节约我们⼤量的⼯作，完全满足HTTP 测试需求。
⼀句话——Python实现的简单易⽤的HTTP库

2.安装

pip3 install requests

3.requests

3.1实例引入

import requests

response = requests.get('https://www.baidu.com/')
print(type(response))
print(response.status_code)
print(type(response.text))
print(response.text)
print(response.cookies)

3.2 各种请求方式

import requests
requests.post('http://httpbin.org/post')
requests.put('http://httpbin.org/put')
requests.delete('http://httpbin.org/delete')
requests.head('http://httpbin.org/get')
requests.options('http://httpbin.org/get')

请求

基本GET请求

基本写法

import requests

response = requests.get('http://httpbin.org/get')
print(response.text)

带参数GET请求

import requests
response = requests.get("http://httpbin.org/get?name=germey&age=22")
print(response.text)

import requests

data = {
   
    'name': 'germey',
    'age': 22
}
response = requests.get("http://httpbin.org/get", params=data)
print(response.text)

解析json

import requests
import json

response = requests.get("http://httpbin.org/get")
print(type(response.text))
print(response.json())
print(json.loads(response.text))
print(type(response.json()))

获取二进制数据

import requests

response = requests.get("https://github.com/favicon.ico")
print(type(response.text), type(response.content))
print(response.text)
print(response.content)

import requests

response = requests.get("https://github.com/favicon.ico")
with open('favicon.ico', 'wb') as f:
    f.write(response.content)
    f.close()

添加headers

import requests

response = requests.get("https://www.zhihu.com/explore")
print(response.text)

import requests

headers = {
   
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'
}
response = requests.get("https://www.zhihu.com/explore", headers=headers)
print(response.text)

基本POST请求

import requests

data = {
   'name': 'germey', 'age': '22'}
response = requests.post("http://httpbin.org/post", data

最低0.47元/天解锁文章