看文档学爬虫(6)——requests

转自：http://f61be319.wiz03.com/share/s/3S6-cp1BIQ952yXKyj02PIM43sCBdM2kcAzb2fp47L2J85Pp

简介

• 用python语言基于urllib编写
• requests是最简单易用的HTTP库

安装

pip install requests

基本get请求

• 无参数

import requests
r = requests.get('http://httpbin.org/get')
print(r.url)

• 有参数

import requests
payload = {
    'key1': 'value1',
    'key2': 'value2'
}
r = requests.get('http://httpbin.org/get', params=payload)
print(r.url)

• 设置header

import requests
headers = {
    'hello': 'world'
}
r = requests.get('http://httpbin.org/get', headers=headers)
print(r.text)

返回结果：

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Connection": "close",
    "Hello": "world",
    "Host": "httpbin.org",
    "User-Agent": "python-requests/2.17.3"
  },
  "origin": "61.52.106.18",
  "url": "http://httpbin.org/get"
}

基本post请求

• 有数据

import requests
payload = {
    'hello': 'world'
}
r = requests.post('http://httpbin.org/post', data=payload)
print(r.text)

返回结果为：

{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "hello": "world"
  },
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Connection": "close",
    "Content-Length": "11",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "User-Agent": "python-requests/2.17.3"
  },
  "json": null,
  "origin": "61.52.106.18",
  "url": "http://httpbin.org/post"
}

• 有数据(json)格式

import requests
import json
payload = {
    'hello': 'world'
}
r = requests.post('http://httpbin.org/post', data=json.dumps(payload))
print(r.text)

运行结果为：

{
  "args": {},
  "data": "{\"hello\": \"world\"}",
  "files": {},
  "form": {},
  "headers": {
   "Accept": "*/*",
   "Accept-Encoding": "gzip, deflate",
   "Connection": "close",
   "Content-Length": "18",
   "Host": "httpbin.org",
   "User-Agent": "python-requests/2.17.3"
  },
  "json": {
   "hello": "world"
  },
  "origin": "61.52.106.18",
  "url": "http://httpbin.org/post"
}

上传文件

在运行文件的目录下创建文本文件test.txt，写入hello world !!!

import requests
url = 'http://httpbin.org/post'
files = {
    'file': open('test.txt', 'rb')
}
r = requests.post(url, files=files)
print(r.text)

运行结果为：

{
  "args": {},
  "data": "",
  "files": {
    "file": "hello world !!!!!!!!!!!!!!!!!!"
  },
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Connection": "close",
    "Content-Length": "174",
    "Content-Type": "multipart/form-data; boundary=19732374a5934f6598891cef4524bee0",
    "Host": "httpbin.org",
    "User-Agent": "python-requests/2.17.3"
  },
  "json": null,
  "origin": "61.52.106.18",
  "url": "http://httpbin.org/post"
}

Cookies

import requests
url = 'http://httpbin.org/cookies'
cookies = dict(cookies_are='working')
r = requests.get(url, cookies=cookies)
print(r.text)

运行结果为：

{
  "cookies": {
    "cookies_are": "working"
  }
}

请求超时配置

使用timeout参数配置，单位为秒。

import requests
url = 'http://github.com'
r = requests.get(url, timeout=60)
print(r.text)

持久会话

没有持久会话时

import requests
requests.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = requests.get('http://httpbin.org/cookies')
print(r.text)

结果为：

{
  "cookies": {}
}

因为第二次get时候没有了cookies，返回的text为空。

设置持久会话的情况

import requests
s = requests.Session()
s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
r = s.get('http://httpbin.org/cookies')
print(r.text)

运行结果为：

{
  "cookies": {
    "sessioncookie": "123456789"
  }
}

代理

import requests
proxies = {
    'https': 'http://41.118.132.69:4433'
}
r = requests.post('https://www.baidu.com', proxies=proxies)
print(r.status_code)