python网络爬虫(requests库)

requsets

get请求

基本样例

import requests
url = "http://http://httpbin.org/get"
r = requests.get(url)
print(r.text)

运行结果如下

post请求

import requests
data = {'name': 'germey','age': '22'}
r = requests.post("http://httpbin.org/post",data=data)
print(r.text)

运行结果如下

响应

import requests
r = requests.get("http://www.jianshu.com")
print(r.status_code)#状态码
print(r.headers)#响应头
print(r.cookies)#cookies

文件上传

有的网址需要上传文件，可以这样实现

import requests
files = {"file":open("favicon.ico","rb")}
r = requests.post("http://httpbin.org/post",files=files)
print(r.text)

Cookies

获取cookies

import requests
r = requests.get("http://www.baidu.com")
print(r.cookies)
for key,value in r.cookies.items():
    print(key + '=' + value)

更改cookies

import requests
headers = {
    'Cookie':'#更改后的Cookie',
    'Host' : 'www.zhihu.com',
    'User-Agent':"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36",
}
r = requests.get("http://www.zhihu.com/explore",headers=headers)
print(r.text)

会话维持

Session对象

import requests
s = requests.Session()
s.get('http://httpbin.org/cookies/set/number/123456789')
r = s.get("http://httpbin.org/cookies")
print(r.text)

SSL证书验证

使用verify参数控制是否检查此证书默认True

import requests
from requests.packages import urllib3#忽视警告
urllib3.disable_warnings()
r = requests.get('https://www.12306.cn',verify=False)
print(r.status_code)

代理设置

使用proxies参数

超时设置

import requests
r = requests.get("http://www.taobao.com",timeout = 1)
print(r.status_code)#建议循环一百次，效果明显

身份认证

import requests
r = requests.get("http://localhost:500",auth=('username','password'))
print(r.stauts_code)#认证成功返回200，失败返回401