python中grequests模块简单应用

最近要使用requests向某个url批量发起POST请求，交互的内容很简单，若开启多线程去访问占用太多资源不合理。
想到了使用协程gevent，grequests 模块相当于是封装了gevent的requests模块，记录一下基础的使用方式。

---

1.安装 grequests模块 

    若是联网安装，直接 pip install grequests 完事
    若是离线安装，需要先安装 greenlet模块 和 gevent模块 ，再安装 grequests模块 ，可以分别去这里https://pypi.org/ 下载对应的离线包
    (windows环境离线安装gevent的whl包不一定会成功，可以下载后缀exe的离线包)

2.简单使用grequests

    实例化请求对象：   grequests.request(method, url, **kwargs)
    发起请求获得响应： grequests.map(requests, stream=False, size=None, exception_handler=None, gtimeout=None)
    如：一个简单的POST请求

# coding:utf-8
adata = json.dumps({"key": "value"})
header = {"Content-type": "appliaction/json", "Accept":"application/json"}
url = "http://www.baidu.com"

task = []
req = grequests.request("POST", url=url, data=adata, headers=header)
task.append(req)
此处map的requests参数是装有实例化请求对象的列表,其返回值也是列表， size参数可以控制并发的数量
resp = grequests.map(task)
print resp
查看返回值的属性值，我们关注的一般就是text json links  url headers 等了
print dir(resp[0])

 其他 PUT DELETE 等请求格式一样，其中GET请求有更简便的写法

# coding:utf-8
import grequests
import json
urls = ["http://www.baidu.com", "http://www.baidu.com", "http://www.baidu.com"]
req = (grequests.get(u) for u in urls)
resp = grequests.map(req)
print resp

3.使用grequests和requests的对比

requests和grequests做一个简单对比，请求某个固定的地址，当然这种对比方式是否合理就仁者见仁了

# coding:utf-8
import grequests
import time
import json
import requests


adata = json.dumps({"key": "value"})
header = {"Content-type": "appliaction/json", "Accept":"application/json"}


def use_grequests(num):
    task = []
    urls = ["http://hao.jobbole.com/python-docx/" for i in range(num)]
    while urls:
        url = urls.pop(0)
        rs = grequests.request("POST", url, data=adata, headers=header)
        task.append(rs)
    resp = grequests.map(task, size=5)
    return resp


def use_requests(num):
    urls = ["http://hao.jobbole.com/python-docx/" for i in range(num)]
    index = 0
    while urls:
        url = urls.pop(0)
        resp = requests.post(url=url, headers=header, data=adata)
        index += 1
        if index % 10 == 0:
            print u'目前是第{}个请求'.format(index)


def main(num):
    time1 = time.time()
    finall_res = use_requests(num)
    print finall_res
    time2 = time.time()
    T = time2 - time1
    print u'use_requests发起{}个请求花费了{}秒'.format(num, T)

    print u'正在使用grequests模块发起请求...'
    time3 = time.time()
    finall_res2 = use_grequests(num)
    print finall_res2
    time4 = time.time()
    T2 = time4 - time3
    print u'use_grequests发起{}个请求花费了{}秒'.format(num, T2)


if __name__ == '__main__':
    main(100)

 运行结果如下图，还是比单个的requests请求要快一点，当网络延迟较高时，grequests优势比较明显
 修改 map的size参数进行尝试,合理的设置size值，不要太大，超过阀值再大也没有用

 

4. 手动使用gevent配合requests模块

# coding:utf-8
import gevent
import time
from gevent import monkey
import requests
monkey.patch_all()

datali = [x for x in range(100)]
task = []
def func(i):
	print u'第{}个请求'.format(i)
    url = "http://hao.jobbole.com/python-docx/"
    resp = requests.get(url=url)
    return resp

time1 = time.time()
for i in datali:
    task.append(gevent.spawn(func, i))
res = gevent.joinall(task)
print len(res)
time2 = time.time()
T = time2 - time1
print u'消耗了{}秒'.format(T)