一、前言 |
刚刚只是在理论上讲述了gevent遇到io自动切换,下面我们就来实际操作一下,在实战过程中我们用协程大面积的爬虫,看看如何用gevent去实现并发的效果的。
二、串行爬网页 |
2.1、串行爬网页
说明:我们先来看看串行效果的爬网页的代码,看看消耗多长时间
from urllib import request #简单的爬虫模块,复杂的不用这个
import time
def f(url):
print("GET:{0}".format(url))
resp = request.urlopen(url) #request.urlopen()函数
data = resp.read() #读取爬到的数据
with open("url.html","wb") as f:
f.write(data)
print('{0} bytes received from {1}'.format(len(data), url))
urls = [
'http://www.163.com/',
'https://www.yahoo.com/',
'https://github.com/'
]
time_start = time.time() #开始时间
for url in urls:
f(url)
print("同步cost",time.time()-time_start) #程序执行消耗的时间
执行结果如下:
GET:http://www.163.com/
658380 bytes received from http://www.163.com/
GET:https://www.yahoo.com/
468153 bytes received from https://www.yahoo.com/
GET:https://github.com/
55467 bytes received from https://github.com/
同步cost 5.505090951919556 #程序消耗的时间
2.2、gevent协程爬虫
说明:刚刚是串行的执行的,我们现在用gevent并发执行一下,看看效果。
from urllib import request
import gevent,time
def f(url):
print("GET:{0}".format(url))
resp = request.urlopen(url)
data = resp.read()
with open("url.html","wb") as f:
f.write(data)
print('{0} bytes received from {1}'.format(len(data), url))
async_time_start = time.time()
gevent.joinall([ #用gevent启动协程
gevent.spawn(f,'http://www.163.com/'), #第二个值是传入参数,之前我们没有讲,因为前面没有传参
gevent.spawn(f,'https://www.yahoo.com/'),
gevent.spawn(f,'https://github.com/'),
])
print("异步cost",time.time()-async_time_start) #计算时间
执行结果如下:
GET:http://www.163.com/
658380 bytes received from http://www.163.com/
GET:https://www.yahoo.com/
466264 bytes received from https://www.yahoo.com/
GET:https://github.com/
55459 bytes received from https://github.com/
异步cost 6.204461574554443 #执行时间
问题:为啥我用了并发,执行的时间没有缩短,反而变的更长了呢?
其实urllib默认跟gevent是没有关系的。urllib现在默认,如果你要通过gevent来去调用,它就是阻塞,gevent现在检测不到urllib的IO操作。它都不知道urllib进行了IO操作,所以它都不会进行切换,所以它就串行了。所以这个urllib和我们之前学的socket交给gevent不好使,因为gevent它不知道你进行了IO操作,所以就会卡住。
三、并发爬网页 |
既然上面那种情况都不行,那怎么让gevent知道urllib正在进行IO操作呢?
答:打补丁,通过导入monkey,来打这个补丁,在程序中什么都不写,就添加一行monkey.patch()即可。
3.1、代码
from urllib import request
import gevent,time
from gevent import monkey #导入monkey
monkey.patch_all() #把当前程序的所有的io操作给我单独的作上标记,且就执行这一句即可
def f(url):
print("GET:{0}".format(url))
resp = request.urlopen(url)
data = resp.read()
with open("url.html","wb") as f:
f.write(data)
print('{0} bytes received from {1}'.format(len(data), url))
urls = [
'http://www.163.com/',
'https://www.yahoo.com/',
'https://github.com/'
]
time_start = time.time()
for url in urls:
f(url)
print("同步cost",time.time()-time_start) #串行时间计算
async_time_start = time.time()
gevent.joinall([
gevent.spawn(f,'http://www.163.com/'),
gevent.spawn(f,'https://www.yahoo.com/'),
gevent.spawn(f,'https://github.com/'),
])
print("异步cost",time.time()-async_time_start) #并发的时间的时间计算
执行结果:
GET:http://www.163.com/
658315 bytes received from http://www.163.com/
GET:https://www.yahoo.com/
467577 bytes received from https://www.yahoo.com/
GET:https://github.com/
55467 bytes received from https://github.com/
同步cost 4.895136833190918 #同步执行的结果
GET:http://www.163.com/
GET:https://www.yahoo.com/
GET:https://github.com/
658315 bytes received from http://www.163.com/
471042 bytes received from https://www.yahoo.com/
55467 bytes received from https://github.com/
异步cost 3.0067789554595947 #异步执行的结果
哈哈,看到效果了吧,其实差距不大,还有一个原因就是网络的原因也有。总之这个是需要通过打补丁的。其实就是说通过打补丁来检测到它有urllib,它就把urllib里面所有涉及到的有可能进行IO操作的地方直接花在前面加一个标记,这个标记就相当于gevent.sleep(),所以把urllib变成一个一有阻塞,它就切换了。
注意了,gevent.sleep()是模拟IO操作的,标记的意思是,这边是IO操作,遇到阻塞就切换。
四、gevent实现单线程下的多socket并发 |
4.1、server端
import sys,gevent,socket,time
from gevent import socket,monkey
monkey.patch_all()
def server(port):
s = socket.socket()
s.bind(('0.0.0.0', port))
s.listen(500)
while True:
cli, addr = s.accept()
gevent.spawn(handle_request, cli) #协程
def handle_request(conn):
try:
while True:
data = conn.recv(1024)
print("recv:", data)
conn.send(data)
if not data:
conn.shutdown(socket.SHUT_WR)
except Exception as ex:
print(ex)
finally:
conn.close()
if __name__ == '__main__':
server(8001)
4.2、client端
import socket
HOST = 'localhost' # The remote host
PORT = 8001 # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
while True:
msg = bytes(input(">>:"),encoding="utf8")
s.sendall(msg)
data = s.recv(1024)
print('Received', repr(data))
s.close()