python多线程图像识别_Python 多线程抓取图片效率对比

目的:

是学习python 多线程的工作原理，及通过抓取400张图片这种IO密集型应用来查看多线程效率对比

import requests

import urlparse

import os

import time

import threading

import Queue

path = '/home/lidongwei/scrapy/owan_img_urls.txt'

#path = '/home/lidongwei/scrapy/cc.txt'

fetch_img_save_path = '/home/lidongwei/scrapy/owan_imgs/'

# 读取保存再文件里面400个urls

with open(path) as f :

urls = f.readlines()

urls = urls[:400]

# 使用Queue来线程通信，因为队列是线程安全的(就是默认这个队列已经有锁)

q = Queue.Queue()

for url in urls:

q.put(url)

start = time.time()

def fetch_img_func(q):

while True:

try:

# 不阻塞的读取队列数据

url = q.get_nowait()

i = q.qsize()

except Exception, e:

print e

break;

print 'Current Thread Name R