python多线程下载大文件_如何使用Python 2.7通过HTTP使用多线程下载文件（异步下载）...

I have a file to download (download path extracted from json. eg: http://testsite/abc.zip).

I need a help to perform, all the 5 threads should download the abc.zip file to the output directory and the download has to be Asynchronous or concurrent.

Currently with the below code it does download the file 5 times but it downloads one by one (Synchronous).

What I want is, the download to be simultaneous.

def dldr(file=file_url, outputdir=out1):

local_fn = str(uuid.uuid4())

if not os.path.exists(outputdir):

os.makedirs(outputdir)

s = datetime.now()

urllib.urlretrieve(file, outputdir + os.sep + local_fn)

e = datetime.now()

time_diff = e - s

logger(out1, local_fn, time_diff)

for i in range(1, 6):

t = threading.Thread(target=dldr())

t.start()

I have read Requests with multiple connections post and it's helpful, but doesn't address the requirement of the question asked.

解决方案

I use threading module for download threads:

Also requests, but you can change that to urllib by yourself.

import threading

import requests

def download(link, filelocation):

r = requests.get(link, stream=True)

with open(filelocation, 'wb') as f:

for chunk in r.iter_content(1024):

if chunk:

f.write(chunk)

def createNewDownloadThread(link, filelocation):

download_thread = threading.Thread(target=download, args=(link,filelocation))

download_thread.start()

for i in range(0,5):

file = "C:\\test" + str(i) + ".png"

print file

createNewDownloadThread("http://stackoverflow.com/users/flair/2374517.png", file)