python ftp批量下载文件_如何从Python中的ftp同时下载一些文件

I'm a newbie in Python programming.

My question is, how to download a few files at the same time. Not file by file but simultaneously from one directory on ftp. Now I use this script but I don't know how I can rebuild this code:

filenames = []

ftp.retrlines("NLST", filenames.append)

print filenames

print path

for filename in filenames:

local_filename = filename

print filename

print local_filename

f = open(local_filename, "wb")

s = ftp.size(local_filename)

sMB = s/(1024*1024)

print "file name: " + local_filename + "\nfile size: " + str(sMB) + " MB"

ftp.retrbinary("RETR %s" % local_filename, f.write)

print "\n Done :) "

time.sleep(2)

f.close()

ftp.quit() #closing connection

time.sleep(5)

It works fine, but not what I need.

解决方案

You could use multiple threads or processes. Make sure you create a new ftplib.FTP object in each thread. The simplest way (code-wise) is to use multiprocessing.Pool:

#!/usr/bin/env python

from multiprocessing.dummy import Pool # use threads

try:

from urllib import urlretrieve

except ImportError: # Python 3

from urllib.request import urlretrieve

def download(url):

url = url.strip()

try:

return urlretrieve(url, url2filename(url)), None

except Exception as e:

return None, e

if __name__ == "__main__":

p = Pool(20) # specify number of concurrent downloads

print(p.map(download, open('urls'))) # perform parallel downloads

where urls contains ftp urls for the files to download e.g., ftp://example.com/path/to/file and url2filename() extracts the filename part from an url e.g.:

import os

import posixpath

try:

from urlparse import urlsplit

from urllib import unquote

except ImportError: # Python 3

from urllib.parse import urlsplit, unquote

def url2filename(url, encoding='utf-8'):

"""Return basename corresponding to url.

>>> print url2filename('http://example.com/path/to/dir%2Ffile%C3%80?opt=1')

fileÀ

"""

urlpath = urlsplit(url).path

basename = posixpath.basename(unquote(urlpath))

if os.path.basename(basename) != basename:

raise ValueError(url) # reject 'dir%5Cbasename.ext' on Windows

return basename