python可不可以同时执行1000个线程_同时在python中运行多个线程-有可能吗？

I'm writing a little crawler that should fetch a URL multiple times, I want all of the threads to run at the same time (simultaneously).

I've written a little piece of code that should do that.

import thread

from urllib2 import Request, urlopen, URLError, HTTPError

def getPAGE(FetchAddress):

attempts = 0

while attempts < 2:

req = Request(FetchAddress, None)

try:

response = urlopen(req, timeout = 8) #fetching the url

print "fetched url %s" % FetchAddress

except HTTPError, e:

print 'The server didn\'t do the request.'

print 'Error code: ', str(e.code) + " address: " + FetchAddress

time.sleep(4)

attempts += 1

except URLError, e:

print 'Failed to reach the server.'

print 'Reason: ', str(e.reason) + " address: " + FetchAddress

time.sleep(4)

attempts += 1

except Exception, e:

print 'Something bad happened in gatPAGE.'

print 'Reason: ', str(e.reason) + " address: " + FetchAddress

time.sleep(4)

attempts += 1

else:

try:

return response.read()

except:

"there was an error with response.read()"

return None

return None

url = ("http://www.domain.com",)

for i in range(1,50):

thread.start_new_thread(getPAGE, url)

from the apache logs it doesn't seems like the threads are running simultaneously, there's a little gap between requests, it's almost undetectable but I can see that the threads are not really parallel.

I've read about GIL, is there a way to bypass it with out calling a C\C++ code?

I can't really understand how does threading is possible with GIL? python basically interpreters the next thread as soon as it finishes with the previous one?

Thanks.

解决方案

As you point out, the GIL often prevents Python threads from running in parallel.

However, that's not always the case. One exception is I/O-bound code. When a thread is waiting for an I/O request to complete, it would typically have released the GIL before entering the wait. This means that other threads can make progress in the meantime.

In general, however, multiprocessing is the safer bet when true parallelism is required.