python内核爆了_Python线程全部在单个内核上执行

I have a Python program that spawns many threads, runs 4 at a time, and each performs an expensive operation. Pseudocode:

for object in list:

t = Thread(target=process, args=(object))

# if fewer than 4 threads are currently running, t.start(). Otherwise, add t to queue

But when the program is run, Activity Monitor in OS X shows that 1 of the 4 logical cores is at 100% and the others are at nearly 0. Obviously I can't force the OS to do anything but I've never had to pay attention to performance in multi-threaded code like this before so I was wondering if I'm just missing or misunderstanding something.

Thanks.

解决方案

Note that in many cases (and virtually all cases where your "expensive operation" is a calculation implemented in Python), multiple threads will not actually run concurrently due to Python's Global Interpreter Lock (GIL).

The GIL is an interpreter-level lock.

This lock prevents execution of

multiple threads at once in the Python

interpreter. Each thread that wants to

run must wait for the GIL to be

released by the other thread, which

means your multi-threaded Python

application is essentially single

threaded, right? Yes. Not exactly.

Sort of.

CPython uses what’s called “operating

system” threads under the covers,

which is to say each time a request to

make a new thread is made, the

interpreter actually calls into the

operating system’s libraries and

kernel to generate a new thread. This

is the same as Java, for example. So

in memory you really do have multiple

threads and normally the operating

system controls which thread is

scheduled to run. On a multiple

processor machine, this means you

could have many threads spread across

multiple processors, all happily

chugging away doing work.

However, while CPython does use

operating system threads (in theory

allowing multiple threads to execute

within the interpreter

simultaneously), the interpreter also

forces the GIL to be acquired by a

thread before it can access the

interpreter and stack and can modify

Python objects in memory all

willy-nilly. The latter point is why

the GIL exists: The GIL prevents

simultaneous access to Python objects

by multiple threads. But this does not

save you (as illustrated by the Bank

example) from being a lock-sensitive

creature; you don’t get a free ride.

The GIL is there to protect the

interpreters memory, not your sanity.

See the Global Interpreter Lock section of Jesse Noller's post for more details.

To get around this problem, check out Python's multiprocessing module.

multiple processes (with judicious use

of IPC) are[...] a much better

approach to writing apps for multi-CPU

boxes than threads.