some comments on python and multi-core

since my first time to dabble at python, i began to keep an eye on its performance in multi-core enviornment. As everyone knows, python adapt GIL to prevent thread from interrupting the data.

the pros is learning and using thread become easy and error-less, the downside is you are unable to take any benefit from using multi-core/cpu. A few years ago, i thought probably the next generation of python (next generation of python refers to python 3000 in context) may remove this bar. However, it is not true.

 

There are few ways of bypassing the GIL and taking benefit from multi-core, after years study and search, i conclude some ways which are listed as follow:

 

1. use OS-related API to write your code where you need multi-thread, and wrap it as module. the downside is the development speed is slowly and error-prone. using os api let cross-platform become more difficult.

 

2. use multi-process instead of multi-thread. If you live in Linux, you will be fine with this choice. The python provide os.fork for multi-process usage, furthmore, you may also use a package called pyprocessing. The usage is similar with threading package. The author claims that the package can be used in both of *nix and win32. However, after a few days of use in win32,  i found out that it probably was not a feasible choice. The restrictions of pyprocess in win32 are more tense than Linux, the argument, shared object, or data put in a Queue should be pickable (io serialization), that imply some un-plain objects like com-object, proxy-object can not be passed or communicate interprocess. It quickly become a harassment. I personally don't recommend use this package in production development.

 

3. it is said that use twisted and prospective broker can take benefit from multi-core, however, i havent' gave it a try. I found someone reported that the use of multi-core in twisted is boloney.

If your app is related to server, it is not a bad choice

 

4. the final choice is to use jython or ironPython instead, these versions of python are based on VM/CLR which supports multi-thread naturally.