python多进程和进程池

写在最前面：

 

linux下可使用 fork 函数

通常使用 multiprocessing更常见

 

我们分别使用单进程和多进程处理run函数

# -*- coding: utf-8 -*-
import time,os
from multiprocessing import Pool


def run(n):
  time.sleep(1)
  print('Run child process %s (%s)...' % (n*n, os.getpid()))


if __name__ == "__main__":
  testFL = [1,2,3,4,5,6]
  print('顺序执行:')    # 顺序执行(也就是串行执行，单进程)
  starttime = time.time()
  for n in testFL:
    run(n)
  t = time.time()
  print("顺序执行时间：", int(t - starttime))

  print ('concurrent:') # 创建多个进程，并行执行
  pool = Pool(10)       # 创建拥有10个进程数量的进程池
  pool.map(run, testFL)
  pool.close()          # 关闭进程池，不再接受新的进程
  pool.join()           # 主进程阻塞等待子进程的退出
  endtime = time.time()
  print("并行执行时间：", int(endtime-t))

执行结果如下：

顺序执行:
Run child process 1 (32669)...
Run child process 4 (32669)...
Run child process 9 (32669)...
Run child process 16 (32669)...
Run child process 25 (32669)...
Run child process 36 (32669)...
顺序执行时间： 6
concurrent:
Run child process 4 (32671)...
Run child process 1 (32670)...
Run child process 16 (32673)...
Run child process 9 (32672)...
Run child process 25 (32674)...
Run child process 36 (32675)...
并行执行时间： 1

Process finished with exit code 0

 

如果不用map，一般采用apply_async(func[, args[, kwds[, callback]]])，这里是非阻塞的且支持结果返回后进行回调，

而apply用于传递不定参数，同python中的apply函数一致（不过内置的apply函数从2.3以后就不建议使用了），主进程会阻塞于函数。

# -*- coding: utf-8 -*-
import time,os
from multiprocessing import Pool


def run(n):
  time.sleep(1)
  print('Run child process %s (%s)...' % (n*n, os.getpid()))


if __name__ == "__main__":
  testFL = [1,2,3,4,5,6]
  print('顺序执行:')    # 顺序执行(也就是串行执行，单进程)
  starttime = time.time()
  for n in testFL:
    run(n)
  t = time.time()
  print("顺序执行时间：", int(t - starttime))

  print ('concurrent:') # 创建多个进程，并行执行
  pool = Pool(10)       # 创建拥有10个进程数量的进程池
  for i in range(1,7):
      pool.apply_async(run,(i,))
  pool.close()          # 关闭进程池，不再接受新的进程
  pool.join()           # 主进程阻塞等待子进程的退出
  endtime = time.time()
  print("并行执行时间：", int(endtime-t))

 

执行结果如下：

顺序执行:
Run child process 1 (32880)...
Run child process 4 (32880)...
Run child process 9 (32880)...
Run child process 16 (32880)...
Run child process 25 (32880)...
Run child process 36 (32880)...
顺序执行时间： 6
concurrent:
Run child process 9 (32885)...
Run child process 1 (32883)...
Run child process 25 (32887)...
Run child process 16 (32886)...
Run child process 4 (32884)...
Run child process 36 (32888)...
并行执行时间： 1

Process finished with exit code 0