python多进程实战[入门篇]

 
 

   前言：随着半导体工艺技术的飞速进步和体系结构的不断发展，CPU设计趋势已逐步从单核向多核演进，现今，多核处理器广泛普及，但实际上多核利用率不高，原因大多数是软件没有针对多核心优化。
   本章实战多进程素数计算 ，对比单进程计算方式有着非常明显的效率提升，希望能给予你一定的启发。

更多精彩，点我传送 >>>>> 梦马韶华CSDN主页
 
 

一. 基于python内置库multiprocess.Process的多进程计算

 

1. 实战：充分利用CPU的多核心、多进程计算1～10000000里的素数

 

# coding = utf-8
# @Author  : TerryJay
# @File    : multi_process.py
# @Software: PyCharm
# @Time    : 2019/5/24 21:54
# I believe that the God rewards the diligent.

import os
import time
from multiprocessing import Process
from math import sqrt


class MyProcess(Process):
	def __init__(self, name, start_n, end_n):
		super().__init__()
		self.name = name
		self.start_n = start_n
		self.end_n = end_n
		self.calc_result = []

	def loops_calc(self):
		for p in range(int(self.start_n), int(self.end_n)):
			if PrimeCalculation.is_prime(p):
				self.calc_result.append(p)
		return self.calc_result

	def run(self):
		print(self.name, "is run, start time is ---", time.time())
		self.loops_calc()
		print(self.name, "is end, completion time is ---", time.time())


class PrimeCalculation:
	def __init__(self, number):
		self.process_num = os.cpu_count()
		self.number = number
		self.calc_block = self.number / self.process_num

		"""任务切片，根据你的cpu实际线程数决定分成多少份，例如：4核8线程，则分为8份"""

	def task_assign(self):
		work_list = list()
		start_num = 1
		slice_num = self.calc_block
		for p in range(self.process_num):
			work_list.append([start_num + p * slice_num, slice_num + p * slice_num])

		return work_list

	"""计算素数"""

	@staticmethod
	def is_prime(n):
		if n == 1:
			return False
		for num in range(2, int(sqrt(n)) + 1):
			if n % num == 0:
				return False
		return True


if __name__ == '__main__':
	print(os.cpu_count())
	main_start_time = time.time()
	f = PrimeCalculation(10000000)

	process_list = []
	work_l = f.task_assign()
	for i in range(len(work_l)):
		m = MyProcess(str(i), work_l[i][0], work_l[i][1])
		process_list.append(m)

	for prs in process_list:
		# prs.daemon = True
		prs.start()

	"""主进程循环检查子进程活动状态,全部为False后程序退出"""
	while process_list:
		for prs in process_list:
			if not prs.is_alive():
				process_list.remove(prs)
	print("所有进程已结束，程序总耗时>>>>>", time.time() - main_start_time)

1.1 多进程 [os.cpu_count()=12] 计算的总耗时 [单位: 秒]

所有进程已结束，程序总耗时>>>>> 27.244723081588745

 
 

2. 实战：单进程计算1～10000000里的素数

 

# coding = utf-8
# @Author  : TerryJay
# @File    : single_process.py
# @Software: PyCharm
# @Time    : 2019/5/24 21:54
# I believe that the God rewards the diligent.

import time
from math import sqrt


class PrimeCalculation:

    @staticmethod
    def is_prime(n):
        if n == 1:
            return False
        for i in range(2, int(sqrt(n)) + 1):
            if n % i == 0:
                return False
        return True

    """计算素数"""

    @staticmethod
    def loops_calc(start_n, end_n):
        calc_result = []
        start_time = time.time()
        for p in range(start_n, end_n):
            if PrimeCalculation.is_prime(p):
                calc_result.append(p)
        # print(calc_result)
        print("单进程计算已完成，总耗时>>>>>", time.time() - start_time)


if __name__ == '__main__':
    PrimeCalculation.loops_calc(1, 10000000)

2.1 单进程计算的总耗时 [单位: 秒]

单进程计算已完成，总耗时>>>>> 125.80246877670288

 

3. 结果分析：多进程和单进程的耗时对比：

计算素数	单进程计算耗时(s)	多进程计算耗时(s)
1～10000000	125.80246877670288	9.129996061325073

 

4. 本章测试代码所用的硬件、软件环境

系统	CPU	python版本
MacOS 10.14.3	i7 8700t 6c12t	3.6.8