python编程图像处理_用于多处理的高级python python编程指南-CSDN博客

python编程图像处理

One of the quickest and safest techniques to scale up the application is to launch multiple Python processes in an application. This also helps us by-pass the famous GIL issue.

扩展应用程序最快，最安全的技术之一就是在应用程序中启动多个Python进程。这也有助于我们绕过著名的GIL问题。

The challenge of launching multiple processes is that it can cause unintended problems if the application is not designed appropriately.

启动多个进程的挑战在于，如果应用程序设计不当，可能会引起意想不到的问题。

I have written 12 top guidelines that I recommend everyone to follow. This article aims to outline the programming guidelines for multiple processing programming.

我已经写了12条最重要的准则，建议大家都遵循。本文旨在概述多处理编程的编程准则。

I recommend this article to everyone who is/or intends in using the Python programming language.

我向所有打算使用Python编程语言的人推荐这篇文章。

If you want to understand the Python programming language from the beginner to an advanced level then I highly recommend the article below:

如果您想从入门到高级地了解Python编程语言，那么我强烈建议您阅读以下文章：

准则1 (Guideline 1)

Regardless of the platform, whether you are using Windows or Linux, try to avoid sharing data between processes as much as possible. If you really have to share data then ensure the data size is as small as possible.

无论使用哪种平台，无论使用Windows还是Linux，都应尽量避免在进程之间共享数据。如果您确实必须共享数据，请确保数据大小尽可能小。

准则2 (Guideline 2)

If you have to share data then attempt to use queues or pipes to exchange data between the processes rather than using lower-level synchronization primitives.

如果必须共享数据，则尝试使用队列或管道在进程之间交换数据，而不是使用较低级别的同步原语。

Therefore, I don’t recommend using the lower level synchronization primitives that are available in the threading module.

因此，我不建议使用threading模块中可用的较低级别的同步原语。

准则3 (Guideline 3)

Make sure the arguments that you are passing to the processes are picklable. In particular, for Windows, ensure that the subclass Process instances are picklable when the start() method is called.

确保传递给流程的参数是可挑剔的。特别是，对于Windows，请确保在调用start()方法时，子类Process实例是可腌制的。

准则4 (Guideline 4)

Ensure your proxy objects are thread-safe.

确保代理对象是线程安全的。

If you are going to use a proxy object and your code has multiple threads then ensure the proxy object is protected with a lock. Having said that, if your application is multiprocessing in nature but uses a single thread within each process then it is not a problem and you don’t have to protect the proxy objects with a lock.

如果要使用代理对象，并且代码具有多个线程，则请确保代理对象受锁保护。话虽如此，如果您的应用程序本质上是多处理的，但在每个进程中使用单个线程，那么这不是问题，并且您不必使用锁来保护代理对象。

For more information on the multiprocessing library and to understand multiprocessing from the very basics, I highly recommend reading this article:

有关多处理库的更多信息并从最基本的方面理解多处理，我强烈建议阅读本文：

准则5 (Guideline 5)

Before your main process completes its execution, explicitly join all of the processes otherwise the processes can become zombie processes until a new process is started. This will ensure your application does not have any memory leaks.

在主流程完成其执行之前，请显式加入所有流程，否则这些流程可能会变成僵尸流程，直到启动新流程为止。这将确保您的应用程序没有任何内存泄漏。

准则6 (Guideline 6)

Don’t terminate a process abruptly. If your application uses shared resources that are being used by the terminated process then they will become broken and unavailable to be used by other processes.

不要突然终止进程。如果您的应用程序使用了终止进程正在使用的共享资源，那么它们将被破坏，无法被其他进程使用。

准则7 (Guideline 7)

If your process is using a queue to exchange shared data between processes then make sure all of the items in the queue are removed before the process is joined.

如果您的流程正在使用队列在流程之间交换共享数据，请确保在加入流程之前删除队列中的所有项目。

If you want to read more about the synchronization primitives then read this article:

如果您想了解有关同步原语的更多信息，请阅读本文：

准则8 (Guideline 8)

Avoid creating global variables to share states. Global variables can be used as module-level constants but if the child processes attempt to access a global variable that is being updated by another variable then it can cause inconsistent behavior in the application.

避免创建全局变量以共享状态。全局变量可以用作模块级常量，但是如果子进程尝试访问由另一个变量更新的全局变量，则可能导致应用程序中的行为不一致。

I recommend passing the object as an argument to the constructor of the child process. This will prevent the garbage collector from removing the object as long as the child process is alive.

我建议将对象作为参数传递给子进程的构造函数。只要子进程处于活动状态，这将防止垃圾收集器删除该对象。

准则9 (Guideline 9)

Whenever we use the multiprocessing library to start a new process, it imports the main module in a new Python interpreter. Therefore it’s important to ensure the main module can be safely imported by a new Python interpreter without causing unintended side effects such as re-creating a new process and failing with a RuntimeError.

每当我们使用多重处理库启动新进程时，它都会在新的Python解释器中导入主模块。因此，重要的是要确保主模块可以由新的Python解释器安全地导入，而不会引起意外的副作用，如重新创建新进程和RuntimeError失败。

The key is to protect the entry point of the program by using:

关键是使用以下方法保护程序的入口点：

if __name__ == '__main__:
   # create new process

准则10 (Guideline 10)

As mentioned above, the shared objects that we pass to the queue/pipes are required to be picklable so that the child processes can use them. Having said that, I would encourage designing your solution in a way that you can inherit the shared object from the ancestor process. This will prevent deadlocks to occur in the application too.

如上所述，传递给队列/管道的共享对象必须是可挑剔的，以便子进程可以使用它们。话虽如此，我鼓励您以可以从祖先流程继承共享对象的方式设计解决方案。这也将防止死锁在应用程序中发生。

准则11 (Guideline 11)

If your application creates a large number of tasks then use the Pool class to create the processes for you. The Pool class will take care of creating, allocating memory, and terminating processes for you. The Pool only allocates memory for the executing processes.

如果您的应用程序创建了大量任务，请使用Pool类为您创建进程。 Pool类将为您创建，分配内存和终止进程。池仅为执行的进程分配内存。

准则12 (Guideline 12)

Leave your multiprocessing application running and assess the memory profile. If the memory footprint is constantly increasing then it means the application is leaking memory. Profile your application by assessing the objects it is creating and storing memory in.

使您的多处理应用程序运行并评估内存配置文件。如果内存占用量不断增加，则意味着应用程序正在泄漏内存。通过评估正在创建的对象并在其中存储内存来分析应用程序。

If you want to read more about how to profile your Python application then read this article:

如果您想了解有关如何配置Python应用程序的更多信息，请阅读本文：