python多线程加速for循环_Python多线程有限线程的for循环

我只是在学习Python，对Multithreading的了解也不多。我正在尝试通过Requests session.post方法发送一些json。我需要遍历字典的许多for循环底部的函数中调用此函数。

有没有办法让它并行运行？

我还必须限制线程的数量，否则发布调用将被阻塞，因为它们彼此之间会很快。帮助将不胜感激。

defdoWork(session,List,RefHashList):foritemRefHashinRefHashList:forequipmentinres['Response']['data']['items']:ifequipment['itemHash']==itemRefHash:ifequipment['characterIndex']!=0:SendJsonViaSession(session,getCharacterIdFromIndex(res,equipment['characterIndex']),itemRefHash,equipment['quantity'])

解决方案

首先，在不增加线程复杂性的情况下，以不同的方式构造代码可以提高速度。

defdoWork(session,res,RefHashList):forequipmentinres['Response']['data']['items']:i=equipment['itemHash']k=equipment['characterIndex']ifiinRefHashListandk!=0:SendJsonViaSession(session,getCharacterIdFromIndex(res,k),i,equipment['quantity'])

首先，我们将查找equipment['itemHash']并equipment['characterIndex']只有一次。

RefHashList您可以使用in运算符来代替显式循环。这会将循环移至速度更快的Python虚拟机中。

if除了嵌套条件之外，您还可以使用单个条件usingand。

注意：我已经删除了未使用的参数List，并将其替换为res。通常最好的做法是编写仅作用于给定参数而不是全局变量的函数。

Second, how much extra performance do you need? How much time is there on average between the SendJsonViaSession calls, and how small can this this time become before calls get blocked? If the difference between those numbers is small, it is probably not worth to implement a threaded sender.

Third, a design feature of the standard Python implementation is that only one thread at a time can be executing Python bytecode. So it is not certain that threading will improve performance.

Edit:

There are several ways to run stuff in parallel in Python. There is multiprocessing.Pool which uses processes, and multiprocessing.dummy.ThreadPool which uses threads. And from Python 3.2 onwards there is concurrent.futures, which can use processes or threads.

The thing is, neither of them has rate limiting. So you could get blocked for making too many calls.

Every time you call SendJsonViaSession you'd have to save the current time somehow so that all processes or threads can use it. And before every call, you would have to read that time and wait if it is too close to the last call.

Edit2:

If a call to SendJsonViaSession only takes 0.3 seconds, you should be able to do 3 calls/second sequentially. But your code only does 1 call/second. This implies that the speed restriction is somewhere else. You'd have to profile your code to see where the problem lies.