Python异步编程之asyncio

1.同步与异步

• 同步：指完成事务的逻辑，先执行第一个事务，如果阻塞了，会一直等待，直到这个事务完成，再执行第二个事务，顺序执行
• 异步：和同步相对的，异步是指在处理调用这个事务的之后，不会等待这个事务的处理结果，直接处理第二个事务去了，通过状态、通知、回调来通知调用者处理结果

2.相关知识

• event_loop:事件循环，程序开启一个无限的循环，程序员会把一些函数注册到事件循环上。当满足事件发生的时候，调用相应的协程函数
• coroutine：协程，协程对象，指一个使用async关键字定义的函数，它的调用不会立即执行函数，而是会返回一个协程对象。协程对象需要注册到事件循环，由事件循环调用
• task:任务，一个协程对象就是一个原生可以挂起的函数，任务则是对协程进一步封装，其中包含任务的各种状态
• future:代表将来执行或没有执行的任务的结果。它和task上没有本质的区别
• async/await:python3.5 用于定义协程的关键字，async定义一个协程，await用于挂起阻塞的异步调用接口

3.举例

（1）定义一个协程

通过async关键字定义一个协程（coroutine），协程也是一种对象。协程不能直接运行，需要把协程加入到事件循环（loop），由后者在适当的时候调用协程。asyncio.get_event_loop方法可以创建一个事件循环，然后使用run_until_complete将协程注册到事件循环，并启动事件循环

import asyncio

async def do_some_work():
    print('do some work')

coroutine = do_some_work()

loop = asyncio.get_event_loop()
loop.run_until_complete(coroutine)

（2）创建一个task

协程对象不能直接运行，在注册事件循环的时候，其实是run_until_complete方法将协程包装成为了一个任务（task）对象。所谓task对象是Future类的子类。保存了协程运行后的状态，用于未来获取协程的结果。

create_task 创建

import asyncio

async def do_some_work():
    print('do some work')

coroutine = do_some_work()
loop = asyncio.get_event_loop()

task = loop.create_task(coroutine)
print(task)
loop.run_until_complete(task)
print(task)

ensure_future创建

import asyncio

async def do_some_work():
    print("do some work")
    return 'done'

coroutine = do_some_work()
loop = asyncio.get_event_loop()
task = asyncio.ensure_future(coroutine)
loop.run_until_complete(task)
print('task result:',task.result())

(3)绑定回调函数

add_done_callback绑定一个回调函数，参数是future对象

import asyncio
import time

async def get_html(url):
    print('start get url')
    await asyncio.sleep(2)
    return 'ywh'

def callback(future):
    print(future.result())

start_time = time.time()
loop = asyncio.get_event_loop()
task = loop.create_task(get_html('http://www.baidu.com'))
task.add_done_callback(callback)
loop.run_until_complete(task)

(4)阻塞和await

使用async可以定义协程对象，使用await可以针对耗时的操作进行挂起，就像生成器里的yield一样，函数让出控制权。协程遇到await，事件循环将会挂起该协程，执行别的协程，直到其他的协程也挂起或者执行完毕，再进行下一个协程的执行

import asyncio

async def work1():
    print('work1 start')
    await asyncio.sleep(2) # 用sleep模拟IO操作
    print('work1 end')

async def work2():
    print('work2 start')
    print('work2 end')

coroutine1 = work1()
coroutine2 = work2()
loop = asyncio.get_event_loop()
task1 = asyncio.ensure_future(coroutine1)
task2 = asyncio.ensure_future(coroutine2)
loop.run_until_complete(task1)


work1 start
work2 start
work2 end
work1 end

(5)嵌套协程

使用async可以定义协程，协程用于耗时的io操作，我们也可以封装更多的io操作过程，这样就实现了嵌套的协程，即一个协程中await了另外一个协程，如此连接起来

import asyncio
import time
 
now = lambda: time.time()
async def do_some_work(x):
    print('Waiting: ', x)
    await asyncio.sleep(x)
    return 'Done after {}s'.format(x)
 
async def main():
    coroutine1 = do_some_work(1)
    coroutine2 = do_some_work(2)
    coroutine3 = do_some_work(4)
    tasks = [
        asyncio.ensure_future(coroutine1),
        asyncio.ensure_future(coroutine2),
        asyncio.ensure_future(coroutine3)
    ]
    dones, pendings = await asyncio.wait(tasks)
    for task in dones:
        print('Task ret: ', task.result())
start = now()
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
print('TIME: ', now() - start)


Waiting:  1
Waiting:  2
Waiting:  4
Task ret:  Done after 1s
Task ret:  Done after 2s
Task ret:  Done after 4s
TIME:  4.0100789070129395

如果使用的是asyncio.gather创建协程对象，那么await的返回值就是协程运行的结果。

results = await asyncio.gather(*tasks)
for result in results:
    print('Task ret: ', result)

或者可以使用as_completed方法

async def main():
    coroutine1 = do_some_work(1)
    coroutine2 = do_some_work(2)
    coroutine3 = do_some_work(4)
    tasks = [
        asyncio.ensure_future(coroutine1),
        asyncio.ensure_future(coroutine2),
        asyncio.ensure_future(coroutine3)
    ]
    for task in asyncio.as_completed(tasks):
        result = await task
        print('Task ret: {}'.format(result))
start = now()
loop = asyncio.get_event_loop()
done = loop.run_until_complete(main())
print('TIME: ', now() - start)

(6)取消协程

asyncio.Task获取事件循环的task

import asyncio
import time

now = lambda: time.time()
 
async def do_some_work(x):
    print('Waiting: ', x)
    await asyncio.sleep(x)
    return 'Done after {}s'.format(x)
 
coroutine1 = do_some_work(1)
coroutine2 = do_some_work(2)
coroutine3 = do_some_work(2)
 
tasks = [
    asyncio.ensure_future(coroutine1),
    asyncio.ensure_future(coroutine2),
    asyncio.ensure_future(coroutine3)
]
 
start = now()
loop = asyncio.get_event_loop()
try:
    loop.run_until_complete(asyncio.wait(tasks))
except KeyboardInterrupt as e:
    print(asyncio.Task.all_tasks())
    for task in asyncio.Task.all_tasks():
        print(task.cancel())
    loop.stop()
    loop.run_forever()
finally:
    loop.close()
 
print('TIME: ', now() - start)


Waiting:  1
Waiting:  2
Waiting:  2
{<Task pending coro=<do_some_work() running at 07asyncio_cancel.py:8> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x000001A8989A9A38>()]> cb=[_wait.<locals>._on_completion() at c:\python\Lib\asyncio\tasks.py:380]>, <Task pending coro=<wait() running at c:\python\Lib\asyncio\tasks.py:313> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x000001A8989A9BE8>()]>>, <Task pending coro=<do_some_work() running at 07asyncio_cancel.py:8> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x000001A898920708>()]> cb=[_wait.<locals>._on_completion() at c:\python\Lib\asyncio\tasks.py:380]>, <Task pending coro=<do_some_work() running at 07asyncio_cancel.py:8> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x000001A8989A9A98>()]> cb=[_wait.<locals>._on_completion() at c:\python\Lib\asyncio\tasks.py:380]>}
True
True
True
True
TIME:  1.0197539329528809

(7)新线程协程

def start_loop(loop):
    asyncio.set_event_loop(loop)
    loop.run_forever()
 
async def do_some_work(x):
    print('Waiting {}'.format(x))
    await asyncio.sleep(x)
    print('Done after {}s'.format(x))
 
def more_work(x):
    print('More work {}'.format(x))
    time.sleep(x)
    print('Finished more work {}'.format(x))
 
start = now()
new_loop = asyncio.new_event_loop()
t = Thread(target=start_loop, args=(new_loop,))
t.start()
print('TIME: {}'.format(time.time() - start))
 
asyncio.run_coroutine_threadsafe(do_some_work(6), new_loop)
asyncio.run_coroutine_threadsafe(do_some_work(4), new_loop)

主线程中创建一个new_loop，然后在另外的子线程中开启一个无限事件循环。主线程通过run_coroutine_threadsafe新注册协程对象。这样就能在子线程中进行事件循环的并发操作，同时主线程又不会被block。一共执行的时间大概在6s左右

不同线程的事件循环

当前线程创建一个事件循环，然后在新建一个线程，在新线程中启动事件循环。当前线程不会被block。

from threading import Thread
 
def start_loop(loop):
    asyncio.set_event_loop(loop)
    loop.run_forever()
 
def more_work(x):
    print('More work {}'.format(x))
    time.sleep(x)
    print('Finished more work {}'.format(x))
 
start = now()
new_loop = asyncio.new_event_loop()
t = Thread(target=start_loop, args=(new_loop,))
t.start()
print('TIME: {}'.format(time.time() - start))
 
new_loop.call_soon_threadsafe(more_work, 6)
new_loop.call_soon_threadsafe(more_work, 3)

启动上述代码之后，当前线程不会被block，新线程中会按照顺序执行call_soon_threadsafe方法注册的more_work方法，后者因为time.sleep操作是同步阻塞的，因此运行完毕more_work需要大致6 + 3

4.asyncio

异步代码举例：

import time
import asyncio

# 定义异步函数
async def hello():
    asyncio.sleep(1)
    print('Hello World:%s' % time.time())

def run():
    for i in range(5):
        loop.run_until_complete(hello())

loop = asyncio.get_event_loop()
if __name__ =='__main__':
    run()

Hello World:1555559847.5627491
Hello World:1555559847.5627491
Hello World:1555559847.5637255
Hello World:1555559847.5676293
Hello World:1555559847.5686066

同步代码对比:

import time

def hello():
    time.sleep(1)

def run():
    for i in range(5):
        hello()
        print('Hello World:%s' % time.time())  # 任何伟大的代码都是从Hello World 开始的！
if __name__ == '__main__':
    run()


Hello World:1555559763.6462507
Hello World:1555559764.6467445
Hello World:1555559765.6497967
Hello World:1555559766.6501706
Hello World:1555559767.6505506