读书笔记:《流畅的Python》第18章 使用asyncio处理并发

# 第18章 使用asyncio处理并发
"""
并发:同时发生
    关于结构
    用于制定方案
并行:同时进行,真正的并行只能是多个cpu核心,一个核心同一时刻只能做一件事情
    关于执行
    用来解决可能并行的问题
asyncio包使用事件循环驱动的协程实现并发
本章内容提要:
    1.对比一个简单的多线程程序和对应的asyncio版,说明多线程和异步任务之间的关系
    2.asyncio.Future和concurrent.futures.Future类之间的区别
    3.17章中下载国旗示例异步版
    4.抛弃线程或进程,如何使用异步编程管理网络应用中的高并发
    5.在异步编程中,与回调相比,协程显著提高性能的方式
    6.如何把阻塞的操作交给线程池处理,从而避免阻塞事件循环
    7.使用asyncio编写服务器,重新审视web应用对高并发的处理方式
    8.为什么asyncio已经准备好对Python生态系统产生重大影响
"""

# 18.1 线程和协程的对比
# 示例 18-1 spinner_thread.py:通过线程以动画形式显示文本式旋转指针
# 示例 18-2 spinner_asyncio.py:通过协程以动画形式显示文本式旋转指针

# 在python控制台或者小型测试脚本中实验future和协程的代码片段
"""
import asyncio
def run_sync(coro_or_future):
    loop = asyncio.get_event_loop()
    return loop.run_until_complete(coro_or_future)
>>> a = run_sync(some_coroutine())
"""

# 18.2使用asyncio和aiohttp下载
# 示例18-5 flags_asyncio.py:使用asyncio和aiohttp包实现异步下载

# 18.3避免阻塞型调用

# 18.4 改进asyncio下载脚本

# 18.4.1使用asyncio.as_completed函数
# 18.4.2 使用Executor对象,防止阻塞事件循环

# 示例18-9 flags_asyncio_exeutor.py:使用默认的TreadingPoolExecutor对象运行save_flag函数


# 18.5从回调到future和协程
# 示例18-10 JavaScript中的回调地狱:嵌套匿名函数,也称为灾难金字塔
"""
api_call1(request1,function(response1){
    //第一步
    var request2 = step1(response1);
    api_call2(request2,function(response2){
        //第二步
        var request3 = step2(response2);
        api_call2(request3,function(response3){
            //第三步
            step3(response3)
}) ;  
});
});"""

# 示例18-11展示python中的回调地狱是什么样子,链式回调
"""
def stage1(reponse1):
    request2 = step1(response1)
    api_call2(request2,stage2)
def stage2(reponse2):
    request3 = step2(response2)
    api_call3(request3,stage3)
def stage3(reponse3):
    step3(response3)
    api_call1(request1,stage1)
"""

# 示例18-12:使用协程和yield from 结构做异步编程,无需使用回调
"""
@asyncio.coroutine
def three_stages(request1):
    response1 = yield from api_call1(request1)
    # 第一步
    request2 = step1(response1)
    response2 = yield from api_call2(request2)
    # 第二步
    request3 = step2(response2)
    response2 = yield from api_call3(request3)
    # 第三步
    step(respose3)
loop.create_task(three_stages(request1))
"""

# 示例18-13 flags3_asyncio.py:再定义几个协程,把职责委托出去,每次下载国旗时,发起两次请求,以获取json中的国家名称

# 18.6 使用asyncio包编写服务器

# 18.6.1 使用asyncio包编写TCP服务器
# 示例18-18 http_charfinder.py

#本章后面的内容主要介绍了asyncio包的使用,但是在python3.8以后不再支持@asyncio.coroutine
# 装饰器,而是使用新语法 async def...但这种句法又不支持yield from
# 所以此处有待后续专门针对asyncio包进行单向突破
# 后面的代码示例为复制的作者在guithub上的源码

# 示例 18-1 spinner_thread.py:通过线程以动画形式显示文本式旋转指针

# 示例 18-1 spinner_thread.py:通过线程以动画形式显示文本式旋转指针
import threading
import itertools
import time
import sys

class Signal: # 1
    go = True

def spin(msg,signal): # 2
    write,flush = sys.stdout.write,sys.stdout.flush
    for char in itertools.cycle("|/-\\"):  # 3
        status = char+' '+msg
        write(status)
        flush()
        write('\x08' * len(status))  # 4
        time.sleep(.1)
        if not signal.go:  # 5
            break
    write(" " * len(status)+ '\x08'*len(status)) # 6

def slow_function():  #7
    # 假装等待I/O一段时间
    time.sleep(3)  # 8
    return 42

def supervisor():  # 9
    signal = Signal()
    spinner = threading.Thread(target=spin,
                               args=("thinking!",signal))
    print('spinner object:',spinner)  # 10
    spinner.start() # 11
    result = slow_function()  #12
    signal.go = False  # 13
    spinner.join()  # 14
    return result

def main():
    result = supervisor() # 15
    print('Answer:',result)

if __name__ == '__main__':
    #注意,这个脚本需要在cmd中执行才能显示旋转指针动画
     main()

"""
解释:
    1.这个类定义了一个简单的可变对象,其中有个go属性,用于从外部控制线程
    2.这个函数会在单独的线程运行,signal参数是Signal类的实例
    3.这其实是一个无限循环,因为cycle会从指定序列中反复不断地生成元素
    4.这是显示动画的诀窍所在,使用'\x08'退格符,把光标移回来
    5.如果go的属性不是True,则退出循环
    6.使用空格清除状态信息,然后把光标移回来
    7.假定这是耗时的计算
    8.调用sleep函数会阻塞主线程,不过一定要这么做,以便释放GIL,创建从属线程
    9.这个函数设置从属线程,显示线程对象,运行耗时的计算,最后杀死线程
    10.显示从属线程对象,输出类似于<Thread(Thread-1, initial)>
    11.启动从属线程
    12运行slow_function函数,阻塞主线程,同时从属线程以动画形式显示旋转指针
    13.改变signal的状态,这回终止spinner函数中的循环
    14.等待spinner线程结束
    15.运行supervisor函数
    
"""

# 示例 18-2 spinner_asyncio.py:通过协程以动画形式显示文本式旋转指针

# 示例 18-2 spinner_asyncio.py:通过协程以动画形式显示文本式旋转指针
import asyncio
import itertools
import sys
@asyncio.coroutine  # 1  注意:python3.8已经弃用了这种写法
def spin(msg): # 2   async def 这是python3.8以后的推荐写法
    write,flush = sys.stdout.write,sys.stdout.flush
    for char in itertools.cycle("|/-\\"):
        status = char+' '+msg
        write(status)
        flush()
        write('\x08' * len(status))
        try:
            yield from asyncio.sleep(.1)  # 3
            # yield  asyncio.sleep(.1)  # 3 async def中不允许有yield from 和return
        except asyncio.CancelledError:  # 4
            break
    write(" " * len(status)+ '\x08'*len(status))

@asyncio.coroutine
def slow_function():  #5
    # 假装等待I/O一段时间
    yield from asyncio.sleep(3)  # 6
    return 42
@asyncio.coroutine
def supervisor():  #7
    # spinner = asyncio.async(spin('thinking!'))  # 8  3.8好像不支持这个方法了
    spinner = asyncio.Task(spin('thinking!'))  # 8
    print('spinner object:',spinner)  # 9
    result = yield from slow_function()  #10
    spinner.cancel()  # 11
    return result

def main():
    loop = asyncio.get_event_loop() #12
    result = loop.run_until_complete(supervisor()) # 13
    loop.close()
    print('Answer:',result)

if __name__ == '__main__':
    #注意,这个脚本需要在cmd中执行才能显示旋转指针动画
     main()

"""
解释:
    1.打算交给asyncio处理的协程要使用@asyncio.coroutine装饰,这不是强制要求,但是强烈建议
    2.这里不需要用来关闭线程的signal参数
    3.使用yield from asyncio.sleep(.1)代替time.sleep(.1)这样的休眠不会阻塞事件循环
    4.如果spin函数苏醒后抛出asyncio.CancelledError异常,其原因是发出了取消请求
    5.现在slow_function()是协程,在用休眠假装进行I/O操作时,使用yield from执行时间循环
    6.from asyncio.sleep(3)把控制权交给主循环,在休眠结束后恢复这个协程
    7.supervisor()也是协程,因此可以使用yield from slow_function()
    8.asyncio.async()使用一个Task对象包装spin协程,并立即返回
        python3.8以后asyncio.async()是个错误语法,可以使用Task类实例化
    9.显示Task对象 类似于<Task pending name='Task-2' coro=<spin()
    10.驱动slow_function()函数,结束后获取返回值,同时,事件循环继续运行,
        因为slow_function()最后使用from asyncio.sleep(3)把控制权交给主循环
    11.Task对象可以取消,取消后会在协程当前暂停的yield初抛出asyncio.CancelledError异常
        协程可以捕获这个异常,也可以延迟取消,甚至拒绝取消
    12.获取事件循环的引用
    13.驱动supervisor协程,让它运行完毕,这个协程的返回值是这次调用的返回值



"""

# 示例18-5 flags_asyncio.py:使用asyncio和aiohttp包实现异步下载

# 示例18-5 flags_asyncio.py:使用asyncio和aiohttp包实现异步下载
import asyncio
import aiohttp  # 1
from flags import BASE_URL,save_flag,show,main # 2

@asyncio.coroutine # 3
def get_flag(cc):
    url = '{}/{cc}/{cc}.gif'.format(BASE_URL,cc=cc.lower())
    resp = yield from aiohttp.request('GET',url)  # 4
    image = yield from resp.read()  # 5
    return image

@asyncio.coroutine
def download_one(cc):  # 6
    image = yield from get_flag(cc)  # 7
    show(cc)
    save_flag(image,cc.lower()+'.gif')
    return cc

def download_many(cc_list):
    loop = asyncio.get_event_loop()  # 8
    to_do = [download_one(cc) for cc in sorted(cc_list)]  # 9
    wait_coro = asyncio.wait(to_do)  # 10
    res,_ = loop.run_until_complete(wait_coro)  # 11
    loop.close()  # 12
    return len(res)

if __name__ == '__main__':
    main(download_many)

# 示例18-9 flags_asyncio_exeutor.py:使用默认的TreadingPoolExecutor对象运行save_flag函数

# 示例18-9 flags_asyncio_exeutor.py:使用默认的TreadingPoolExecutor对象运行save_flag函数
import asyncio
import collections
import aiohttp
from aiohttp import web
import tqdm
from flags2_common import main,HTTPStatus,Result,save_flag
from flags2_asyncio import *
@asyncio.coroutine
def download_one(cc,base_url,semaphore,verbose):
    try:
        with (yield from semaphore):
            image = yield  from get_flag(base_url,cc)
    except web.HTTPNotFound:
        status = HTTPStatus.not_found
        msg = 'not found'
    except Exception as exc:
        raise FetchError(cc) from exc
    else:
        loop = asyncio.get_event_loop()
        loop.run_in_executor(None,
                             save_flag,image,cc.lower()+".gif")
        status = HTTPStatus.ok
        msg = 'ok'
    if verbose and msg:
        print(cc,msg)
    return Result(status,cc)

# flags2_asyncio.py

# flags2_asyncio.py
import asyncio
import collections
import aiohttp
from aiohttp import web
import tqdm
from flags2_common import main,HTTPStatus,Result,save_flag

# 默认设置较小的值,防止远程网络出错
DEFAULT_CONCUR_REQ = 5
MAX_CONCUR_REQ = 1000

class FetchError(Exception):
    def __init__(self,country_code):
        self.country_code = country_code

@asyncio.coroutine
def get_flag(base_url,cc):
    url = '{}/{cc}/{cc}.gif'.format(base_url, cc=cc.lower())
    resp = yield from aiohttp.request('GET', url)
    if resp.status ==200:
        image = yield from resp.read()
        return image
    elif resp.status == 404:
        raise web.HTTPNotFound()
    else:
        raise aiohttp.HTTPProcessingError(
            code = resp.status,message=resp.reason,
            headers = resp.headers
        )
@asyncio.coroutine
def download_one(cc,base_url,semaphore,verbose):
    try:
        with (yield from semaphore):
            image = yield  from get_flag(base_url,cc)
    except web.HTTPNotFound:
        status = HTTPStatus.not_found
        msg = 'not found'
    except Exception as exc:
        raise FetchError(cc) from exc
    else:
        save_flag(image,cc.lower()+".gif")
        status = HTTPStatus.ok
        msg = 'ok'
    if verbose and msg:
        print(cc,msg)
    return Result(status,cc)

@asyncio.coroutine
def downloader_coro(cc_list,base_url,verbose,concur_req):
    counter = collections.Counter()
    semaphore = asyncio.Semaphore(concur_req)
    to_do = [download_one(cc,base_url,semaphore,verbose)
             for cc in sorted(cc_list)]
    to_do_iter = asyncio.as_completed(to_do)
    if not verbose:
        to_do_iter = tqdm.tqdm(to_do_iter,total=len(cc_list))
    for future in to_do_iter:
        try:
            res = yield from future
        except FetchError as exc:
            country_code = exc.country_code
            try:
                error_msg = exc.__cause__.args[0]
            except IndexError:
                error_msg = exc.__cause__.__class__.__name__
                if verbose and error_msg:
                    msg = '*** Error for {}: {}'
                    print(msg.format(country_code,error_msg))
                    status = HTTPStatus.error
        else:
            status = res.status
        counter[status] += 1
    return counter

def download_many(cc_list,base_url,verbose,concur_req):
    loop = asyncio.get_event_loop()
    coro = downloader_coro(cc_list,base_url,verbose,concur_req)
    counts = loop.run_until_complete(coro)
    loop.close()
    return counts

if __name__ == '__main__':
    main(download_many,DEFAULT_CONCUR_REQ,MAX_CONCUR_REQ)

# flags3_asyncio.py:
# 再定义几个协程,把职责委托出去,每次下载国旗时,发起两次请求,以获取json中的国家名称

# flags3_asyncio.py:
# 再定义几个协程,把职责委托出去,每次下载国旗时,发起两次请求,以获取json中的国家名称
import asyncio
import collections
import aiohttp
from aiohttp import web
import tqdm
from flags2_common import main,HTTPStatus,Result,save_flag
@asyncio.coroutine
def http_get(url):
    res = yield from aiohttp.request('GET', url)
    if res.status == 200:
        ctype = res.headers.get('Content-type','').lower()
        if 'json' in ctype or url.endswith('json'):
            data = yield from res.json()
        else:
            data = yield from res.read()
        return data
    elif res.status == 404:
        raise web.HTTPNotFound()
    else:
        raise aiohttp.HTTPProcessingError(
            code=res.status, message=res.reason,
            headers=res.headers)
@asyncio.coroutine
def get_country(base_url,cc):
    url = "{}/{cc}/metadata.json".format(base_url,cc=cc.lower())
    metadata = yield from http_get(url)
    return metadata['country']
@asyncio.coroutine
def get_flag(base_url,cc):
    url = '{}/{cc}/{cc}.gif'.format(base_url, cc=cc.lower())
    return (yield from http_get(url))

@asyncio.coroutine
def download_one(cc,base_url,semaphore,verbose):
    try:
        with (yield from semaphore):
            image = yield  from get_flag(base_url,cc)
        with (yield from semaphore):
            country = yield from get_country(base_url, cc)
    except web.HTTPNotFound:
        status = HTTPStatus.not_found
        msg = 'not found'
    except Exception as exc:
        raise FetchError(cc) from exc
    else:
        country = country.replace(' ','_')
        filename = '{}-{}.gif'.format(country,cc)
        loop = asyncio.get_event_loop()
        loop.run_in_executor(None,save_flag,image,filename)
        status = HTTPStatus.ok
        msg = 'ok'
    if verbose and msg:
        print(cc,msg)
    return Result(status,cc)

 作者源码:

charfinder.py

#!/usr/bin/env python3

"""
Unicode character finder utility:
find characters based on words in their official names.
This can be used from the command line, just pass words as arguments.
Here is the ``main`` function which makes it happen::
    >>> main('rook')  # doctest: +NORMALIZE_WHITESPACE
    U+2656  ♖  WHITE CHESS ROOK
    U+265C  ♜  BLACK CHESS ROOK
    (2 matches for 'rook')
    >>> main('rook', 'black')  # doctest: +NORMALIZE_WHITESPACE
    U+265C  ♜  BLACK CHESS ROOK
    (1 match for 'rook black')
    >>> main('white bishop')  # doctest: +NORMALIZE_WHITESPACE
    U+2657  ♗   WHITE CHESS BISHOP
    (1 match for 'white bishop')
    >>> main("jabberwocky's vest")
    (No match for "jabberwocky's vest")
For exploring words that occur in the character names, there is the
``word_report`` function::
    >>> index = UnicodeNameIndex(sample_chars)
    >>> index.word_report()
        3 SIGN
        2 A
        2 EURO
        2 LATIN
        2 LETTER
        1 CAPITAL
        1 CURRENCY
        1 DOLLAR
        1 SMALL
    >>> index = UnicodeNameIndex()
    >>> index.word_report(10)
    75821 CJK
    75761 IDEOGRAPH
    74656 UNIFIED
    13196 SYLLABLE
    11735 HANGUL
     7616 LETTER
     2232 WITH
     2180 SIGN
     2122 SMALL
     1709 CAPITAL
Note: characters with names starting with 'CJK UNIFIED IDEOGRAPH'
are indexed with those three words only, excluding the hexadecimal
codepoint at the end of the name.
"""

import sys
import re
import unicodedata
import pickle
import warnings
import itertools
import functools
from collections import namedtuple

RE_WORD = re.compile(r'\w+')
RE_UNICODE_NAME = re.compile('^[A-Z0-9 -]+$')
RE_CODEPOINT = re.compile('U\+([0-9A-F]{4,6})')

INDEX_NAME = 'charfinder_index.pickle'
MINIMUM_SAVE_LEN = 10000
CJK_UNI_PREFIX = 'CJK UNIFIED IDEOGRAPH'
CJK_CMP_PREFIX = 'CJK COMPATIBILITY IDEOGRAPH'

sample_chars = [
    '$',  # DOLLAR SIGN
    'A',  # LATIN CAPITAL LETTER A
    'a',  # LATIN SMALL LETTER A
    '\u20a0',  # EURO-CURRENCY SIGN
    '\u20ac',  # EURO SIGN
]

CharDescription = namedtuple('CharDescription', 'code_str char name')

QueryResult = namedtuple('QueryResult', 'count items')


def tokenize(text):
    """return iterable of uppercased words"""
    for match in RE_WORD.finditer(text):
        yield match.group().upper()


def query_type(text):
    text_upper = text.upper()
    if 'U+' in text_upper:
        return 'CODEPOINT'
    elif RE_UNICODE_NAME.match(text_upper):
        return 'NAME'
    else:
        return 'CHARACTERS'


class UnicodeNameIndex:

    def __init__(self, chars=None):
        self.load(chars)

    def load(self, chars=None):
        self.index = None
        if chars is None:
            try:
                with open(INDEX_NAME, 'rb') as fp:
                    self.index = pickle.load(fp)
            except OSError:
                pass
        if self.index is None:
            self.build_index(chars)
        if len(self.index) > MINIMUM_SAVE_LEN:
            try:
                self.save()
            except OSError as exc:
                warnings.warn('Could not save {!r}: {}'
                              .format(INDEX_NAME, exc))

    def save(self):
        with open(INDEX_NAME, 'wb') as fp:
            pickle.dump(self.index, fp)

    def build_index(self, chars=None):
        if chars is None:
            chars = (chr(i) for i in range(32, sys.maxunicode))
        index = {}
        for char in chars:
            try:
                name = unicodedata.name(char)
            except ValueError:
                continue
            if name.startswith(CJK_UNI_PREFIX):
                name = CJK_UNI_PREFIX
            elif name.startswith(CJK_CMP_PREFIX):
                name = CJK_CMP_PREFIX

            for word in tokenize(name):
                index.setdefault(word, set()).add(char)

        self.index = index

    def word_rank(self, top=None):
        res = [(len(self.index[key]), key) for key in self.index]
        res.sort(key=lambda item: (-item[0], item[1]))
        if top is not None:
            res = res[:top]
        return res

    def word_report(self, top=None):
        for postings, key in self.word_rank(top):
            print('{:5} {}'.format(postings, key))

    def find_chars(self, query, start=0, stop=None):
        stop = sys.maxsize if stop is None else stop
        result_sets = []
        for word in tokenize(query):
            chars = self.index.get(word)
            if chars is None:  # shortcut: no such word
                result_sets = []
                break
            result_sets.append(chars)

        if not result_sets:
            return QueryResult(0, ())

        result = functools.reduce(set.intersection, result_sets)
        result = sorted(result)  # must sort to support start, stop
        result_iter = itertools.islice(result, start, stop)
        return QueryResult(len(result),
                           (char for char in result_iter))

    def describe(self, char):
        code_str = 'U+{:04X}'.format(ord(char))
        name = unicodedata.name(char)
        return CharDescription(code_str, char, name)

    def find_descriptions(self, query, start=0, stop=None):
        for char in self.find_chars(query, start, stop).items:
            yield self.describe(char)

    def get_descriptions(self, chars):
        for char in chars:
            yield self.describe(char)

    def describe_str(self, char):
        return '{:7}\t{}\t{}'.format(*self.describe(char))

    def find_description_strs(self, query, start=0, stop=None):
        for char in self.find_chars(query, start, stop).items:
            yield self.describe_str(char)

    @staticmethod  # not an instance method due to concurrency
    def status(query, counter):
        if counter == 0:
            msg = 'No match'
        elif counter == 1:
            msg = '1 match'
        else:
            msg = '{} matches'.format(counter)
        return '{} for {!r}'.format(msg, query)


def main(*args):
    index = UnicodeNameIndex()
    query = ' '.join(args)
    n = 0
    for n, line in enumerate(index.find_description_strs(query), 1):
        print(line)
    print('({})'.format(index.status(query, n)))

if __name__ == '__main__':
    if len(sys.argv) > 1:
        main(*sys.argv[1:])
    else:
        print('Usage: {} word1 [word2]...'.format(sys.argv[0]))

tcp_charfinder.py

#!/usr/bin/env python3

# BEGIN TCP_CHARFINDER_TOP
import sys
import asyncio

from charfinder import UnicodeNameIndex  # <1>

CRLF = b'\r\n'
PROMPT = b'?> '

index = UnicodeNameIndex()  # <2>

@asyncio.coroutine
def handle_queries(reader, writer):  # <3>
    while True:  # <4>
        writer.write(PROMPT)  # can't yield from!  # <5>
        yield from writer.drain()  # must yield from!  # <6>
        data = yield from reader.readline()  # <7>
        try:
            query = data.decode().strip()
        except UnicodeDecodeError:  # <8>
            query = '\x00'
        client = writer.get_extra_info('peername')  # <9>
        print('Received from {}: {!r}'.format(client, query))  # <10>
        if query:
            if ord(query[:1]) < 32:  # <11>
                break
            lines = list(index.find_description_strs(query)) # <12>
            if lines:
                writer.writelines(line.encode() + CRLF for line in lines) # <13>
            writer.write(index.status(query, len(lines)).encode() + CRLF) # <14>

            yield from writer.drain()  # <15>
            print('Sent {} results'.format(len(lines)))  # <16>

    print('Close the client socket')  # <17>
    writer.close()  # <18>
# END TCP_CHARFINDER_TOP

# BEGIN TCP_CHARFINDER_MAIN
def main(address='127.0.0.1', port=2323):  # <1>
    port = int(port)
    loop = asyncio.get_event_loop()
    server_coro = asyncio.start_server(handle_queries, address, port,
                                loop=loop) # <2>
    server = loop.run_until_complete(server_coro) # <3>

    host = server.sockets[0].getsockname()  # <4>
    print('Serving on {}. Hit CTRL-C to stop.'.format(host))  # <5>
    try:
        loop.run_forever()  # <6>
    except KeyboardInterrupt:  # CTRL+C pressed
        pass

    print('Server shutting down.')
    server.close()  # <7>
    loop.run_until_complete(server.wait_closed())  # <8>
    loop.close()  # <9>


if __name__ == '__main__':
    main(*sys.argv[1:])  # <10>
# END TCP_CHARFINDER_MAIN

---

这一章内容读得好痛苦!