一种在 python 中用 asyncio 和协程实现的IO并发

最新推荐文章于 2022-02-24 23:04:12 发布

竹影半墙

最新推荐文章于 2022-02-24 23:04:12 发布

阅读量7.1k

点赞数

本文链接：https://blog.csdn.net/misskissC/article/details/103549169

版权

都市专栏收录该内容

56 篇文章 3 订阅

订阅专栏

An IO-concurrency implement by async && coroutine in python

1 此篇文字围绕的主题

基于IO驱动程序¹，本机从IO读取或发送数据——读/写IO的有效方法。

+--------------------------+     +----+
|                          | +==>|    |
|            +--+    +---+ | |   +----+ 1
| +-----+    |  |    | I | | |      .
| | CPU |<==>|MM|<==>|   |<==+      .
| +-----+    |  |    | O | | |      .
|            +--+    +---+ | |   +----+
|                          | +==>|    |
+--------------------------+     +----+ n
local-computer               remote-computers

MM: memory。
<==>: 数据流向。

2 IO多路复用 ≈ IO并发

基于IO驱动程序对IO的读/写，可看作是对IO缓冲区的读/写。

+--------------------------------------+       +----+
|                                      |  +===>|    |
|                        +--+    +---+ |  |1路 +----+ 1
|                    +==>|b1|<==>|   | |  |
| +-----+ concurrency|   |--|    | I | |  |     .
| | CPU |<===========+   |MM|    |   |<===+     .
| +-----+            |   |--|    | O | |  |     .
|                    +==>|bn|<==>|   | |  |   
|                        +--+    +---+ |  |    +----+
|                                      |  +===>|    |
+--------------------------------------+   n路 +----+ n
local-computer                            remote-computers

b: IO buffer。

本机IO能实时接收远程计算机传输来的数据，（对于诸如网卡类IO，）当数据目的地为本机时其会被上传到IO缓冲区。

IO多路复用指本机与远程计算机建立多路数据传输的软连接（每一路数据对应一个IO缓冲区），CPU通过调度CPU并发单位² 并发读/写IO缓冲区。

由CPU调度CPU并发单位读/写IO缓冲区实现了IO硬件并发传输各路数据。相比于CPU层面的并发，IO并发粒度更大（一定程度上削弱了IO的并发现象）。此篇文字将IO多路复用称为大粒度并发——这是本小节标题“IO多路复用 ≈ IO并发”的由来。

另外，此篇文字认为构成IO并发粒度比CPU并发粒度大的原因主要有三个。
[1] 数据粒度——让IO一次性（不切换）发送某路数据的量较大（如一整个缓冲区）；
[2] IO速度比CPU低几个数量级；
[3] 较本机数据的传输时间，来自远程计算机数据的传输时间较长（相比前两个原因，这个是主要原因，可参考后续程序例子）。
利用IO并发时主要是为了提升IO利用率——当某路数据中断时去处理另一路数据。

3 IO同步并发 && IO异步并发

IO同步指一个CPU并发单位会阻塞等待IO缓冲区可用并完成对IO的读/写。

IO异步指若IO缓冲区可用则完成对IO的读/写；若IO缓冲区不可用则跳过IO读/写继续调度并发单位后续指令运行，待IO缓冲区可用时再完成对其的读/写操作。

3.1 IO同步并发

通过多个CPU并发单位读/写某IO下的多路IO缓冲区可实现IO同步并发。

当无不可用IO缓冲区时，各IO缓冲区由CPU调度CPU并发单位并发读/写；当当前CPU并发单位读/写的IO缓冲区不可用时则调度另一CPU并发单位尝试读/写相应的IO缓冲区。

如由n个线程分别读/写m(m >= n)路IO数据的IO缓冲区。

3.2 IO异步并发

由于CPU与IO的速度差异³，除能通过CPU多并发单位实现IO异步并发外，单CPU并发单位也可以实现异步IO在IO层面的并发。

最直接的方式是让CPU轮询读/写IO缓冲区：

for (io = data_conn->start; ; io = data_conn->next)
    read or write current io buffer
    create one data connection when data connections are less than max

考虑本机IO利用率满（远程计算机将本机IO填满且都上传到IO缓冲区）的场景，当本机可调度并发量未饱和情况下，轮询方式可以满足异步IO并发的读/写。

4 IO并发——IO异步与协程的搭配

若单CPU并发已能满足当前场景下的IO并发，那么实现目标最友好方式就是“单CPU并发+IO异步”的方式啦⁴。

对于比“CPU轮询+IO异步”稍复杂的IO并发业务，可尝试在单CPU并发单位内使用协程。协程可在应用程序中创建和调度。除具调度式并发特点外，其还有两个优点：
[1] 对CPU层面并发单位的友好性——其创建和调度开销不在内核上；
[2] 上下文切换开销比线程更小，协程执行效率可接近于3.2中的轮询方式。

额，到此为止，这篇文字个人理解文字的戏也太多了点吧…在接下来的篇章里不如直接写个例子算了。

5 写个小例子吧

用什么写个什么呢？

用python写个请求OneNET视频平台部分HTTP-APIs的例子吧。嗯，这种网络IO场景适合用“IO异步+协程”来实现。

5.1 初定功能

[1] 比较OneNET-Video-Platform-HTTP-APIs 同步请求和异步请求的时间差异；
[2] 在异步情况下保证某些HTTP-APIs请求的先后顺序；
[3] 提供参数配置入口，如命令行参数，配置文件。

5.2 asyncio && aiohttp

此文使用python的asyncio和aiohttp来完成HTTP-API的请求。这两个模块对IO异步的设置、协程实现和调度进行了封装，下一篇文字再探讨协程相关内容吧。

#!/usr/bin/python3
# -*- coding: utf-8 -*-

"""
asyctn.py, namely async coroutine, 
wrapper-functions for aiohttp contained.

lxr, 2019.12.15
"""

import json
import asyncio
try:
    from aiohttp import ClientSession
except:
    exit('you should run following command in your own environment ' + 
         'to install the third party libraries\npip3(or pip) install -r requirements.txt')

_index = 0
    
async def _w_method(asyctn_method, url, 
            hdrs=None, params=None, data=None, timeout=None):
    ''' wrapper for the asyctn_method,e.g. aiohttp.ClientSession.get '''
    async with asyctn_method(url     = url,
                             headers = hdrs,
                             params  = params,
                             data    = data,
                             timeout = timeout) as resp:
        try: return await resp.text(encoding='utf-8')
        except Exception as e: print(e)
    
    return None

def _method_mapping(session, method):
    ''' mapping string 'get', 'post', 'delete', 'put' to session.get, 
        session.post, session.delete, session.put respectively.
    '''
    if   method == 'get':
        return session.get
    elif method == 'post':
        return session.post
    elif method == 'delete':
        return session.delete
    elif method == 'put':
        return session.put
    else:
        return None
        
async def w_request(url, method,
            hdrs=None, params=None, data=None, timeout=None):
    ''' wrapper for aiohttp.ClientSession.method(),
        make it be coroutine && async
    '''
    global _index
    
    async with ClientSession() as session:
        method = _method_mapping(session, method)
        _index += 1
        index = _index
        try: resp = await _w_method(method, url, 
                                hdrs=hdrs, params=params,
                                data=data, timeout=timeout)
        except Exception as e: print(e);return None
        return resp

class AsyCtnLoop():
    ''' wrapper for asyncio '''
    def __init__(self):
        try:
            self.loop = asyncio.get_event_loop()
        except Exception as e:
            exit(e)

    def w_loop(self, requests):
        ''' to run ont requests asynchronously by asyncio '''
        try:
            self.loop.run_until_complete(asyncio.wait(requests))
        except Exception as e: 
            print(e)
        else:
            print('')
        
    def w_close(self):
        self.loop.close()

if __name__ == '__main__':
    ''' for testing asyctn.py '''
    async def w_req_test_f(index):
        resp = await w_request('https://mijisou.com/', 'get')
        print(f'{index} {resp[0:3] if resp != None else None}')
    async def w_req_test_s(index):
        resp = await w_request('https://mijisou.com/', 'get')
        print(f'{index} {resp[0:3] if resp != None else None}')
        
    requests_f = []
    requests_s = []
    for i in range(5):
        requests_f.append(w_req_test_f(i + 1))
    for i in range(5, 10):
        requests_s.append(w_req_test_s(i + 1))
    
    asyctn_loop = AsyCtnLoop()
    asyctn_loop.w_loop(requests_f)
    asyctn_loop.w_loop(requests_s)
    asyctn_loop.w_close(

由于asyctn.py包含第三方模块aiohttp，为了给使用者在缺乏aiohttp库时一个友好提示，此文用pipreqs为本python工程生成了requirements.txt。

先看看asyctn.py的运行体验吧。

> python asyctn.py
2 <!DOCTYPE
1 <!DOCTYPE
4 <!DOCTYPE
3 <!DOCTYPE
5 <!DOCTYPE

10 <!DOCTYPE
8 <!DOCTYPE
7 <!DOCTYPE
9 <!DOCTYPE
6 <!DOCTYPE

根据运行体验结果，可以明确两个结果：
[1] 组内API以异步方式执行；
[2] 组间API顺序执行。

5.3 提供配置入口

配置入口以命令行参数形式提供还是以配置文件提供呢？此文特别不擅长了解到他人喜好，为了具有尽量友好的理念，只有两种方式都提供了。

此文选择argparse作为命令行参数解析是因为其是python的内置模块，不涉及第三方模块的管理。

#!/usr/bin/python3
# -*- coding: utf-8 -*-

"""
ontApi_argparser.py,
to parse configurations from command or config file.

lxr, 2019.12.15
"""

import argparse
import json
import sys
import os

class OntApiCmdArgs():
    ''' to parse the command arguments for 
        OneNET Video-Platform by argparse module.
        
        instance variables PARSER, ARGS, HAS_CFG, HAS_ARGS,
        can be used directlly by outside.
    '''
    
    def __init__(self):
        self.HAS_CFG  = False
        self.HAS_ARGS = False
        self.PARSER   = self.ARGS = None
        
        parser = self.__create_cmdarg_parser()
        self.__add_optargs(parser)
        
        args = self.__parse_cmdargs(parser)
        self.__default_configurations(args)
        self.__update_cfg_depend_on_user(args)
        self.__tips_for_logs_output_file(args)
        
        self.ARGS   = args
        self.PARSER = parser
        
    def __create_cmdarg_parser(self):
        ARGS = argparse.ArgumentParser(
                description = self._descriprion + '-' * len(self._descriprion),
                epilog      = self._epilog      + '-' * len(self._epilog),
                formatter_class = argparse.RawTextHelpFormatter)
        return ARGS
    
    def __add_optargs(self, parser):
        self.__add_version_optarg(parser)
        self.__add_platform_optargs(parser)
        self.__add_device_optargs(parser)
        self.__add_log_optargs(parser)
        self.__add_cfg_optargs(parser)
        self.__add_default_optargs(parser)
        
    def __add_version_optarg(self, parser):
        parser.add_argument('-v', '--version',
                action  = 'version',
                version = f'{sys.argv[0]} 0.0.1' )
    
    def __add_platform_optargs(self, parser):
        group = parser.add_argument_group(self._optarg_desc['platform'])
        group.add_argument('-s',
                metavar = '  IP',
                dest    = 'ip', 
                type    = str,
                default = self._default_cfg['ip'],
                help    = self._help['ip'] )
        group.add_argument('-p',
                metavar = '  PORT',
                dest    = 'port',
                type    = int,
                default = self._default_cfg['port'],
                help    = self._help['port'] )
        group.add_argument('-hdr',
                metavar = 'HEADER',
                dest    = 'header',
                type    = str,
                default = self._default_cfg['header'],
                help    = self._help['header'])
    
    def __add_device_optargs(self, parser):
        group = parser.add_argument_group(self._optarg_desc['device'])
        group.add_argument('-did',
                metavar = 'DEVICE_ID',
                dest    = 'did',
                type    = int,
                default = self._default_cfg['did'],
                help    = self._help['did'] )
        group.add_argument('-pid',
                metavar = 'PRODUCT_ID',
                dest    = 'pid',
                type    = int,
                default = self._default_cfg['pid'],
                help    = self._help['pid'] )
    
    def __add_log_optargs(self, parser):
        def str2bool(_str):
            return True if _str.lower() not in ['false', '0'] else False
        group = parser.add_argument_group(self._optarg_desc['log'])
        group.add_argument('-req', 
                metavar = '0/1',
                dest    = 'req',
                type    = str2bool,
                default = self._default_cfg['req'],
                help    = self._help['req'] )
        group.add_argument('-rep', 
                metavar = '0/1',
                dest    = 'rep',
                type    = str2bool,
                default = self._default_cfg['rep'],
                help    = self._help['rep'] )
        group.add_argument('-jsi', 
                metavar = 'NR',
                dest    = 'jsi',
                type    = int,
                default = self._default_cfg['jsi'],
                choices = range(1, 8),
                help    = self._help['jsi'] )
        group.add_argument('--log',
                metavar = 'LOGFILE',
                type    = argparse.FileType('w', encoding='UTF-8'),
                default = self._default_cfg['log'],
                help    = self._help['log'] )
    
    def __add_cfg_optargs(self, parser):
        group = parser.add_argument_group(self._optarg_desc['cfg'])
        group.add_argument('--cfg',
                metavar = 'CONFIGFILE',
                type    = str,
                default = self._default_cfg['cfg'],
                help    = self._help['cfg'])
    
    def __add_default_optargs(self, parser):
        group = parser.add_argument_group(self._optarg_desc['dft'])
        group.add_argument('-dft', '--default',
                action = 'store_true',
                help   = self._help['dft'])
        
    def __parse_cmdargs(self, parser):
        try: return parser.parse_args()
        except IOError as e: exit(e)
    
    def __tips_for_logs_output_file(self, args):
        if args.__dict__['log'].name != self._default_cfg['log'].name:
            tips = f"\nplease to check the * {args.__dict__['log'].name} * for results"
            print(tips, '-' * len(tips), sep='\n')
    
    def __default_configurations(self, args):
        if args.default:
            _default_cfg_dict = {
                'IP':     self._default_cfg['ip'],
                'PORT':   self._default_cfg['port'],
                'HEADER': self._default_cfg['header'],
                'DEVICE_ID':  self._default_cfg['did'],
                'PRODUCT_ID': self._default_cfg['pid'],
                'req': self._default_cfg['req'],
                'rep': self._default_cfg['rep'],
                'jsi': self._default_cfg['jsi'],
                'log': self._default_cfg['log'].name,
                'cfg': self._default_cfg['cfg'],
            }
            print(json.dumps(_default_cfg_dict, indent=4))
            dft_tips = 'the default configurations here you are\n'
            exit(dft_tips + '-' * len(dft_tips)) if 2 == len(sys.argv) else None
            
    def __update_cfg_depend_on_user(self, args):
        if 1 < len(sys.argv): 
            self.HAS_ARGS = True
            yon = input(f'update the command arguments to the * {args.cfg} * (y/n)?')
            if yon.lower() in ('y', 'yes'): 
                self.__update_cfg(args.cfg, args.__dict__)
        
        if os.path.exists(args.cfg):
            self.HAS_CFG = True
        
    def __update_cfg(self, cfg, cfg_v):
        v = {
            'ip':     cfg_v['ip'],
            'port':   cfg_v['port'],
            'header': cfg_v['header'],
            
            'pid': cfg_v['pid'],
            'did': cfg_v['did'],
            
            'req': cfg_v['req'],
            'rep': cfg_v['rep'],
            'jsi': cfg_v['jsi'],
            'log': cfg_v['log'].name,
            'cfg': cfg_v['cfg']
        }
        with open(cfg, mode='w', encoding='utf-8') as f:
            try: f.write(json.dumps(v, indent=4))
            except Exception as e: exit(e)
        print(f"the command arguments have been updated to the * {cfg_v['cfg']} *")
    
    ''' the default configurations of OneNET-Video-Platform '''
    _default_cfg = {
        'ip': 'api.heclouds.com',
        'port': 80,
        'header': 'api-key:IqvnbbtJWUN23SHQRGFyfW=8ICc=',
        
        'pid': 174585,
        'did': 527776306,
        
        'req': True,
        'rep': False,
        'jsi': 1,
        'log': sys.stdout,
        'cfg': 'ontApiTest_config.json'
    }
    
    _help = {
        'ip':     "set API's         ip to be IP",
        'port':   "set API's     header to be HEADER",
        'header': "set API's     header to be HEADER",
        
        'did': "set API's device  id to be DEVICE_ID",
        'pid': "set API's product id to be PRODUCT_ID",
        
        'req': 'set flag which decides whether to show requests',
        'rep': 'set flag which decides whether to show responses',
        'jsi': 'set NR-indent to show responses in json format',
        
        'log': 'set LOGFILE  to show the log of this program',
        'cfg': 'set config file to be CONFIGFILE',
        'dft': 'show the default configurations',
    }
    
    _optarg_desc = {
        'platform': 'optional arguments on OneNET Video-Platform',
        'device':   'optional arguments on device',
        'log':      'optional arguments on log',
        'cfg':      'optional arguments on config file',
        'dft':      'optional arguments on default',
    }
    
    _descriprion = f'{sys.argv[0]}  arguments manual\n'
    _epilog = 'those arguments can be configured in CONFIGFILE statically too\n'


if __name__ == '__main__':
    ''' for testing ontApi_argparser.py '''
    cmdargs = OntApiCmdArgs()
    cmdargs.ARGS.log.write(f'{cmdargs.ARGS.__dict__}')
    cmdargs.ARGS.log.close

根据ontApi_argparser.py的运行体验看看其大概能做些什么吧。
[1] help

> python ontApi_argparser.py -h
usage: ontApi_argparser.py [-h] [-v] [-s   IP] [-p   PORT] [-hdr HEADER]
                           [-did DEVICE_ID] [-pid PRODUCT_ID] [-req 0/1]
                           [-rep 0/1] [-jsi NR] [--log LOGFILE]
                           [--cfg CONFIGFILE] [-dft]

ontApi_argparser.py  arguments manual
--------------------------------------

optional arguments:
  -h, --help        show this help message and exit
  -v, --version     show program's version number and exit

optional arguments on OneNET Video-Platform:
  -s   IP           set API's         ip to be IP
  -p   PORT         set API's     header to be HEADER
  -hdr HEADER       set API's     header to be HEADER

optional arguments on device:
  -did DEVICE_ID    set API's device  id to be DEVICE_ID
  -pid PRODUCT_ID   set API's product id to be PRODUCT_ID

optional arguments on log:
  -req 0/1          set flag which decides whether to show requests
  -rep 0/1          set flag which decides whether to show responses
  -jsi NR           set NR-indent to show responses in json format
  --log LOGFILE     set LOGFILE  to show the log of this program

optional arguments on config file:
  --cfg CONFIGFILE  set config file to be CONFIGFILE

optional arguments on default:
  -dft, --default   show the default configurations

those arguments can be configured in CONFIGFILE statically too
---------------------------------------------------------------

[2] 查看默认配置

> python ontApi_argparser.py --default
{
    "IP": "api.heclouds.com",
    "PORT": 80,
    "HEADER": "api-key:IqvnbbtJWUN23SHQRGFyfW=8ICc=",
    "DEVICE_ID": 527776306,
    "PRODUCT_ID": 174585,
    "req": true,
    "rep": false,
    "jsi": 1,
    "log": "<stdout>",
    "cfg": "ontApiTest_config.json"
}
the default configurations here you are
----------------------------------------

[3] 同步新配置到配置文件中
ontApi_argparser.py将默认配置内置在argparse中，且默认不生成配置文件（曾有人挑剔说配置文件是尾巴）。

若习惯或不得已要使用配置文件，可通过以下两种方式生成得到配置文件：
[1] 使用–default查看配置文件名和内容格式，然后手动配置；
[2] 在命令行中指定某配置时同步配置到配置文件中（推荐）。

> python ontApi_argparser.py -req 0
update the command arguments to the * ontApiTest_config.json * (y/n)?y
the command arguments have been updated to the * ontApiTest_config.json *

[4] 生成日志文件
除help外内容外，程序打印内容可输出到指定的日志文件中。

> python ontApi_argparser.py --log output.log
update the command arguments to the * ontApiTest_config.json * (y/n)?y
the command arguments have been updated to the * ontApiTest_config.json *

please to check the * output.log * for results
-----------------------------------------------

5.4 抽取HTTP-API通用部分

为了后期更方便的编写各个API请求，前期先观察下有没有通用部分吧，免得增大重复工作量占比。

#!/usr/bin/python3
# -*- coding: utf-8 -*-

"""
ontApi_comm.py,
wrapper for common code for OneNET-Video-Platform-HTTP-APIs.

lxr, 2019.12.15
"""

import json
import ontApi_argparser as _argparser

class Common():
    ''' instance variables
        CMD  the information of commad argument-parsing
        LOG  output file of log
        CFGN the dict of command arguments
    '''
    def __init__(self):
        self.CMD  = _argparser.OntApiCmdArgs()
        self.LOG  = self.CMD.ARGS.log
        self.CFGN = self.__get_cfgn()
        
    def __get_cfgn(self):
        if (not self.CMD.HAS_ARGS) and self.CMD.HAS_CFG:
            cfgn   = self.__get_cfgn_from_cfg(self.CMD.ARGS.cfg)
        else: cfgn = self.__get_cfgn_from_cmd(self.CMD.ARGS)
        
        self.__check_cfgn_if_valid(cfgn)
        return cfgn
        
    def __get_cfgn_from_cfg(self, cfg):
        with open(cfg, mode='r', encoding='utf-8') as f:
            try: return json.loads(f.read())
            except Exception as e: exit(f'{cfg}: {e}')
        
    def __get_cfgn_from_cmd(self, args):
        return args.__dict__
    
    @staticmethod
    def __check_cfgn_if_valid(cfgn):
        necessary_kyes = {'ip', 'port', 'header', 'pid', 'did'}
        if not necessary_kyes < set(cfgn.keys()):
            exit(f"three are no necessary fields in {cfgn['cfg']}")
        # the other checks
    
    ''' to get configuration by below method besides by CFGN '''
    def get_url_comm_part(self):
        addr, port = self.CFGN['ip'], self.CFGN['port']
        if not any(prtcl in addr for prtcl in ['http://', 'https://']):
            addr = 'http://' + addr
        return '%s:%d/%s' % (addr, port, 'ipc/video')
    
    ''' to get the configuration by following method '''
    
    def get_header(self):
        return self.CFGN['header']
    
    def get_product_id(self):
        return self.CFGN['pid']
    
    def get_device_id(self):
        return self.CFGN['did']
    
    def get_request_showing_flag(self):
        return self.CFGN['req'] if 'req' in self.CFGN.keys() else True
    
    def get_response_showing_flag(self):
        return self.CFGN['rep'] if 'rep' in self.CFGN.keys() else False
    
    def get_response_json_indent(self):
        return self.CFGN['jsi'] if 'jsi' in self.CFGN.keys() else 1

if __name__ == '__main__':
    ''' for testing ontApi_comm.py '''
    comm = Common()
    comm.LOG.write(f'{comm.CFGN}')
    comm.LOG.close()

ontApi_comm.py具ontApi_argparser.py所有功能，以类方法对外提供配置信息。
简单运行体验下。

> python ontApi_comm.py
{'ip': 'api.heclouds.com', 'port': 80, 'header': 'api-key:IqvnbbtJWUN23SHQRGFyfW=8ICc=', 'did': 527776306, 'pid': 174585, 'req': True, 'rep': False, 'jsi': 1, 'log': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, 'cfg': 'ontApiTest_config.json', 'default': False}

5.5 包装API && 异步请求

在正式编写HTTP请求API之前再写一些包装函数。

def _show_log(index, url, resp, dtm, comm):
    ''' to show some information according to configurations '''
    if comm.get_request_showing_flag():
        comm.LOG.write(f'{index} {url} {"%8.2f" % (dtm)}ms\n')

    indent = comm.get_response_json_indent()
    indent = None if indent == 1 else indent
    json_flag = 1
    try:
        resp_dict = json.loads(resp, encoding='utf-8')
        if resp_dict['errno'] != 0 and resp_dict['errno'] != 200:
            comm.LOG.write(json.dumps(resp_dict, indent=indent) + '\n\n')
            return None
    except Exception as e: 
        json_flag = 0
        resp = resp if resp != None else 'None'
        comm.LOG.write('none-json response: \n' + resp + '\n\n')
    
    if comm.get_response_showing_flag() and json_flag:
        comm.LOG.write(json.dumps(json.loads(resp), indent=indent) + '\n\n')

_time_sync_total = 0
def _log_decorator(api_fn):
    ''' one decorator for OneNET-Video-Platform-API to show logs '''
    async def _w(index, comm):
        global _time_sync_total
        
        ltm = time.time_ns()
        url, resp = await api_fn(index, comm)
        dtm = (time.time_ns() - ltm) / 1.0e+6
        _time_sync_total += dtm
        
        _show_log(index, url, resp, dtm, comm)
        
    return _w

def _get_url_header(url_path, comm):
    ''' wrapper for the whole url and headers for API '''
    url  = comm.get_url_comm_part() + url_path
    hdrs = {comm.get_header().split(':', )[0] : comm.get_header().split(':', )[1]}
    return url, hdrs

随便找一组APIs实现一下，看看如何使用之前的包装函数来进一步实现HTTP-API请求。

''' 通道管理 '''
@_log_decorator
async def channel_query(index, comm):
    url, hdrs = _get_url_header('/device/QryChannel', comm)
    params    = {'device_id': comm.get_device_id()}
    
    resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
    return url, resp

_channle_test_id = 55 + random.randint(0, 10)
@_log_decorator
async def channel_delete(index, comm):
    url, hdrs = _get_url_header('/device/DelChannel', comm)
    params    = {'device_id':  comm.get_device_id(),
        'channel_id': _channle_test_id
    }
    
    resp = await asyctn.w_request(url, 'delete', hdrs=hdrs, params=params)
    return url, resp

@_log_decorator
async def channle_add(index, comm):
    url, hdrs = _get_url_header('/device/AddChannel', comm)
    params    = {'product_id': comm.get_product_id(),
        'device_id':  comm.get_device_id(),
        'channel_id': _channle_test_id,
        'title': 'addchannel',
        'desc':  'addchanneltest'
    }
    
    resp = await asyctn.w_request(url, 'post', hdrs=hdrs, params=params)
    return url, resp

@_log_decorator
async def channel_modify(index, comm):
    url, hdrs = _get_url_header('/device/ModifyChannel', comm)
    params    = {'device_id':  comm.get_device_id(),
        'channel_id': _channle_test_id,
        'title':      'titlemodifytest',
        'desc':       'descmodifytest'
    }
    
    resp = await asyctn.w_request(url, 'post', hdrs=hdrs, params=params)
    return url, resp

后续APIs都是这么封装啦，来看下完整实现。

#!/usr/bin/python3
# -*- coding: utf-8 -*-

"""
ontApi_main.py,
the main routines of async && coroutine example.

lxr, 2019.12.15
"""

import json
import time
import random
import asyctn
import ontApi_comm as _comm
from platform import python_version

'''
some wrappers for APIs of OneNET-Video-Platform below,
none special ticks.
'''
def _check_python_version():
    ''' just provide friendly tips to the best of my ability '''
    cv = python_version()
    tv = '3.7'
    if cv < tv:
        exit(f"current python version is {cv}, python version can't be less than {tv}\n")

def _apis_done_tips(sync_time, async_time, comm):
    tips = 'time saved about  %8.2fms from syncio to asyncio-coroutine' % \
           (sync_time - async_time)
    comm.LOG.write(  'syncio            total time: %8.2fms' % sync_time  + 
                  f'\nasyncio-coroutine total time: %8.2fms' % async_time + 
                  f'\n{tips}' + 
                  f'\n{"-" * (round(len(tips)/2)-2)}done{"-" * round((len(tips)/2)-2)}')

def _show_log(index, url, resp, dtm, comm):
    ''' to show some information according to configurations '''
    if comm.get_request_showing_flag():
        comm.LOG.write(f'{index} {url} {"%8.2f" % (dtm)}ms\n')

    indent = comm.get_response_json_indent()
    indent = None if indent == 1 else indent
    json_flag = 1
    try:
        resp_dict = json.loads(resp, encoding='utf-8')
        if resp_dict['errno'] != 0 and resp_dict['errno'] != 200:
            comm.LOG.write(json.dumps(resp_dict, indent=indent) + '\n\n')
            return None
    except Exception as e: 
        json_flag = 0
        resp = resp if resp != None else 'None'
        comm.LOG.write('none-json response: \n' + resp + '\n\n')
    
    if comm.get_response_showing_flag() and json_flag:
        comm.LOG.write(json.dumps(json.loads(resp), indent=indent) + '\n\n')

_time_sync_total = 0
def _log_decorator(api_fn):
    ''' one decorator for OneNET-Video-Platform-API to show logs '''
    async def _w(index, comm):
        global _time_sync_total
        
        ltm = time.time_ns()
        url, resp = await api_fn(index, comm)
        dtm = (time.time_ns() - ltm) / 1.0e+6
        _time_sync_total += dtm
        
        _show_log(index, url, resp, dtm, comm)
        
    return _w

def _get_url_header(url_path, comm):
    ''' wrapper for the whole url and headers for API '''
    url  = comm.get_url_comm_part() + url_path
    hdrs = {comm.get_header().split(':', )[0] : comm.get_header().split(':', )[1]}
    return url, hdrs


'''
the code on partial(easier to call) APIs of OneNET-Video-Platform,
none special tricks.
'''
''' 通道管理 '''
@_log_decorator
async def channel_query(index, comm):
    url, hdrs = _get_url_header('/device/QryChannel', comm)
    params    = {'device_id': comm.get_device_id()}
    
    resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
    return url, resp

_channle_test_id = 55 + random.randint(0, 10)
@_log_decorator
async def channel_delete(index, comm):
    url, hdrs = _get_url_header('/device/DelChannel', comm)
    params    = {'device_id':  comm.get_device_id(),
        'channel_id': _channle_test_id
    }
    
    resp = await asyctn.w_request(url, 'delete', hdrs=hdrs, params=params)
    return url, resp

@_log_decorator
async def channle_add(index, comm):
    url, hdrs = _get_url_header('/device/AddChannel', comm)
    params    = {'product_id': comm.get_product_id(),
        'device_id':  comm.get_device_id(),
        'channel_id': _channle_test_id,
        'title': 'addchannel',
        'desc':  'addchanneltest'
    }
    
    resp = await asyctn.w_request(url, 'post', hdrs=hdrs, params=params)
    return url, resp

@_log_decorator
async def channel_modify(index, comm):
    url, hdrs = _get_url_header('/device/ModifyChannel', comm)
    params    = {'device_id':  comm.get_device_id(),
        'channel_id': _channle_test_id,
        'title':      'titlemodifytest',
        'desc':       'descmodifytest'
    }
    
    resp = await asyctn.w_request(url, 'post', hdrs=hdrs, params=params)
    return url, resp

''' 设备在线状态查询 '''
@_log_decorator
async def device_online_status_query(index, comm):
    url, hdrs = _get_url_header('/device/QryDevStatus', comm)
    params    = {'devIds': comm.get_device_id()}
    
    resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
    return url, resp

''' 命令下发 '''
@_log_decorator
async def cmmd_send(index, comm):
    url, hdrs = _get_url_header('/cmds', comm)
    params    = {'device_id': comm.get_device_id(),
        'qos':  1,
        'type': 0
    }
    data = {'type': 'video',
        'cmdId': 6,
        'cmd':{
            'channel_id': 1,
            'level': 1
        }
    }
    
    resp = await asyctn.w_request(url, 'post', 
                    hdrs=hdrs, params=params, data=data)
    return url, resp

''' 透传信息查询 '''
@_log_decorator
async def device_msg_query(index, comm):
    url, hdrs = _get_url_header('/device/QryDevMsg', comm)
    params    = {'device_id': comm.get_device_id(),
        'begin_time': '2019-12-07 19:35:00',
        'end_time':   '2020-12-07 19:35:00'
    }
    
    resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
    return url, resp

''' 拉流地址 '''
_player_address_url_path = '/play_address'
@_log_decorator
async def player_address_rtmp_get(index, comm):
    url, hdrs = _get_url_header(_player_address_url_path, comm)
    params    = {'device_id': comm.get_device_id(),
        'channel_id':    1,
        'protocol_type': 0,
    }
    
    resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
    return url, resp

@_log_decorator
async def player_address_hls_get(index, comm):
    url, hdrs = _get_url_header(_player_address_url_path, comm)
    params    = {'device_id': comm.get_device_id(),
        'channel_id':    1,
        'protocol_type': 1,
    }
    
    resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
    return url, resp

''' 平台图片相关接口 '''
_picture_test_chid   = 1
_picture_test_format = 'jpg'
_picture_test_name   = 'ontApiPictureTest'
_picture_fp = 'test_file/' + _picture_test_name + '.' + _picture_test_format
@_log_decorator
async def picture_upload(index, comm):
    url, hdrs = _get_url_header('/picture/upload', comm)
    params    = {'device_id': comm.get_device_id(),
        'channel_id': _picture_test_chid,
        'format':     _picture_test_format,
        'name':       _picture_test_name,
        'desc':       'pictureuploadtest'}
    with open(_picture_fp, mode='rb') as pic:
        try: data = pic.read()
        except Exception as e: exit(f'{_picture_fp}: {e}')
    
    resp = await asyctn.w_request(url, 'put', 
                    hdrs=hdrs, params=params, data=data)
    return url, resp

@_log_decorator
async def picture_list_get(index, comm):
    url, hdrs = _get_url_header('/picture/get_list', comm)
    params    = {'device_id': comm.get_device_id(),
        'channel_id': _picture_test_chid,
        'page_start': 1,
        'page_size':  10
    }
    
    resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
    return url, resp

@_log_decorator
async def picture_info_get(index, comm):
    url, hdrs = _get_url_header('/picture/get_info', comm)
    params    = {'device_id': comm.get_device_id(),
        'channel_id': _picture_test_chid,
        'name':       _picture_test_name
    }
    
    resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
    return url, resp
    
    
@_log_decorator
async def picture_delete(index, comm):
    url, hdrs = _get_url_header('/picture/delete_picture', comm)
    params    = {'device_id': comm.get_device_id(),
        'channel_id': _picture_test_chid,
        'name':       _picture_test_name
    }
    
    resp = await asyctn.w_request(url, 'delete', hdrs=hdrs, params=params)
    return url, resp

''' 引导机地址 '''
@_log_decorator
async def boot_addr_get(index, comm):
    url  = comm.get_url_comm_part() + '/boot_address'
    resp = await asyctn.w_request(url, 'get')
    return url, resp

''' 设备数据透传 '''
@_log_decorator
async def device_data_ttrans(index, comm):
    url, hdrs = _get_url_header('/dev_active', comm)
    params    = {'device_id': comm.get_device_id()}
    data      = 'd' * 127;
    
    resp = await asyctn.w_request(url, 'post', 
                    hdrs=hdrs, params=params, data=data)
    return url, resp

''' 点播相关接口 '''
_vod_test_chid = 1
@_log_decorator
async def vod_list_get(index, comm):
    url, hdrs = _get_url_header('/vod/get_video_list', comm)
    params    = {'device_id': comm.get_device_id(),
        'channel_id': _vod_test_chid,
        'page_start': 1,
        'page_size':  10
    }
    
    resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
    return url, resp

''' 公共对象存储 '''
_filename = 'fileTest.dat'
_file_fp  = 'test_file/' + _filename
@_log_decorator
async def file_upload(index, comm):
    url, hdrs = _get_url_header('/file/upload', comm)
    params    = {'product_id': comm.get_product_id(),
        'name': _filename,
        'desc': 'fileuploadtest'
    }
    with open(_file_fp, mode='rb') as f:
        try: data = f.read()
        except Exception as e: exit(f'{_file_fp}: {e}')
    
    resp = await asyctn.w_request(url, 'put', 
                    hdrs=hdrs, params=params, data=data)
    return url, resp
    
@_log_decorator
async def file_download(index, comm):
    url, hdrs = _get_url_header('/file/download', comm)
    params    = {'product_id': comm.get_product_id(),
        'name': _filename
    }
    
    resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
    return url, resp

@_log_decorator
async def file_list(index, comm):
    url, hdrs = _get_url_header('/file/list', comm)
    params    = {'product_id': comm.get_product_id(),
        'name': _filename
    }
    
    resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
    return url, resp
    
@_log_decorator
async def file_delete(index, comm):
    url, hdrs = _get_url_header('/file/delete', comm)
    params    = {'product_id': comm.get_product_id(),
        'name': _filename
    }
    
    resp = await asyctn.w_request(url, 'delete', hdrs=hdrs, params=params)
    return url, resp

''' 运营-内部和外部使用 '''
@_log_decorator
async def stat_onlinelist_get(index, comm):
    url, hdrs = _get_url_header('/stat/getonlinelist', comm)
    params    = {'productid': comm.get_product_id(),
        'deviceid': comm.get_device_id(),
        'protocol_type': 'rtmp',
    }
    
    resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
    return url, resp

_stat_test_chid = 1
@_log_decorator
async def stat_pushlist_get(index, comm):
    url, hdrs = _get_url_header('/stat/getpushlist', comm)
    params    = {'productid': comm.get_product_id(),
        'deviceid':  comm.get_device_id(),
        'channelid': _stat_test_chid,
    }
    
    resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
    return url, resp
    
''' 存储管理配置接口 '''
_storage_test_chid = 1
@_log_decorator
async def storage_info_query(index, comm):
    url, hdrs = _get_url_header('/storage/product_scheme_info', comm)
    params    = {'product_id': comm.get_product_id(),
        'device_id':  comm.get_device_id(),
        'channel_id': _storage_test_chid,
        'page_start': 1,
        'page_size':  10
    }
    
    resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
    return url, resp

if __name__ == '__main__':
    ''' for request of OneNET-Video-Platform-APIs running '''
    
    _check_python_version()
    comm  = _comm.Common()
    
    # request first
    apis_first = [
       channle_add(1, comm),
       cmmd_send  (2, comm),
       file_upload(3, comm),
       boot_addr_get (4, comm),
       picture_upload(5, comm),
    ]
    
    # request then
    apis_second = [
       file_list    (6, comm),
       file_download(7, comm),
       picture_list_get(8, comm),
       picture_info_get(9, comm),
       channel_modify  (10, comm),
       device_msg_query(11, comm),
       device_online_status_query(12, comm),
    ]
    
    # request finally
    apis_third = [
        player_address_rtmp_get(13, comm),
        player_address_hls_get (14, comm),
        vod_list_get  (15, comm),
        file_delete   (16, comm),
        channel_delete(17, comm),
        picture_delete(18, comm),
        device_data_ttrans (20, comm),
        stat_pushlist_get  (21, comm),
        stat_onlinelist_get(22, comm),
        storage_info_query (23, comm),
    ]
    
    time_async_start = time.time_ns()
    asyctn_loop = asyctn.AsyCtnLoop()
    asyctn_loop.w_loop(apis_first)
    asyctn_loop.w_loop(apis_second)
    asyctn_loop.w_loop(apis_third)
    asyctn_loop.w_close()
    time_async_end = time.time_ns()
    
    _apis_done_tips(_time_sync_total, (time_async_end - time_async_start) / 1.0e+6, comm

6 例子运行体验

> python ontApi_main.py
4 http://api.heclouds.com:80/ipc/video/boot_address    64.88ms
2 http://api.heclouds.com:80/ipc/video/cmds   140.33ms
{"errno": 3, "error": "device not online"}
1 http://api.heclouds.com:80/ipc/video/device/AddChannel    70.10ms
3 http://api.heclouds.com:80/ipc/video/file/upload    75.15ms
5 http://api.heclouds.com:80/ipc/video/picture/upload    95.29ms

9 http://api.heclouds.com:80/ipc/video/picture/get_info    32.76ms
8 http://api.heclouds.com:80/ipc/video/picture/get_list    32.76ms
10 http://api.heclouds.com:80/ipc/video/device/ModifyChannel    32.76ms
6 http://api.heclouds.com:80/ipc/video/file/list    32.76ms
7 http://api.heclouds.com:80/ipc/video/file/download    32.76ms
none-json response: 
oh.... filedownload response is not the unform json format
11 http://api.heclouds.com:80/ipc/video/device/QryDevMsg    32.76ms
12 http://api.heclouds.com:80/ipc/video/device/QryDevStatus    40.20ms

13 http://api.heclouds.com:80/ipc/video/play_address    37.36ms
{"errno": 4, "error": "device not online"}
20 http://api.heclouds.com:80/ipc/video/dev_active    37.36ms
{"errno": 3, "error": "device not online"}
17 http://api.heclouds.com:80/ipc/video/device/DelChannel    37.36ms
14 http://api.heclouds.com:80/ipc/video/play_address    37.36ms
{"errno": 4, "error": "device not online"}
18 http://api.heclouds.com:80/ipc/video/picture/delete_picture    37.36ms
23 http://api.heclouds.com:80/ipc/video/storage/product_scheme_info    37.36ms
16 http://api.heclouds.com:80/ipc/video/file/delete    37.36ms
15 http://api.heclouds.com:80/ipc/video/vod/get_video_list    37.36ms
21 http://api.heclouds.com:80/ipc/video/stat/getpushlist    69.62ms
22 http://api.heclouds.com:80/ipc/video/stat/getonlinelist    67.59ms

syncio            total time:  1118.56ms
asyncio-coroutine total time:   277.38ms
time saved about    841.18ms from syncio to asyncio-coroutine
----------------------------done----------------------------

ontApi_main.py拥有ontApi_comm.py的功能，其将默认打印请求url和异常响应。

[1, 5], [6, 12], [13, 23]三组API各组以先后顺序执行，组内API异步执行。这可以满足有先后执行顺序需求的APIs。

另外可以看出，各APIs在一个线程中以同步方式执行完毕时约耗1s，以异步方式执行约耗0.3s。由于此处只有23个APIs，所以同步和异步的时间差异体现不大。在一定范围内，随着API数增加，同步与异步之间时间差也会增加。

根据此次运行体验，一个OneNET-Video-HTTP-APIs请求平均会耗 1118.56ms / 23 ≈ 48.6ms。Linux0.1.1中，一个进程的时间片为150ms。所以…可尝试为您的并发单位争取一下类似的时间片段。

IO驱动程序，如Linux的IO驱动程序。 ↩︎
CPU并发单位，如已形成软件概念的进程，线程。 ↩︎
此文没有在特定计算机中验证过CPU与IO速度差异。只可借助《CSAPP》中对存储器件速度差异描述体现一下此处提到的差异了：CPU访问内部寄存器最快只需1个CPU周期，访问SRAM（cache）需几个CPU周期，访问主存需几十到几百个CPU周期，访问SSD/HHD会打千万级CPU周期。更何况是网络IO呢。 ↩︎
进程或线程会增加CPU层面的创建和调度开销；随着IO并发量增大，进程和线程更容易达到计算机资源上限。可在更复杂的业务场景考虑多种并发方式的结合。 ↩︎

竹影半墙

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
一种在 python 中用 asyncio 和协程实现的IO并发

1 此篇文字围绕的主题基于IO驱动程序[^1]，本机从IO读取或发送数据——读/写IO的有效方法。+--------------------------+ +----+| | +==>| || +--+ +---+ | | +----+ 1| +-----+ | | | I...
复制链接

扫一扫