WSGI 是什么

1 WSGI 是什么?

WSGI 的全称 Web Server Gateway Interface ; web服务器网关接口,它只是一个规范,是一个协议.

WSGI 不是服务器,python模块,框架,API或任何类型的软件。 它只是服务器和应用程序通信的接口规范。

服务器和应用程序接口端都在 PEP 3333 中指定。如果将应用程序（或框架或工具包）写入WSGI规范，则它将在写入该规范的任何服务器上运行.

WSGI 指定特定的标准接口 在 web server 和 web application 之间 通信.

WSGI 协议 规定了Web服务器与Python Web应用程序或框架之间的建议标准接口，以促进跨各种Web服务器的Web应用程序可移植性。WSGI的目标是促进现有服务器和应用程序或web框架的轻松互连.

大概就是下图:

简单点说: 规定 了一组规则,保证按照这个规则,就可以
实现 server 和application 如何 通信,实现解耦合.
这样 webserver 开发人员就能专注开发server 开发, app开发人员专注业务逻辑的实现.

1-1容易混淆的几个概念

几个概念 WebServer 提供web 服务的服务器软件 , 常用NGINX , Apache ,Tomcat 等 .

WSGI Server 是指 实现了 WSGI 协议的server端的服务器软件 , 常用 就是 Gunicron , uWSGI 这些都是常用的Python WSGI HTTP Server , 这就是说这些服务器软件 就是 上图中的 Web Server .

uwsgi , uWSGI

WSGI

Gunicorn , uWSGI 这些 都属于 Web 服务器, 而不是一个协议

Bottle, Flask, Django : 这些是 web framework , 实现 WSGI 协议中 的 web application/ web framework

uWSGI: 是一个Web服务器，它实现了WSGI协议、uwsgi、http等协议。

uwsgi: 是uWSGI服务器实现的独有的协议 ,uwsgi协议是uWSGI服务器使用的本地协议。它是一个二进制协议，可以携带任何类型的数据。一个uwsgi分组的头4个字节描述了这个分组包含的数据类型。

uwsgi协议 https://uwsgi-docs-zh.readthedocs.io/zh_CN/latest/Protocol.html

该图片来自: https://pic4.zhimg.com/v2-041d8e24988727fa3e4b386e93175a45_r.jpg

2 协议内容

协议内容 分为两点内容 , application 端 和 server 端. 下面分别来说明 这两个地方的实现.

2-1 application 端

The Application/ Framework Side

WSGI application接口应该实现为一个可调用对象.

这个可调用对象 一般接收 两个位置参数 environment, start_response

1. accept two positional parameters:

• A dictionary containing CGI like variables;
• and a callback function that will be used by the application to send HTTP status code/message and HTTP headers to the server.
• exc_info The exc_info argument, if supplied, must be a Python sys.exc_info() tuple

2. return the response body to the server as strings wrapped in an iterable.

简单 来说:

参数:

​ environment 是一个 字典, 里面存放 CGI的变量, 还会存放用户的一些环境 变量.

​ start_reponse 是一个回调函数. application 将使用该函数将HTTP状态代码/消息和HTTP标头(respond headers) 发送到服务器。 这个函数 是由 serve 或者 Gateway 来实现的. 这个函数用于开始HTTP响应,并且它必须返回一个write（body_data）可调用. 这个write 也是一个可调用的.

exc_info 是一个元祖 . 这个参数 是可选的. 这个参数代表异常信息. 如果将 application 将 http 的header 设置为

“200 OK” (但还没有发送) , 并且执行的时候遇到问题(报错),则可以将http 的header 改成其他内容 “500 Internal Server Error” .

为了做到这一点, 一开始假设一切都是正常的. 当发生错误的时候, 会再次调用 start_response() 会将新的状态码与http 的header 和exc_info 传入进去. 替换原有内容. 如果第二次调用没有提供 exc_info 则会报错.而且必须在发送 http 的header 之前 第二次调用start_response() . 如果发送完之后 在调用. 则会 抛出 异常. exc_info[0], exc_info[1],exc_info[2] 等. --python 核心编程第3版.

返回值:

​ 返回一个 iterable , 可迭代对象.

注意 : 这里 environ 和 start_reponse 都是web server 端提供的参数. 无需 application 端实现.

下面看一个比较简单 的 application

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
@User    : frank 
@Time    : 2019/8/29 16:11
@File    : my_application.py
@Email  : frank.chang@xinyongfei.cn
"""

def application(environ, start_response, exc_info=None):
    """
    :param environ: dict ,  environ:是一个CGI式的字典
    :param start_response:  是一个回调函数：application用来向server传递http状态码/消息/http头
    :param exc_info:  异常信息
    :return:
  
    """
    status = '200 OK'
    # 构建响应头
    response_headers = [('Content-type', 'text/plain')]
    # 调用作为参数传入的start_response，其已经在server中定义
    start_response(status, response_headers)
    # 返回可迭代的响应体（即使只有一行，也要写成列表的形式）
    return [b'Hello world, Frank ! \n']

environ 字典 一般 包括以下部分

{
	"wsgi.version": (1, 0),
	"wsgi.url_scheme": "http",
	"wsgi.input": < _io.BufferedReader name = 10 > ,
	"wsgi.errors": < _io.TextIOWrapper name = "<stderr>"
	mode = "w"
	encoding = "UTF-8" > ,
	"wsgi.multithread": True,
	"wsgi.multiprocess": False,
	"wsgi.run_once": False,
	"werkzeug.server.shutdown": < function WSGIRequestHandler.make_environ. < locals > .shutdown_server at 0x1108113b0 > ,
	"SERVER_SOFTWARE": "Werkzeug/0.15.5",
	"REQUEST_METHOD": "GET",
	"SCRIPT_NAME": "",
	"PATH_INFO": "/",
	"QUERY_STRING": "",
	"REQUEST_URI": "/",
	"RAW_URI": "/",
	"REMOTE_ADDR": "127.0.0.1",
	"REMOTE_PORT": 60754,
	"SERVER_NAME": "0.0.0.0",
	"SERVER_PORT": "5000",
	"SERVER_PROTOCOL": "HTTP/1.1",
	"HTTP_HOST": "0.0.0.0:5000",
	"HTTP_CONNECTION": "keep-alive",
	"HTTP_CACHE_CONTROL": "max-age=0",
	"HTTP_UPGRADE_INSECURE_REQUESTS": "1",
	"HTTP_USER_AGENT": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36",
	"HTTP_ACCEPT": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3",
	"HTTP_ACCEPT_ENCODING": "gzip, deflate",
	"HTTP_ACCEPT_LANGUAGE": "zh-CN,zh;q=0.9",
	"HTTP_COOKIE": "remember_token=1|a4932066a88423347421447e6aa3fcfb871520221fd3625bffc508d593eb67d6befa260dbb9a6b74b6c6b265b67e559bd3dd06df2088380ee861fddc4d81dd90",
	"werkzeug.request": < Request "http://0.0.0.0:5000/" [GET] >
}

2-2 server 端 或者 gateway端

Web Server 的条件:

1 要提供一个 start_response 的一个函数. 接收 status , http headers

2 要提供 envrion 的字典, 这个里面基本上CGI 协议的定义的一些参数 .

Web Server操作的步骤如下：

根据HTTP协议内容构建envrion,
提供一个start_response函数，接收HTTP STATUS 和 HTTP HEADERS
将envrion和start_response作为参数调用Application
接收Application返回的结果
按照HTTP协议，顺序写入HTTP 起始行,首部（start_response接收），HTTP响应体（Application返回结果）

首先HTTP 协议中 返回的 响应报文response

有三部分 : 起始行, 首部 ,主体

上图摘自: HTTP 权威指南

The server or gateway invokes the application callable once for each request it receives from an HTTP client, that is directed at the application. To illustrate, here is a simple CGI gateway, implemented as a function taking an application object.

服务器或网关 为从HTTP客户端接收的每个请求调用一次可调用的应用程序，该请求针对应用程序。为了说明，这里是一个简单的CGI网关，实现为一个获取应用程序对象的函数。

这段代码 摘自 PEP 3333 文档

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
@User    : frank 
@Time    : 2019/8/29 18:58
@File    : defualtserver.py
@Email  : frank.chang@xinyongfei.cn
"""

import os, sys

enc, esc = sys.getfilesystemencoding(), 'surrogateescape'


def unicode_to_wsgi(u):
    # Convert an environment variable to a WSGI "bytes-as-unicode" string
    return u.encode(enc, esc).decode('iso-8859-1')


def wsgi_to_bytes(s):
    return s.encode('iso-8859-1')

def run_with_cgi(application):
    environ = {k: unicode_to_wsgi(v) for k, v in os.environ.items()}
    environ['wsgi.input'] = sys.stdin.buffer
    environ['wsgi.errors'] = sys.stderr
    environ['wsgi.version'] = (1, 0)
    environ['wsgi.multithread'] = False
    environ['wsgi.multiprocess'] = True
    environ['wsgi.run_once'] = True

    if environ.get('HTTPS', 'off') in ('on', '1'):
        environ['wsgi.url_scheme'] = 'https'
    else:
        environ['wsgi.url_scheme'] = 'http'

    headers_set = []
    headers_sent = []

    def write(data):
        out = sys.stdout.buffer

        if not headers_set:
            raise AssertionError("write() before start_response()")

        elif not headers_sent:
            # Before the first output, send the stored headers
            status, response_headers = headers_sent[:] = headers_set
            out.write(wsgi_to_bytes('Status: %s\r\n' % status))
            for header in response_headers:
                out.write(wsgi_to_bytes('%s: %s\r\n' % header))
            out.write(wsgi_to_bytes('\r\n'))

        out.write(data)
        out.flush()

    def start_response(status, response_headers, exc_info=None):
        if exc_info:
            try:
                if headers_sent:
                    # Re-raise original exception if headers sent
                    raise exc_info[1].with_traceback(exc_info[2])
            finally:
                exc_info = None  # avoid dangling circular ref
        elif headers_set:
            raise AssertionError("Headers already set!")

        headers_set[:] = [status, response_headers]
       
        # Note: error checking on the headers should happen here,
        # *after* the headers are set.  That way, if an error
        # occurs, start_response can only be re-called with
        # exc_info set.
        return write

    result = application(environ, start_response)
    try:
        for data in result:
            if data:  # don't send headers until body appears
                write(data)
        if not headers_sent:
            write('')  # send headers now if body was empty
    finally:
        if hasattr(result, 'close'):
            result.close()


if __name__ == '__main__':
  	from my_application import application
    run_with_cgi(application=application)
    pass

运行 default_server 这个脚本 , 就可以直接看到下面的信息.

Status: 200 OK
Content-type: text/plain

Hello world, Frank ! 

可以简单分析 以上的流程 .

首先 application 端 会拿到 server 提供的 environment , start_reponse 这两个 参数 ,

application 端要给出状态码 ,以及 响应头 response_header ,之后 调用 服务端提供的 start_response(status,response_headers) ,之后 返回一个 可迭代对象 .

HTTP的响应 需要包含status，headers和body， 所以在application对象将body作为返回值return之前，需要先调用start_response，将status和headers的内容返回给server，这同时也是告诉server，application对象要开始返回body了。

可见，server负责接收HTTP请求，根据请求数据组装environ，定义start_response函数，将这两个参数提供给application。application根据environ信息执行业务逻辑，将结果返回给server。响应中的status、headers由start_response函数返回给server，响应的body部分被包装成iterable作为application的返回值，server将这些信息组装为HTTP响应 返回给请求方。 (浏览器,postman 等 )

2-3 start_response 为啥要返回一个 write() 一个可调用的对象?

我一开始对这个 也比较疑惑 为啥要这样做呢?

找到pep 3333 的文档

Some existing application framework APIs support unbuffered output in a different manner than WSGI. Specifically, they provide a "write" function or method of some kind to write an unbuffered block of data, or else they provide a buffered "write" function and a "flush" mechanism to flush the buffer.

一些现有的应用程序框架API以不同于WSGI的方式支持无缓冲的输出。 具体地说，它们提供某种“写”功能或某种方法来编写无缓冲的数据块，或者它们提供缓冲的“写”功能和“刷新”机制来刷新缓冲区。

Unfortunately, such APIs cannot be implemented in terms of WSGI's "iterable" application return value, unless threads or other special mechanisms are used.
不幸运的是 ,不能根据WSGI那样规定的那样 返回 一个 iterable 应用程序 返回值 实现此API,除非使用多线程或者其他特殊的机制.

Therefore, to allow these frameworks to continue using an imperative API, WSGI includes a special write() callable, returned by the start_response callable.
因此,运行这些框架 继续使用 这些必要API ,  WSGI 包含了这个特殊的 write()可调用对象，由 start_response 可调用对象返回。

其实就是为了兼容一些已经存在的web server 端的代码. 一些web server 端的代码, 提供了一个write 写的功能,为了兼容,所以 start_response 来返回 一个 write 可调用对象.

New WSGI applications and frameworks should not use the write() callable if it is possible to avoid doing so. The write()callable is strictly a hack to support imperative streaming APIs.
新的WSGI应用程序和框架不应该使用write（）可调用 如果他们能够避免使用 write()。实际上 write() 是一种hack技术, 为了支持那些已经存在的必要的流式API.

 In general, applications should produce their output via their returned iterable, as this makes it possible for web servers to interleave other tasks in the same Python thread, potentially providing better throughput for the server as a whole.
 通常，应用程序应该通过返回的可迭代产生它们的输出，因为这使得Web服务器可以在同一个Python线程中交错其他任务，从而可能为整个服务器提供更好的吞吐能力。

实际上来说, web server 的代码 远比上面复杂 ,需要处理异常, 需要完成 请求处理的handler .

3 从 web framework Flask 中学习WSGI

flask 是一个python中比较出名的web framework , 这个框架 是严格按照 WSGI 的内容来的 .

让我们回忆一下WSGI 中 , application 端的要求.

1 首先 application 是一个可调用对象.

2 environment, start_response 有两个位置参数

3 返回一个 iterable 可迭代对象.

下面这段代码来自
/xxxx/lib/python3.6/site-packages/flask/app.py

class Flask(_PackageBoundObject):
  
  def __call__(self, environ, start_response):
    """The WSGI server calls the Flask application object as the
          WSGI application. This calls :meth:`wsgi_app` which can be
          wrapped to applying middleware."""
    return self.wsgi_app(environ, start_response)
	

所以要实现 一个可调用的对象, 实现的魔术方法 __call__ 方法.

app = Flask()

app(environ,start_response)

第三 要返回一个可迭代对象. 要看下 wsgi_app 里面 是如何实现的.

    def wsgi_app(self, environ, start_response):
        """
        :param environ: A WSGI environment.
        :param start_response: A callable accepting a status code,
            a list of headers, and an optional exception context to
            start the response.
        """
        ctx = self.request_context(environ)
        error = None
        try:
            try:
                ctx.push()
                # 注意这里返回
                response = self.full_dispatch_request()
            except Exception as e:
                error = e
                # 注意这里返回
                response = self.handle_exception(e)
            except:  # noqa: B001
                error = sys.exc_info()[1]
                raise
            # 注意这里返回
            return response(environ, start_response)
        finally:
            if self.should_ignore_error(error):
                error = None
            ctx.auto_pop(error)

/xxxxxxx/lib/python3.7/site-packages/werkzeug/wrappers/base_response.py

response = self.full_dispatch_request()

这里实际上 会调用 werkzeug 里面的response 里面已经有写好的response ,flask 并没有自己实现自己的response 类, 这样就生成了response对象.

class BaseResponse(object):
		pass

    def __call__(self, environ, start_response):
        """Process this response as WSGI application.

        :param environ: the WSGI environment.
        :param start_response: the response callable provided by the WSGI
                               server.
        :return: an application iterator
        """
        app_iter, status, headers = self.get_wsgi_response(environ)
        start_response(status, headers)
        return app_iter

这个类实现了 __call__ 方法, 所以在 flask return 的时候就调用 上面的 __call__ 方法 , 这个方法就 实现了 application 端的逻辑 .

第一 给 start_response 提供了 status, 和 response_headers ,

第二 返回了 一个可迭代对象 app_iter ,实际上就是body .

4 从 gunicorn中学习WSGI

Gunicorn 对应 WSGI 中 就是 web server 端,由于没有 仔细阅读过 gunicorn 的代码

/xxxxxx/lib/python3.7/site-packages/gunicorn/http/wsgi.py
在这个里面有一个Response 类

这个类里面 实现 了 start_response, write

class Response(object):

    def __init__(self, req, sock, cfg):
        self.req = req
        self.sock = sock
        self.version = SERVER_SOFTWARE
        self.status = None
        self.chunked = False
        self.must_close = False
        self.headers = []
        self.headers_sent = False
        self.response_length = None
        self.sent = 0
        self.upgrade = False
        self.cfg = cfg

    def start_response(self, status, headers, exc_info=None):
        if exc_info:
            try:
                if self.status and self.headers_sent:
                    reraise(exc_info[0], exc_info[1], exc_info[2])
            finally:
                exc_info = None
        elif self.status is not None:
            raise AssertionError("Response headers already set!")

        self.status = status

        # get the status code from the response here so we can use it to check
        # the need for the connection header later without parsing the string
        # each time.
        try:
            self.status_code = int(self.status.split()[0])
        except ValueError:
            self.status_code = None

        self.process_headers(headers)
        self.chunked = self.is_chunked()
        return self.write
    
    
    def write(self, arg):
        self.send_headers()
        if not isinstance(arg, binary_type):
            raise TypeError('%r is not a byte' % arg)
        arglen = len(arg)
        tosend = arglen
        if self.response_length is not None:
            if self.sent >= self.response_length:
                # Never write more than self.response_length bytes
                return

            tosend = min(self.response_length - self.sent, tosend)
            if tosend < arglen:
                arg = arg[:tosend]

        # Sending an empty chunk signals the end of the
        # response and prematurely closes the response
        if self.chunked and tosend == 0:
            return

        self.sent += tosend
        util.write(self.sock, arg, self.chunked)

gunicorn 里面的 worker 同步的worker 的实现

/xxxxxxxx/lib/python3.7/site-packages/gunicorn/workers/sync.py

class SyncWorker(base.Worker):
	
    def accept(self, listener):
        client, addr = listener.accept()
        client.setblocking(1)
        util.close_on_exec(client)
        self.handle(listener, client, addr)

    def run(self):
        # if no timeout is given the worker will never wait and will
        # use the CPU for nothing. This minimal timeout prevent it.
        timeout = self.timeout or 0.5

        # self.socket appears to lose its blocking status after
        # we fork in the arbiter. Reset it here.
        for s in self.sockets:
            s.setblocking(0)

        if len(self.sockets) > 1:
            self.run_for_multiple(timeout)
        else:
            self.run_for_one(timeout)
    
    def handle_request(self, listener, req, client, addr):
        environ = {}
        resp = None
        try:
            self.cfg.pre_request(self, req)
            request_start = datetime.now()
            # 这里 就是 获取  header_response  , environ
            resp, environ = wsgi.create(req, client, addr,
                    listener.getsockname(), self.cfg)
            # Force the connection closed until someone shows
            # a buffering proxy that supports Keep-Alive to
            # the backend.
            resp.force_close()
            self.nr += 1
            if self.nr >= self.max_requests:
                self.log.info("Autorestarting worker after current request.")
                self.alive = False
            respiter = self.wsgi(environ, resp.start_response)
            try:
                if isinstance(respiter, environ['wsgi.file_wrapper']):
                    resp.write_file(respiter)
                else:
                    for item in respiter:
                        resp.write(item)
                resp.close()
                request_time = datetime.now() - request_start
                self.log.access(resp, req, environ, request_time)
            finally:
                if hasattr(respiter, "close"):
                    respiter.close()
        except EnvironmentError:
            # pass to next try-except level
            six.reraise(*sys.exc_info())
        except Exception:
            if resp and resp.headers_sent:
                # If the requests have already been sent, we should close the
                # connection to indicate the error.
                self.log.exception("Error handling request")
                try:
                    client.shutdown(socket.SHUT_RDWR)
                    client.close()
                except EnvironmentError:
                    pass
                raise StopIteration()
            raise
        finally:
            try:
                self.cfg.post_request(self, req, environ, resp)
            except Exception:
                self.log.exception("Exception in post_request hook")

resp, environ = wsgi.create(req, client, addr,
                    listener.getsockname(), self.cfg)

这里就是 获取resp, environ 两个变量. 这个 resp 有 start_response 方法

5 一个简单的demon

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
@User    : frank 
@Time    : 2019/9/3 17:08
@File    : defaultserver.py
@Email  : frank.chang@xinyongfei.cn
# from http.server import BaseHTTPRequestHandler as HandlerBase
# from wsgiref.handlers import BaseHandler as BaseHandler

"""
from http.server import HTTPServer

from wsgiref.simple_server import WSGIRequestHandler as BaseHandler

from demo1.my_application1 import application2 as app


class WSGIServer(HTTPServer):
    """BaseHTTPServer that implements the Python WSGI protocol"""

    application = None

    def server_bind(self):
        """Override server_bind to store the server name."""
        super().server_bind()
        self.setup_environ()

    def setup_environ(self):
        # Set up base environment
        env = self.base_environ = {}
        env['SERVER_NAME'] = self.server_name
        env['GATEWAY_INTERFACE'] = 'CGI/1.1'
        env['SERVER_PORT'] = str(self.server_port)
        env['REMOTE_HOST'] = ''
        env['CONTENT_LENGTH'] = ''
        env['SCRIPT_NAME'] = ''

    def get_app(self):
        return self.application

    def set_app(self, application):
        self.application = application


def make_server(host, port, app, wsgi_server_class=WSGIServer, handler_class=BaseHandler):
    """

    :param host:
    :param port:
    :param app:
    :param wsgi_server_class:
    :param handler_class:
    :return:
    """
    server = wsgi_server_class((host, port), RequestHandlerClass=handler_class)
    server.set_app(app)
    return server


def test_start():
    host, port = ('localhost', 5002)
    server = WSGIServer((host, port), RequestHandlerClass=BaseHandler)
    server.set_app(application=app)
    print(f"Serving HTTP on port {port}...  http://localhost:{port}")
    server.serve_forever()


if __name__ == '__main__':
    host, port = ('127.0.0.1', 5002)

    server = make_server(host, port, app=app)
    print(f"Serving HTTP on port {port}...  http://localhost:{port}")
    server.serve_forever()

/xxxxxxxxxxx/lib/python3.7/wsgiref/handlers.py

BaseHandler 类 实现了 start_response

6 总结

本文简单 介绍了一下 WSGI 协议, 以及对 WSGI 协议的一点疑惑, 同时 看了一下框架 如何 支持这样 的 协议的.

7 参考文档

PEP 333 – Python Web Server Gateway Interface https://www.python.org/dev/peps/pep-0333/

PEP 3333 – Python Web Server Gateway Interface v1.0.1 . https://www.python.org/dev/peps/pep-3333/#preface-for-readers-of-pep-333

WSGI规范(PEP 3333) 第一部分（概述) https://zhuanlan.zhihu.com/p/27600327

WSGI规范(PEP 3333) 第二部分（细节） https://zhuanlan.zhihu.com/p/27654172

wsgi 官方文档 https://wsgi.readthedocs.io/en/latest/learn.html

理解wsgi 理解Python WSGI https://www.letiantian.me/2015-09-10-understand-python-wsgi/

python wsgi 文档翻译 https://www.pyfdtic.com/2018/03/13/python-wsgi-doc/

阅读gunicorn代码文档 https://gunicorn.readthedocs.io/en/latest/index.html

分享快乐,留住感动. '2019-09-04 17:02:09' --frank