grpc-python

1.什么是RPC

RPC就是从一台机器(客户端)上通过参数传递的方式调用另一台机器(服务器)上的一个函数或方法(可以统称为服务)并得到返回的结果。
简言之，RPC使得程序能够像访问本地系统资源一样，去访问远端系统资源。
比较关键的一些方面包括：通讯协议、序列化、资源（接口）描述、服务框架、性能、语言支持等。

gRPC可以通过protobuf来定义接口，从而可以有更加严格的接口约束条件。

另外，通过protobuf可以将数据序列化为二进制编码，这会大幅减少需要传输的数据量，从而大幅提高性能。

gRPC可以方便地支持流式通信(理论上通过http2.0就可以使用streaming模式, 但是通常web服务的restful api似乎很少这么用，通常的流式数据应用如视频流，一般都会使用专门的协议如HLS，RTMP等，这些就不是我们通常web服务了，而是有专门的服务器应用。）

2.RPC调用过程

1）服务消费者（client客户端）通过调用本地服务的方式调用需要消费的服务；
2）客户端存根（client stub）接收到调用请求后负责将方法、入参等信息序列化（组装）成能够进行网络传输的消息体；
3）客户端存根（client stub）找到远程的服务地址，并且将消息通过网络发送给服务端；
4）服务端存根（server stub）收到消息后进行解码（反序列化操作）；
5）服务端存根（server stub）根据解码结果调用本地的服务进行相关处理；
6）本地服务执行具体业务逻辑并将处理结果返回给服务端存根（server stub）；
7）服务端存根（server stub）将返回结果重新打包成消息（序列化）并通过网络发送至消费方；
8）客户端存根（client stub）接收到消息，并进行解码（反序列化）；
9）服务消费方得到最终结果；
而RPC框架的实现目标则是将上面的第2-8步完好地封装起来，也就是把调用、编码/解码的过程给封装起来，让用户感觉上像调用本地服务一样的调用远程服务。

3.RPC的实现基础

1）需要有非常高效的网络通信，比如一般选择Netty作为网络通信框架；
2）需要有比较高效的序列化框架，比如谷歌的Protobuf序列化框架；
3）可靠的寻址方式（主要是提供服务的发现），比如可以使用Zookeeper来注册服务等等；（我司使用server mesh）
4）如果是带会话（状态）的RPC调用，还需要有会话和状态保持的功能

4.GRPC和restful对比

5.proto buffers常用数据类型

6.proto buffers特殊字符

7.GRPC常用错误码

8.Protocol Buffer的优点

1）序列化速度 & 反序列化速度快
    a. 编码 / 解码 方式简单（只需要简单的数学运算 = 位移等等）
    b. 采用 Protocol Buffer 自身的框架代码 和 编译器 共同完成
2）数据压缩效果好，即序列化后的数据量体积小
    a. 采用了独特的编码方式，如Varint、Zigzag编码方式等等
    b. 采用T-L-V的数据存储方式：减少了分隔符的使用 & 数据存储得紧凑
        T-L-V（Tag-Length-Value，一个消息就可以看成是多个字段的TLV序列拼接成的一个二进制字节流）

9.rpc框架为我们解决的问题

1）通讯问题，即A与B之间的通信，建立TCP连接
2）寻址问题，A通过RPC框架连接到B的服务器及特定端口和调用的方法名
3）参数序列化与反序列化，发起远程调用参数值需要二进制化，服务接收到二进制参数后需要反序列化

10.使用案例

// helloworld.proto
syntax = "proto3";

service Greeter {
    rpc SayHello(HelloRequest) returns (HelloResponse) {}
    rpc ClientReceiveStream(ClientReceiveStreamRequest) returns (stream ClientReceiveStreamResponse) {}
    rpc ClientSendStream(stream ClientSendStreamRequest) returns (ClientSendStreamResponse) {}
    rpc ClientServerStream(stream ClientServerStreamRequest) returns (stream ClientServerStreamResponse) {}
    rpc HeaderTest(HeaderTestRequest) returns (HeaderTestResponse) {}
    rpc ZipTransTest(ZipTransTestRequest) returns (ZipTransTestResponse) {}
    rpc InterceptorTest(InterceptorTestRequest) returns (InterceptorTestResponse) {}

    rpc PBTest(PBTestRequest) returns (PBTestResponse) {}
}

message HelloRequest {
    string data = 1;
}
message HelloResponse {
    string result = 1;
}


message ClientReceiveStreamRequest {
    string data = 1;
}
message ClientReceiveStreamResponse {
    string result = 1;
}


message ClientSendStreamRequest {
    string data = 1;
}
message ClientSendStreamResponse {
    string result = 1;
}


message ClientServerStreamRequest {
    string data = 1;
}
message ClientServerStreamResponse {
    string result = 1;
}


message HeaderTestRequest {
    string data = 1;
}
message HeaderTestResponse {
    string result = 1;
}


message ZipTransTestRequest {
    string data = 1;
}
message ZipTransTestResponse {
    string result = 1;
}


message InterceptorTestRequest {
    string data = 1;
}
message InterceptorTestResponse {
    string result = 1;
}


message PBTestRequest {
    string name = 1;
    int32 age = 2;

    message Phone {
        string color = 1;
        int32 size = 2;
    }

    repeated Phone phones = 3;
}
message PBTestResponse {
    string result = 1;
}

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Author: yuming.hou
# @Date:   2020-09-16 09:56
# server.py

from concurrent import futures
import time
import socket
import sys
import multiprocessing
import grpc
from contextlib import contextmanager
import helloworld_pb2 as pb2
import helloworld_pb2_grpc as grpc_pb2
from signals import Signal


PROCESS_COUNT = multiprocessing.cpu_count()
THREAD_COUNT = PROCESS_COUNT


def _abort(code, detail):
    def terminate(context):
        context.abort(code, detail)
    return grpc.unary_unary_rpc_method_handler(terminate)


class TestInterceptor(grpc.ServerInterceptor):
    def __init__(self, key, value, code, detail):
        self.key = key
        self.value = value
        self._abort = _abort(code, detail)

    def intercept_service(self, continuation, handler_call_details):
        headers = dict(handler_call_details.invocation_metadata)
        if headers.get(self.key) != self.value:
            return self._abort
        return continuation(handler_call_details)


class Greeter(grpc_pb2.GreeterServicer):
    def SayHello(self, request, context):
        data = request.data
        if data == 'hou':
            context.set_details('bug')
            context.set_code(grpc.StatusCode.DATA_LOSS)
            raise context
        return pb2.HelloResponse(result='recive data: %s' % data)

    def ClientReceiveStream(self, request, context):
        ind = 0
        while context.is_active():
            data = request.data
            if data == 'close':
                # cancel方式会导致客户端报异常，break不会
                context.cancel()
            time.sleep(1)
            ind += 1
            yield pb2.ClientReceiveStreamResponse(
                result='send %s %s' % (ind, data)
            )

    def ClientSendStream(self, request_iterator, context):
        ind = 0
        for request in request_iterator:
            print(ind, request.data)
            ind += 1
            if ind == 10:
                break

        return pb2.ClientSendStreamResponse(
            result="ok"
        )

    def ClientServerStream(self, request_iterator, context):
        ind = 0
        while context.is_active():
            for request in request_iterator:
                data = request.data
                ind += 1
                if data == 'close' or ind == 10:
                    break
                yield pb2.ClientServerStreamResponse(result='server send %s %s' % (ind, data))

    def HeaderTest(self, request, context):
        # 服务器接收客户端传的headers
        headers = context.invocation_metadata()
        for item in headers:
            print(item.key, item.value)

        data = request.data
        if data == 'hou':
            # 元组中的k，v必须都为字符串类型
            context.set_trailing_metadata((('name', data), ('age', '18')))
        else:
            context.set_trailing_metadata((('name', 'hou'), ('age', '123')))
        return pb2.HeaderTestResponse(result='hello')

    def ZipTransTest(self, request, context):
        # 服务器端添加压缩发送数据功能，如果想全局函数都添加则直接在server层，创建server时候添加参数
        # 解压缩不需要，grpc自动解压缩
        context.set_compression(grpc.Compression.Gzip)
        data = request.data
        return pb2.ZipTransTestResponse(result=data)

    def InterceptorTest(self, request, context):
        data = request.data
        return pb2.InterceptorTestResponse(result='Interceptor recive %s' % data)
        
    def PBTest(self, request, context):
        print(request.name, request.age)
        for i in request.phones:
            print(i.color, i.size)
        return pb2.PBTestResponse(result='hou')


def serve(port, thread_count):
    sig = Signal()
    # 添加拦截器
    validator = TestInterceptor('name', 'hou', grpc.StatusCode.UNAUTHENTICATED, 'Access denined')
    # 启动 rpc 服务, options中的两个参数为调整grpc接收和发送数据量大小限制，默认限制4MB
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=thread_count),
                         compression=grpc.Compression.Gzip,
                         interceptors=(validator,),
                         options=[
                             ('grpc.max_send_message_length', 50*1024*1024),
                             ('grpc.max_receive_message_length', 10*1024*1024),
                             ('grpc.so_reuesport', 1)
                         ])
    # 注册本地服务
    grpc_pb2.add_GreeterServicer_to_server(Greeter(), server)
    # 监听端口
    server.add_insecure_port('[::]:' + str(port))
    # 开始接收请求进行服务
    server.start()
    try:
        while True:
            if sig.sigquit():
                server.stop(0)
                break
            time.sleep(0.1)
    except KeyboardInterrupt:
        server.stop(0)


@contextmanager
def _reserve_port():
    sock = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)

    if sock.getsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT) == 0:
        raise RuntimeError("Failed to set SO_REUSEPORT")
    sock.bind(('', 50051))
    try:
        yield sock.getsockname()[1]
    finally:
        sock.close()


def main():
    with _reserve_port() as port:
        sys.stdout.flush()
        workers = []
        for _ in range(PROCESS_COUNT):
            worker = multiprocessing.Process(target=serve, args=(port, THREAD_COUNT))
            worker.start()
            workers.append(worker)

        for worker in workers:
            worker.join()

        # with futures.ProcessPoolExecutor(max_workers=PROCESS_COUNT) as executor:
        #     for _ in range(PROCESS_COUNT):
        #         workers.append(executor.submit(serve, port, THREAD_COUNT))
        #     for item in workers:
        #         item.result()


if __name__ == '__main__':
    main()
    # serve(50051, 4)

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Author: yuming.hou
# @Date:   2020-09-16 09:56
# client.py

import time
import random
import grpc
import helloworld_pb2 as pb2
import helloworld_pb2_grpc as grpc_pb2


def gen_data():
    ind = 0
    while True:
        time.sleep(1)
        data = str(random.randint(1, 10))
        ind += 1
        if ind == 6:
            break
        yield pb2.ClientSendStreamRequest(data=data)


def run():
    # 连接 rpc 服务器
    channel = grpc.insecure_channel('localhost:' + str(50051))
    # 调用 rpc 服务
    stub = grpc_pb2.GreeterStub(channel)

    # 双非流
    # 可以采用以下形式进行错误码传递，也可以在pb协议中定义状态码和msg，在业务数据中进行传输
    try:
        response = stub.SayHello(pb2.HelloRequest(data='hou'))
        print("client received: " + response.result)
    except Exception as e:
        print(e.details(), e.code().name, e.code().value)

    # 客户端接收流
    response = stub.ClientReceiveStream(pb2.ClientReceiveStreamRequest(data='close'))
    for item in response:
        print("client received stream: " + item.result)

    # 客户端发送流
    response = stub.ClientSendStream(gen_data())
    print(response.result)

    # 双流
    # timeout后客户端自动断开
    response = stub.ClientServerStream(gen_data(), timeout=10)
    for res in response:
        print(res.result)

    # header传输测试，with_call调用，metadata为客户端加的headers
    response, call = stub.HeaderTest.with_call(pb2.HeaderTestRequest(data='hou'),
                                               metadata=(('key1', 'value1'), ('key2', 'value2')))
    print(response.result)
    headers = call.trailing_metadata()
    for item in headers:
        print(item.key, item.value)
    channel.close()

    # 压缩传输数据
    response, call = stub.ZipTransTest.with_call(pb2.HeaderTestRequest(data='hou'),
                                           compression=grpc.Compression.Gzip)
    print(response.result)

    # grpc拦截器测试
    response = stub.InterceptorTest(pb2.InterceptorTestRequest(data='hou'),
                                    metadata=(('name', 'hou'),))
    print(response.result)
    
    # repeated修饰符测试
    req_body = pb2.PBTestRequest()
    req_body.name = 'hou'
    req_body.age = 18
    for i in range(3):
        phone = req_body.phones.add()
        phone.color = 'red'
        phone.size = i+1
    response = stub.PBTest(req_body)
    print(response.result)


if __name__ == '__main__':
    run()

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Author: yuming.hou
# @Date:   2020-09-16 09:56
# signals.py

import signal


class Signal(object):
    def __init__(self):
        self.quit = False
        self.reload = False

        signal.signal(signal.SIGUSR1, self.handler)
        signal.signal(signal.SIGUSR2, self.handler)

    def handler(self, signum, frame):
        if signal.SIGUSR1 == signum:
            self.quit = True

        if signal.SIGUSR2 == signum:
            self.reload = True

    def reset_reload(self):
        self.reload = False

    def sigquit(self):
        return self.quit

    def sigreload(self):
        return self.reload

# Makefile
proto:
	python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. helloworld.proto

#!/bin/sh
# 启动脚本
BASE_DIR=` cd "$(dirname "$0")"; cd ..;  pwd `

CONFIG_PATH=$BASE_DIR/src/conf
PID_FILE=$BASE_DIR/script/gunicorn.pid

PORT=`cat $CONFIG_PATH/config.py | grep port | awk -F'=' '{print $NF}' | sed 's/ //g' | awk -F"'" '{print $2}'`
#macos和linux不一样
PIDS=`/usr/sbin/lsof -i:$PORT | grep Python | awk '{print $2}'`

CMD_START="python3 server.py"

function get_master() {
    master=`cat $PID_FILE`
    for pid in $PIDS;
    do
        if [[ $pid -eq $master ]]
        then
            echo $master
            exit
        fi
    done

    echo -1
    exit
}

function do_start() {
    cd $BASE_DIR/src && nohup $CMD_START & 2>&1
    echo "grpc start"
}

function do_stop() {
    kill -USR1 $PIDS
    echo "grpc stop"
}

case "$1" in
    start)
        do_start
        ;;
    stop)
        do_stop
        ;;
    *)
        echo "Usage: $0 { start | stop }"
        exit 1
        ;;
esac

11.手撕RPC

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# client.py

import rpc_client

c = rpc_client.RPCClient()
c.conn('127.0.0.1', 50001)
res = c.add(1, 5)
print res

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# rpc_client.py

import socket
import json


class TCPClient(object):
    def __init__(self):
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    def conn(self, host, port):
        self.sock.connect((host, port))

    def send(self, data):
        self.sock.send(data)

    def recv(self, length):
        return self.sock.recv(length)


class RPCStub(object):
    def __getattr__(self, func):
        def _func(*args, **kwargs):
            d = {'method_name': func, 'method_args': args, 'method_kwargs': kwargs}
            self.send(json.dumps(d))
            data = self.recv(1024)
            return data

        setattr(self, func, _func)
        return _func


class RPCClient(TCPClient, RPCStub):
    pass

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# server.py
import rpc_server

def add(a, b):
    return a + b

s = rpc_server.RPCServer()
s.register(add)
s.loop(50001)

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# rpc_server.py
import socket
import json


class TCPServer(object):
    def __init__(self):
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    def bind_listen(self, port):
        self.sock.bind(('0.0.0.0', port))
        self.sock.listen(5)

    def get_client_message_and_return(self):
        client, _ = self.sock.accept()
        msg = client.recv(1024)
        res = self.get_result(msg)
        client.sendall(res)
        client.close()


class RPCServer(TCPServer):
    def __init__(self):
        TCPServer.__init__(self)
        self.funcs = dict()

    def register(self, func):
        self.funcs[func.__name__] = func

    def loop(self, port):
        self.bind_listen(port)
        while True:
            self.get_client_message_and_return()

    def get_result(self, msg):
        data = json.loads(msg)
        method_name = data['method_name']
        method_args = data['method_args']
        method_kwargs = data['method_kwargs']
        res = self.funcs[method_name](*method_args, **method_kwargs)
        return json.dumps({'res': res})