gRPC python实现文件上传，以及使用流式通信上传超大文件

通过gRPC接口上传文件的优点

使用gRPC上传文件有许多优点：

1. 实现简单。就是简单地写个接口，而且rprotobuf 定义好了参数，Server端与Client对接方便
2. 无须部署额外软件。生产环境不需要部署额外软件，如WEB服务器，或第3方软件(如RabbitMQ)。
3. 支持超大文件传输。 分块传输，可重传，速度快。
4. 支持加密，还可以使用SSL/TLS加密，token验证等。

本文内容概要

• 使用单向通信方式文件
• 使用流式通信上传大文件

单向通信方式上传文件

这种方式编程简单，适合文件不大的应用场景。实现过程如下

定义 protobuf 接口文件

新建1个项目目录，创建uploadfile.proto文件

syntax="proto3";

message UploadRequest {
    string file_name = 1;
    int32  file_size = 2;
    bytes  content =3; 
}

message UploadResponse {
    string file_name =1; 
    int32  file_size = 2;
}

service UploadFiles {
    rpc UploadFile (UploadRequest) returns (UploadResponse);
}

编译proto文件

python -m grpc_tools.protoc --proto_path=. --python_out=. --grpc_python_out=. uploadfile.proto

生成如下文件：
uploadfile_pb2.py
uploadfile_pb2_grpc.py

服务端代码

提供1个 UploadFile() 接口函数，供客户端调用

import grpc
from concurrent.futures import ThreadPoolExecutor
import uploadfile_pb2
import uploadfile_pb2_grpc
import os
import timeit

class uploadfiles_service(uploadfile_pb2_grpc.UploadFilesServicer):
    """ 定义uploadfiles服务"""
    storage_path = "./upload/"   
    
    def UploadFile(self,request, context):
        """uploadfile 接口函数 """
        fname = request.file_name 
        fsize = request.file_size
        file_content = request.content 
        
        file_path = os.path.join(self.storage_path, fname)
        
        with open(file_path, "wb") as fo:
            fo.write(file_content) 
            print("Received file is saved in ", file_path)
        
        # resposne with parameters in protobuf message UploadResponse.
        return uploadfile_pb2.UploadResponse( 
                                             file_name=fname,
                                             file_size=os.path.getsize(file_path),
                                            )

class upload_server():
    
    def __init__(self,port=50001):
        self.port = port
         
    
    def setup_server(self):
        self.server = grpc.server(ThreadPoolExecutor(max_workers=10))
        uploadfile_pb2_grpc.add_UploadFilesServicer_to_server(uploadfiles_service(), self.server)        
    
    
    def start(self):
        host_address = f"[::]:{self.port}"
        self.server.add_insecure_port(host_address)
        self.server.start()
        self.server.wait_for_termination()


if __name__ == "__main__":
    server_a = upload_server()
    server_a.setup_server()
    server_a.start()
    print("-----gRPC service is ended----")

客户端代码

import grpc 

import uploadfile_pb2
import uploadfile_pb2_grpc
import os 
import timeit


def uploadfile_to_server(stub):
    fname = "img-1.jpg"
    with open(fname,'rb') as fo:
        file_data = fo.read()
    fsize = os.path.getsize(fname)
    print("original file size ",fsize)
    res = stub.UploadFile(uploadfile_pb2.UploadRequest(
        file_name=fname,
        file_size=fsize,
        content=file_data,
        )
    )
    if res:
        print("Response: ", res)
    else: 
        print("Abnormal:  no response from server")

with  grpc.insecure_channel("localhost:50001") as channel:
    stub = uploadfile_pb2_grpc.UploadFilesStub(channel)
    start_time = timeit.default_timer()
    uploadfile_to_server(stub)
    time_1 = timeit.default_timer() - start_time
    print("调用 gRPC接口时长: %.3f 秒"%(time_1))

测试

分别开启两个命令窗口，运行 server.py, client.py
客户端输出

original file size  2672290
Response:  file_name: "img-1.jpg"
file_size: 2672290

调用 gRPC接口时长: 0.049 秒

服务器端输出

Received file is saved in  ./upload/img-1.jpg

使用流式通信分块上传大文件

如果文件比较大，上传过程中很容易出现丢包现象，而且在1次传送又很占内存，特别是几百M的大文件，服务器侧也无法承受，因此解决思路为：

• 在Client 端将大文件分成多个更小的chunk，如64K, 用streaming方式上传，读１次上传1次
• 利用protobuf message 的oneof 字段修饰符，在第1个上传请求中，发送文件名，后面的request是文件内容chunk.
• 发送完成，server保存文件后，发送1条response, 其中包含 status, 指明结果

protobuf 接口定义文件代码:

syntax = "proto3";
package UploaderPkg;

message FileUploadRequest {
    oneof data {
        string name = 1;
        bytes chunk = 2;
    }
}

enum Status {
  PENDING = 0;
  IN_PROGRESS = 1;
  SUCCESS = 2;
  FAILED = 3;
}

message FileUploadResponse {
    string name = 1;
    Status status = 2;
}

service UploaderService {
    rpc uploadFile(stream FileUploadRequest) returns (FileUploadResponse) {}
}

测试结果：
client 输出:

D:\workplace\python\Simple_project\grpc\upload_largefile>py client.py
name: "img-1.jpg"
status: SUCCESS

调用 gRPC接口时长: 0.013 秒

服务端输出：

D:\workplace\python\Simple_project\grpc\upload_largefile>py server.py
Started gRPC server: 0.0.0.0:50051
Received file name:  img-1.jpg
received chunk 1
received chunk 2
received chunk 3
received chunk 4
received chunk 5
received chunk 6
received chunk 7
received chunk 8
received chunk 9
received chunk 10
received chunk 11
received chunk 12
received chunk 13
received chunk 14
received chunk 15
received chunk 16
received chunk 17
received chunk 18
received chunk 19
received chunk 20
received chunk 21
save file img-1.jpg successfully

完整的实现代码请点击 此地址下载