安装grpc
gRPC,即google Remote Procedure Call Protocol;在gRPC里,客户端可以直接调用不同机器上的服务应用的方法,就像本地对象一样,所以创建分布式应用和服务就变简单了。
gRPC是基于定义一个服务,指定一个可以远程调用的带有参数和返回类型的的方法。在服务端,服务实现这个接口并且运行gRPC服务处理客户端调用。在客户端,有一个stub提供和服务端相同的方法。
准备
- 假设要将MY_INSTALL_DIR 作为安装目录。
- 注意一定执行要export,设置好设置 CMAKE_INSTALL_PREFIX,这样有利于example项目的编译
$ export MY_INSTALL_DIR=$HOME/.local
# 确保路径已经存在
$ mkdir -p $MY_INSTALL_DIR
# 将bin目录安装到环境中
$ export PATH="$MY_INSTALL_DIR/bin:$PATH"
- 安装cmake 3.13之后的版本
$ wget -q -O cmake-linux.sh https://github.com/Kitware/CMake/releases/download/v3.19.6/cmake-3.19.6-Linux-x86_64.sh
$ sh cmake-linux.sh -- --skip-license --prefix=$MY_INSTALL_DIR
CMake Installer Version: 3.19.6, Copyright (c) Kitware
This is a self-extracting archive.
The archive will be extracted to: /home/oceanstar/.local
Using target directory: /home/oceanstar/.local
Extracting, please wait...
Unpacking finished successfully
// 效果: /home/oceanstar/.local目录下多了bin、doc、man三个文件
$ rm cmake-linux.sh
- 安装一下其他必备工具
$ sudo yum/apt install -y build-essential autoconf libtool pkg-config
下载
- 下载grpc源码:
git clone https://gitee.com/jiangxy__loey/grpc.git
-
- 下载grpc依赖的三方库
cd grpc
-
- 修改.gitmodules文件,替换其中的github源为gitee源
[submodule "third_party/zlib"]
path = third_party/zlib
#url = https://github.com/madler/zlib
url = https://gitee.com/jiangxy__loey/zlib.git
# When using CMake to build, the zlib submodule ends up with a
# generated file that makes Git consider the submodule dirty. This
# state can be ignored for day-to-day development on gRPC.
ignore = dirty
[submodule "third_party/protobuf"]
path = third_party/protobuf
url = https://gitee.com/jiangxy__loey/protobuf.git
#url = https://github.com/google/protobuf.git
[submodule "third_party/gflags"]
path = third_party/gflags
#url = https://github.com/gflags/gflags.git
url = https://gitee.com/jiangxy__loey/gflags.git
[submodule "third_party/googletest"]
path = third_party/googletest
#url = https://github.com/google/googletest.git
url = https://gitee.com/jiangxy__loey/googletest.git
[submodule "third_party/benchmark"]
path = third_party/benchmark
#url = https://github.com/google/benchmark
url = https://gitee.com/jiangxy__loey/benchmark.git
[submodule "third_party/boringssl-with-bazel"]
path = third_party/boringssl-with-bazel
#url = https://github.com/google/boringssl.git
url = https://gitee.com/jiangxy__loey/boringssl-with-bazel.git
[submodule "third_party/re2"]
path = third_party/re2
#url = git://github.com/google/re2.git
url = https://gitee.com/hejuncheng1/re2.git
[submodule "third_party/cares/cares"]
path = third_party/cares/cares
#url = https://github.com/c-ares/c-ares.git
url = https://gitee.com/jiangxy__loey/cares.git
#branch = cares-1_12_0
[submodule "third_party/bloaty"]
path = third_party/bloaty
#url = https://github.com/google/bloaty.git
url = https://gitee.com/jiangxy__loey/bloaty.git
[submodule "third_party/abseil-cpp"]
path = third_party/abseil-cpp
#url = https://github.com/abseil/abseil-cpp.git
url = https://gitee.com/jiangxy__loey/abseil-cpp.git
branch = lts_2020_02_25
[submodule "third_party/envoy-api"]
path = third_party/envoy-api
#url = https://github.com/envoyproxy/data-plane-api.git
url = https://gitee.com/jiangxy__loey/envoy-api.git
[submodule "third_party/googleapis"]
path = third_party/googleapis
#url = https://github.com/googleapis/googleapis.git
url = https://gitee.com/jiangxy__loey/googleapis.git
[submodule "third_party/protoc-gen-validate"]
path = third_party/protoc-gen-validate
#url = https://github.com/envoyproxy/protoc-gen-validate.git
url = https://gitee.com/jiangxy__loey/protoc-gen-validate.git
[submodule "third_party/udpa"]
path = third_party/udpa
#url = https://github.com/cncf/udpa.git
url = https://gitee.com/jiangxy__loey/udpa.git
[submodule "third_party/libuv"]
path = third_party/libuv
#url = https://github.com/libuv/libuv.git
url = https://gitee.com/jiangxy__loey/libuv.git
- 下载三方库
git submodule update --init
第一次
$ git clone --recurse-submodules -b v1.50.0 --depth 1 --shallow-submodules https://gitclone.com/github.com/grpc/grpc.git
如果速度慢,取消SSL验证并且手动到third_party 下载如下:
- git config --global http.sslVerify false
- git clone https://gitclone.com/github.com/cncf/xds.git
- git clone https://gitclone.com/github.com/google/re2.git
- git clone https://gitclone.com/github.com/google/protobuf.git
- git clone https://gitclone.com/github.com/open-telemetry/opentelemetry-proto.git
- git clone https://gitclone.com/github.com/google/googletest.git
- https://gitclone.com/github.com/google/bloaty.git
- git clone https://gitclone.com/github.com/boringssl-with-bazel.git
cd grpc
git submodule update --init
安装
$ cd grpc
$ mkdir -p cmake/build
$ pushd cmake/build
$ cmake -DgRPC_INSTALL=ON \
-DgRPC_BUILD_TESTS=OFF \
-DCMAKE_INSTALL_PREFIX=$MY_INSTALL_DIR \
../..
$ make -j 4
$ make install
$ popd
快速开始
C++
编译运行
- 选择一个目录,然后编译
$ cd examples/cpp/helloworld
$ mkdir -p cmake/build
$ pushd cmake/build
$ cmake -DCMAKE_PREFIX_PATH=$MY_INSTALL_DIR ../..
$ make -j 4
- 从示例构建目录examples/cpp/helloworld/cmake/build运行示例:
# 运行服务端
$ ./greeter_server
# 新开一个终端
$ ./greeter_client
Greeter received: Hello world
更新grpc
-
让我们来看看如何在服务器上使用一个额外的方法来更新应用程序,以便客户端调用。
-
当前,服务器和客户端存根都有一个SayHello()RPC方法,该方法从客户端获取HelloRequest参数,并从服务器返回HelloReply,该方法的定义如下:
// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {}
}
// The request message containing the user's name.
message HelloRequest {
string name = 1;
}
// The response message containing the greetings
message HelloReply {
string message = 1;
}
第一步, 打开 examples/protos/helloworld.proto,添加一个新方法
// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {}
// Sends another greeting
rpc SayHelloAgain (HelloRequest) returns (HelloReply) {}
}
// The request message containing the user's name.
message HelloRequest {
string name = 1;
}
// The response message containing the greetings
message HelloReply {
string message = 1;
}
重新编译proto
cd examples/cpp/helloworld/cmake/build
make -j 4
更新服务端
greeter_server.cc中实现新方法SayHelloAgain
class GreeterServiceImpl final : public Greeter::Service {
Status SayHello(ServerContext* context, const HelloRequest* request,
HelloReply* reply) override {
// ...
}
Status SayHelloAgain(ServerContext* context, const HelloRequest* request,
HelloReply* reply) override {
std::string prefix("Hello again ");
reply->set_message(prefix + request->name());
return Status::OK;
}
};
实现客户端
GreeterClient.cc中实现一个新方法
class GreeterClient {
public:
// ...
std::string SayHello(const std::string& user) {
// ...
}
std::string SayHelloAgain(const std::string& user) {
// Follows the same pattern as SayHello.
HelloRequest request;
request.set_name(user);
HelloReply reply;
ClientContext context;
// Here we can use the stub's newly available method we just added.
Status status = stub_->SayHelloAgain(&context, request, &reply);
if (status.ok()) {
return reply.message();
} else {
std::cout << status.error_code() << ": " << status.error_message()
<< std::endl;
return "RPC failed";
}
}
main方法中调用
int main(int argc, char** argv) {
// ...
std::string reply = greeter.SayHello(user);
std::cout << "Greeter received: " << reply << std::endl;
reply = greeter.SayHelloAgain(user);
std::cout << "Greeter received: " << reply << std::endl;
return 0;
}
重新运行
cd examples/cpp/helloworld/cmake/build
make -j 4
# 运行服务端
$ ./greeter_server
# 新开一个终端
$ ./greeter_client
Greeter received: Hello world
python(不要看)
grpcio-tools和protobuf存在严重的版本冲突
准备
- 要求python3.5之上
- 要求pip9.0.1之上
你可以升级你的pip
$ python3 -m pip install --upgrade pip
安装grpc
python3 -m pip install grpcio
python3 -m pip install grpcio-tools
运行实例
$ cd grpc/examples/python/helloworld
$ python3 greeter_server.py
$ python3 greeter_client.py
Greeter client received: Hello, you!
错误
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
1. Downgrade the protobuf package to 3.20.x or lower.
2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
解决:
python3 -m pip uninstall protobuf
python3 -m pip install protobuf==3.19.0
import 的时候可能会报错:
- ImportError: cannot import name 'builder' from 'google.protobuf.internal'
```py
这时候需要升级:
- pip3 install --upgrade protobuf
# basic
## C++
### 编译示例
```cpp
$ cd examples/cpp/route_guide
$ mkdir -p cmake/build
$ cd cmake/build
$ cmake -DCMAKE_PREFIX_PATH=$MY_INSTALL_DIR ../..
定义服务
比如 examples/protos/route_guide.proto中用service
关键字定义了一个服务:
service RouteGuide {
...
}
然后在服务中定义rpc方法,指定请求和响应类型。
python
如下所示,文件名称为helloworld.proto
syntax = "proto3";
service Greeter {
rpc SayHello(HelloRequest) returns (HelloReply) {}
}
message HelloRequest {
string name = 1;
}
message HelloReply {
string message = 1;
}
在Python中,我们首先需要安装第三方的库来操作gRPC的协议,具体安装的库如下:
pip3 install grpcio
pip3 install grpcio-tools
安装成功后,执行如下命令生成pb2文件和pb2_grpc的文件,执行命令为:
python3 -m grpc_tools.protoc --python_out=. --grpc_python_out=. -I. helloworld.proto
执行成功后,在当前的目录下就会生成名称为:helloworld_pb2.py和helloworld_pb2_grpc.py的文件,helloworld_pb2_grpc.py文件主要是服务端的内容,文件内容如下:
# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
"""Client and server classes corresponding to protobuf-defined services."""
import grpc
import helloworld_pb2 as helloworld__pb2
class GreeterStub(object):
"""Missing associated documentation comment in .proto file."""
def __init__(self, channel):
"""Constructor.
Args:
channel: A grpc.Channel.
"""
self.SayHello = channel.unary_unary(
'/Greeter/SayHello',
request_serializer=helloworld__pb2.HelloRequest.SerializeToString,
response_deserializer=helloworld__pb2.HelloReply.FromString,
)
class GreeterServicer(object):
"""Missing associated documentation comment in .proto file."""
def SayHello(self, request, context):
"""Missing associated documentation comment in .proto file."""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def add_GreeterServicer_to_server(servicer, server):
rpc_method_handlers = {
'SayHello': grpc.unary_unary_rpc_method_handler(
servicer.SayHello,
request_deserializer=helloworld__pb2.HelloRequest.FromString,
response_serializer=helloworld__pb2.HelloReply.SerializeToString,
),
}
generic_handler = grpc.method_handlers_generic_handler(
'Greeter', rpc_method_handlers)
server.add_generic_rpc_handlers((generic_handler,))
# This class is part of an EXPERIMENTAL API.
class Greeter(object):
"""Missing associated documentation comment in .proto file."""
@staticmethod
def SayHello(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(request, target, '/Greeter/SayHello',
helloworld__pb2.HelloRequest.SerializeToString,
helloworld__pb2.HelloReply.FromString,
options, channel_credentials,
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
helloworld_pb2.py的文件内容为:
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# source: helloworld.proto
"""Generated protocol buffer code."""
from google.protobuf import descriptor as _descriptor
from google.protobuf import message as _message
from google.protobuf import reflection as _reflection
from google.protobuf import symbol_database as _symbol_database
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
DESCRIPTOR = _descriptor.FileDescriptor(
name='helloworld.proto',
package='',
syntax='proto3',
serialized_options=None,
create_key=_descriptor._internal_create_key,
serialized_pb=b'\n\x10helloworld.proto\"\x1c\n\x0cHelloRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\"\x1d\n\nHelloReply\x12\x0f\n\x07message\x18\x01 \x01(\t23\n\x07Greeter\x12(\n\x08SayHello\x12\r.HelloRequest\x1a\x0b.HelloReply\"\x00\x62\x06proto3'
)
_HELLOREQUEST = _descriptor.Descriptor(
name='HelloRequest',
full_name='HelloRequest',
filename=None,
file=DESCRIPTOR,
containing_type=None,
create_key=_descriptor._internal_create_key,
fields=[
_descriptor.FieldDescriptor(
name='name', full_name='HelloRequest.name', index=0,
number=1, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=b"".decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
],
extensions=[
],
nested_types=[],
enum_types=[
],
serialized_options=None,
is_extendable=False,
syntax='proto3',
extension_ranges=[],
oneofs=[
],
serialized_start=20,
serialized_end=48,
)
_HELLOREPLY = _descriptor.Descriptor(
name='HelloReply',
full_name='HelloReply',
filename=None,
file=DESCRIPTOR,
containing_type=None,
create_key=_descriptor._internal_create_key,
fields=[
_descriptor.FieldDescriptor(
name='message', full_name='HelloReply.message', index=0,
number=1, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=b"".decode('utf-8'),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
serialized_options=None, file=DESCRIPTOR, create_key=_descriptor._internal_create_key),
],
extensions=[
],
nested_types=[],
enum_types=[
],
serialized_options=None,
is_extendable=False,
syntax='proto3',
extension_ranges=[],
oneofs=[
],
serialized_start=50,
serialized_end=79,
)
DESCRIPTOR.message_types_by_name['HelloRequest'] = _HELLOREQUEST
DESCRIPTOR.message_types_by_name['HelloReply'] = _HELLOREPLY
_sym_db.RegisterFileDescriptor(DESCRIPTOR)
HelloRequest = _reflection.GeneratedProtocolMessageType('HelloRequest', (_message.Message,), {
'DESCRIPTOR' : _HELLOREQUEST,
'__module__' : 'helloworld_pb2'
# @@protoc_insertion_point(class_scope:HelloRequest)
})
_sym_db.RegisterMessage(HelloRequest)
HelloReply = _reflection.GeneratedProtocolMessageType('HelloReply', (_message.Message,), {
'DESCRIPTOR' : _HELLOREPLY,
'__module__' : 'helloworld_pb2'
# @@protoc_insertion_point(class_scope:HelloReply)
})
_sym_db.RegisterMessage(HelloReply)
_GREETER = _descriptor.ServiceDescriptor(
name='Greeter',
full_name='Greeter',
file=DESCRIPTOR,
index=0,
serialized_options=None,
create_key=_descriptor._internal_create_key,
serialized_start=81,
serialized_end=132,
methods=[
_descriptor.MethodDescriptor(
name='SayHello',
full_name='Greeter.SayHello',
index=0,
containing_service=None,
input_type=_HELLOREQUEST,
output_type=_HELLOREPLY,
serialized_options=None,
create_key=_descriptor._internal_create_key,
),
])
_sym_db.RegisterServiceDescriptor(_GREETER)
DESCRIPTOR.services_by_name['Greeter'] = _GREETER
# @@protoc_insertion_point(module_scope)
下面我们来编写服务端的代码,具体实现代码如下:
#!/usr/bin/env python
#!coding:utf-8
import grpc
import helloworld_pb2
import helloworld_pb2_grpc
from concurrent import futures
import time
import os
class HelloWorldServicer(helloworld_pb2_grpc.GreeterServicer):
def SayHello(self,request,context):
'''客户端与服务端之间进行交互'''
return helloworld_pb2.HelloReply(message='Hello {0}'.format(request.name))
def serve():
server=grpc.server(futures.ThreadPoolExecutor(max_workers=os.cpu_count()))
helloworld_pb2_grpc.add_GreeterServicer_to_server(HelloWorldServicer(),server)
#指定服务端的测试地址信息
server.add_insecure_port('[::]:50051')
server.start()
try:
while True:
time.sleep(60*60*24)
except KeyboardInterrupt:
server.stop(0)
if __name__ == '__main__':
serve()
不管是客户端的代码还是服务端的代码,都需要导入helloworld_pb2和helloworld_pb2_grpc。
客户端实现的源码如下:
#!/usr/bin/env python
#!coding:utf-8
import grpc
import helloworld_pb2
import helloworld_pb2_grpc
import requests
def getData():
r=requests.get(url='https://api.muxiaoguo.cn/api/QqInfo?qq=2839168630')
return r.text
def run():
#连接服务端的程序
channel=grpc.insecure_channel('localhost:50051')
#调用rcp的服务
stub=helloworld_pb2_grpc.GreeterStub(channel)
r=stub.SayHello(helloworld_pb2.HelloRequest(name=getData()))
print('server response data:{0}'.format(r.message))
if __name__ == '__main__':
run()
API
针对gRPC的协议中,它的交互主要是单向,应答流,请求流,和双向流
- 普通grpc:一问一答
- 应答流:一次请求,服务端N次返回结果,也就是说通过循环的方式拿到服务端的数据
- 请求流:N次请求,一次返回结果,请求流需要使迭代器的方式,也就是yield的方式来进行发送请求
- 双向流:N次请求,N次回应,请求方是迭代器的方式,拿到服务端就是循环的方式。
其实在gRPC协议中,针对这种流式的请求在编程模式中更多实用的是异步编程,同步编程方式很难复合流式的诉求,比如N次请求N次回应,这中间本身就是一个持续的过程,而同步交互更多简单粗暴的就是请求了得尽快拿到回应数据,而异步就是在N次发送请求中,不断的发送,服务端然后逐步的返回来结果信息。既然是异步,也就是离不开协程,而协程本身在计算机的层面是不存在的,计算机里面更多的是进程和线程,而协程更多是用户行为控制的一种线程,它的优势是遇到IO就切换,这很符合异步的交互。不管是同步交互还是同步通信,它最大的缺陷就是当有一个请求存在占用太多的计算能力以及逻辑存在问题,这就导致队列堵塞,而造成消息(任务)的积压,从而形成TimeOut的超时情况。
python
版本
grpc.__version__
创建客户端
grpc.insecure_channel(target, options=None, compression=None)
功能:
- 创建到服务器的不安全通道。
- 返回的Channel是线程安全的。
参数
- target–服务器地址
- options–用于配置通道的键值对(gRPC Core运行时中的channel_arguments)的可选列表。
- compression - 一个可选的值,表示在通道的有效期内使用的压缩方法。这是一个试验性的选项。
返回:
- 一个通道
grpc.secure_channel(target, credentials, options=None, compression=None)
功能:
- 创建到服务器的安全通道。
- 返回的Channel是线程安全的。
参数
- target–服务器地址
- credentials – 证书实例
- options–用于配置通道的键值对(gRPC Core运行时中的channel_arguments)的可选列表。
- compression - 一个可选的值,表示在通道的有效期内使用的压缩方法。这是一个试验性的选项。
返回:
- 一个通道
grpc.intercept_channel(channel, *interceptors)
功能:
- 通过一组拦截器拦截通道
参数:
- channel:通道
- interceptors
返回:
- 一个通道
拦截器的作用:根据我们的函数调用(是unary_unary,还是其他)来确定接收到对应的方法消息时,我们就截下来进行处理。还能进行判断是否符合处理要求。 在 gRPC 中通信大概会有两种 RPC 方法(流和一元)与拦截器的对应的是:
- 一元通信模式:一元拦截器。在一元 RPC通信模式中,gRPC 服务器端和客户端在通信时始终只有一个请求和一个响应。
- 流通信模式:流拦截器。有三种细分的拦截器。指一端或者两端一次性返回多条消息;
分类:
- unary_unary :一元通信模式,方法注册后,对应的消息都会被拦截( 服务器端和 客户端在通信时始终只有一个请求和一个响应);
- stream_stream :双向流 RPC (客户端会发送多个请求给服务端,服务器在收到客户端的请求后,也会发送多个响应的序列,这就叫“流”)
- stream_unary :请求流 RPC (客户端会发送多个请求给服务端,但服务端只会发送一条响应到客户端)
- unary_stream :应答流 RPC(客户端发送一个请求给服务端,服务端收到客户端请求后会发多个响应序列)
创建证书客户端
grpc.ssl_channel_credentials(root_certificates=None, private_key=None, certificate_chain=None)
grpc.metadata_call_credentials(metadata_plugin, name=None)
grpc.access_token_call_credentials(access_token)
grpc.composite_call_credentials(*call_credentials)
grpc.composite_channel_credentials(channel_credentials, *call_credentials)
grpc.local_channel_credentials(local_connect_type=grpc.LocalConnectionType.LOCAL_TCP)
创建服务器
grpc.server(thread_pool, handlers=None, interceptors=None, options=None, maximum_concurrent_rpcs=None, compression=None, xds=False)
创建带证书的服务器
grpc.ssl_server_credentials(private_key_certificate_chain_pairs, root_certificates=None, require_client_auth=False)
grpc.ssl_server_certificate_configuration(private_key_certificate_chain_pairs, root_certificates=None)
grpc.dynamic_ssl_server_credentials(initial_certificate_configuration, certificate_configuration_fetcher, require_client_authentication=False)
grpc.local_server_credentials(local_connect_type=grpc.LocalConnectionType.LOCAL_TCP)
创建本地连接
class grpc.LocalConnectionType(value)
用于创建本地凭据的本地连接类型。
- UDS: Unix domain socket connections
- LOCAL_TCP:本地TCP连接
RPC方法处理程序
grpc.unary_unary_rpc_method_handler(behavior, request_deserializer=None, response_serializer=None)
Parameters
behavior – The implementation of an RPC that accepts one request and returns one response.
request_deserializer – An optional deserializer for request deserialization.
response_serializer – An optional serializer for response serialization.
Returns
An RpcMethodHandler object that is typically used by grpc.Server.