Google Protobuf
##简介
Protocol Buffers是一种轻量高效的结构化数据存储格式,独立于语言,平台,以二进制传输,可用于网络传输,配置,数据存储等.
Google提供了多种语言的实现,每一种都包含了相应的语言的编译器和库文件
优点
- 跨语言、支持多种语言, 包括 C++、Java、Go和 Python.
- 编解码的性能非常高.
- 相比与 xml json等传统的序列化工具, 它更小、更快、更简单.
- 支持不同协议版本的前向兼容.
- 支持定义可选和必选字段.
环境搭建
安装protoc编译工具
-
windows
从 官方仓库 上下载对应系统的最新发布包.
解压后会有一个bin文件夹下名为protoc.exe的应用程序.之后将protoc.exe所在的路径添加到环境变量 -
linux
-
安装依赖工具(联网)
$ sudo apt-get install autoconf automake libtool curl make g++ unzip libffi-dev -y -
进入protobuf文件
cd protobuf/ -
进行安装检测 并生成自动安装脚本
./autogen.sh
./configure -
进行编译C代码
make -
进行安装
sudo make install -
刷新linux共享库关系
sudo ldconfig
-
-
mac
brew install protobuf
测试
protoc -h
如果正常输出 相关指令 没有报任何error,为安装成功
安装go语言的protobuf编译器插件protoc-gen-go
-
方式一
go get -v -u github.com/golang/protobuf/protoc-gen-go
-
方式二
-
将压缩包github.com-golang-protobuf.zip下载并解压到$GOPATH/src/github.com/golang
-
$ cd $GOPATH/src/github.com/golang/protobuf/protoc-gen-go
-
$ go build
-
$ sudo cp protoc-gen-go $GOPATH/bin
-
在 Golang 中使用 protobuf
protoc-gen-go根据.proto文件生成go代码
格式
完整的.proto文件示例:
// [START declaration]
syntax = "proto3";
package ex;
import "google/protobuf/timestamp.proto";
// [END declaration]
// [START messages]
message Person {
string name = 1;
int32 id = 2; // Unique ID number for this person.
string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
string number = 1;
PhoneType type = 2;
}
repeated PhoneNumber phones = 4;
google.protobuf.Timestamp last_updated = 5;
}
// Our address book file is just one of these.
message AddressBook {
repeated Person people = 1;
}
// [END messages]
-
protobuf使用的.proto文件以包声明开始,包声明和golang中的pakcage对应,在某个包声明中定义的消息,会出现在对应的namespace命名空间中。import语句用来导入其他.proto文件中的消息定义,这样就可以在多个.proto文件中定义消息,然后关联使用了。
-
然后,你需要定义消息结构。一个消息包括多个带类型的成员。protobuf有许多标准的简单数据类型,包括bool, int32, float,double以及string, protobuf自带的.proto文件中也有一些消息结构定义,例如上面出现的google.protobuf.Timestamp。当然,你也可以根据这些类型,进一步构造其他消息,例如上面的Person包含了PhoneNumber消息,AddressBook包含了Person消息。你也可以在其他消息中定义消息类型,例如上面出现在PhoneNUmber在Person中进行定义。你还可以定义enum类型,例如上面的PhoneType,包含MOBILE,HOME和WORK三个可选值。
“=1”, “=2”是用来在二进制编码中标识对应字段的tag [1,15]之内的标识号在编码的时候会占用一个字节。[16,2047]之内的标识号则占用2个字节。所以应该为那些频繁出现的消息元素保留 [1,15]之内的标识号。切记:要为将来有可能添加的、频繁出现的标识号预留一些标识号。
最小的标识号可以从1开始,最大到2^29 - 1。不可以使用其中的[19000-19999]的标识号, Protobuf协议实现中对这些进行了预留。如果非要在.proto文件中使用这些预留标识号,编译时就会报警。
另外,每一个tag可以使用如下修饰符修饰:
-
singular: 表示这个字段可以有一个,也可以没有。如果没有的话,在编码的时候,不会占用空间。
-
repeated: 相当于go的切片,表示这个字段会重复0次或者更多次,这个字段里的值会按照顺序编码。
编译proto文件
定义完了.proto文件,下一步就是编译这个proto文件,我们假设这个proto文件名为addressbook.proto。为了编译这个文件,运行如下的语句:
protoc -I=$SRC_DIR --go_out=$DST_DIR $SRC_DIR/addressbook.proto
如果编译在proto文件目录进行,同时想编译到proto目录,可以使用如下命令:
protoc --go_out=. addressbook.proto
其中-I指定proto文件所在的位置, D S T _ D I R 指 定 生 成 文 件 所 在 的 位 置 , 这 里 − − c p p _ o u t 表 示 生 成 文 件 为 C + + 文 件 , 生 成 目 录 在 DST\_DIR指定生成文件所在的位置,这里--cpp\_out表示生成文件为C++文件,生成目录在 DST_DIR指定生成文件所在的位置,这里−−cpp_out表示生成文件为C++文件,生成目录在DST_DIR,$SRC_DIR/addressbook.proto。
编译会生成一个addressbook.pb.go文件。我们简单查看一下addressbook.pb.go文件:
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.23.0
// protoc v3.11.4
// source: addressbook.proto
package ex
import (
proto "github.com/golang/protobuf/proto"
timestamp "github.com/golang/protobuf/ptypes/timestamp"
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
reflect "reflect"
sync "sync"
)
const (
// Verify that this generated code is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
// Verify that runtime/protoimpl is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)
// This is a compile-time assertion that a sufficiently up-to-date version
// of the legacy proto package is being used.
const _ = proto.ProtoPackageIsVersion4
type Person_PhoneType int32
const (
Person_MOBILE Person_PhoneType = 0
Person_HOME Person_PhoneType = 1
Person_WORK Person_PhoneType = 2
)
// Enum value maps for Person_PhoneType.
var (
Person_PhoneType_name = map[int32]string{
0: "MOBILE",
1: "HOME",
2: "WORK",
}
Person_PhoneType_value = map[string]int32{
"MOBILE": 0,
"HOME": 1,
"WORK": 2,
}
)
func (x Person_PhoneType) Enum() *Person_PhoneType {
p := new(Person_PhoneType)
*p = x
return p
}
func (x Person_PhoneType) String() string {
return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x))
}
func (Person_PhoneType) Descriptor() protoreflect.EnumDescriptor {
return file_addressbook_proto_enumTypes[0].Descriptor()
}
func (Person_PhoneType) Type() protoreflect.EnumType {
return &file_addressbook_proto_enumTypes[0]
}
func (x Person_PhoneType) Number() protoreflect.EnumNumber {
return protoreflect.EnumNumber(x)
}
// Deprecated: Use Person_PhoneType.Descriptor instead.
func (Person_PhoneType) EnumDescriptor() ([]byte, []int) {
return file_addressbook_proto_rawDescGZIP(), []int{0, 0}
}
// [START messages]
type Person struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
Id int32 `protobuf:"varint,2,opt,name=id,proto3" json:"id,omitempty"` // Unique ID number for this person.
Email string `protobuf:"bytes,3,opt,name=email,proto3" json:"email,omitempty"`
Phones []*Person_PhoneNumber `protobuf:"bytes,4,rep,name=phones,proto3" json:"phones,omitempty"`
LastUpdated *timestamp.Timestamp `protobuf:"bytes,5,opt,name=last_updated,json=lastUpdated,proto3" json:"last_updated,omitempty"`
}
func (x *Person) Reset() {
*x = Person{}
if protoimpl.UnsafeEnabled {
mi := &file_addressbook_proto_msgTypes[0]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *Person) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*Person) ProtoMessage() {}
func (x *Person) ProtoReflect() protoreflect.Message {
mi := &file_addressbook_proto_msgTypes[0]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use Person.ProtoReflect.Descriptor instead.
func (*Person) Descriptor() ([]byte, []int) {
return file_addressbook_proto_rawDescGZIP(), []int{0}
}
func (x *Person) GetName() string {
if x != nil {
return x.Name
}
return ""
}
func (x *Person) GetId() int32 {
if x != nil {
return x.Id
}
return 0
}
func (x *Person) GetEmail() string {
if x != nil {
return x.Email
}
return ""
}
func (x *Person) GetPhones() []*Person_PhoneNumber {
if x != nil {
return x.Phones
}
return nil
}
func (x *Person) GetLastUpdated() *timestamp.Timestamp {
if x != nil {
return x.LastUpdated
}
return nil
}
// Our address book file is just one of these.
type AddressBook struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
People []*Person `protobuf:"bytes,1,rep,name=people,proto3" json:"people,omitempty"`
}
func (x *AddressBook) Reset() {
*x = AddressBook{}
if protoimpl.UnsafeEnabled {
mi := &file_addressbook_proto_msgTypes[1]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *AddressBook) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*AddressBook) ProtoMessage() {}
func (x *AddressBook) ProtoReflect() protoreflect.Message {
mi := &file_addressbook_proto_msgTypes[1]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use AddressBook.ProtoReflect.Descriptor instead.
func (*AddressBook) Descriptor() ([]byte, []int) {
return file_addressbook_proto_rawDescGZIP(), []int{1}
}
func (x *AddressBook) GetPeople() []*Person {
if x != nil {
return x.People
}
return nil
}
type Person_PhoneNumber struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Number string `protobuf:"bytes,1,opt,name=number,proto3" json:"number,omitempty"`
Type Person_PhoneType `protobuf:"varint,2,opt,name=type,proto3,enum=ex.Person_PhoneType" json:"type,omitempty"`
}
func (x *Person_PhoneNumber) Reset() {
*x = Person_PhoneNumber{}
if protoimpl.UnsafeEnabled {
mi := &file_addressbook_proto_msgTypes[2]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
}
func (x *Person_PhoneNumber) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*Person_PhoneNumber) ProtoMessage() {}
func (x *Person_PhoneNumber) ProtoReflect() protoreflect.Message {
mi := &file_addressbook_proto_msgTypes[2]
if protoimpl.UnsafeEnabled && x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use Person_PhoneNumber.ProtoReflect.Descriptor instead.
func (*Person_PhoneNumber) Descriptor() ([]byte, []int) {
return file_addressbook_proto_rawDescGZIP(), []int{0, 0}
}
func (x *Person_PhoneNumber) GetNumber() string {
if x != nil {
return x.Number
}
return ""
}
func (x *Person_PhoneNumber) GetType() Person_PhoneType {
if x != nil {
return x.Type
}
return Person_MOBILE
}
var File_addressbook_proto protoreflect.FileDescriptor
var file_addressbook_proto_rawDesc = []byte{
0x0a, 0x11, 0x61, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73, 0x62, 0x6f, 0x6f, 0x6b, 0x2e, 0x70, 0x72,
0x6f, 0x74, 0x6f, 0x12, 0x02, 0x65, 0x78, 0x1a, 0x1f, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f,
0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2f, 0x74, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61,
0x6d, 0x70, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0xaf, 0x02, 0x0a, 0x06, 0x50, 0x65, 0x72,
0x73, 0x6f, 0x6e, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28,
0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x02, 0x20,
0x01, 0x28, 0x05, 0x52, 0x02, 0x69, 0x64, 0x12, 0x14, 0x0a, 0x05, 0x65, 0x6d, 0x61, 0x69, 0x6c,
0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x65, 0x6d, 0x61, 0x69, 0x6c, 0x12, 0x2e, 0x0a,
0x06, 0x70, 0x68, 0x6f, 0x6e, 0x65, 0x73, 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x16, 0x2e,
0x65, 0x78, 0x2e, 0x50, 0x65, 0x72, 0x73, 0x6f, 0x6e, 0x2e, 0x50, 0x68, 0x6f, 0x6e, 0x65, 0x4e,
0x75, 0x6d, 0x62, 0x65, 0x72, 0x52, 0x06, 0x70, 0x68, 0x6f, 0x6e, 0x65, 0x73, 0x12, 0x3d, 0x0a,
0x0c, 0x6c, 0x61, 0x73, 0x74, 0x5f, 0x75, 0x70, 0x64, 0x61, 0x74, 0x65, 0x64, 0x18, 0x05, 0x20,
0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f,
0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52,
0x0b, 0x6c, 0x61, 0x73, 0x74, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x64, 0x1a, 0x4f, 0x0a, 0x0b,
0x50, 0x68, 0x6f, 0x6e, 0x65, 0x4e, 0x75, 0x6d, 0x62, 0x65, 0x72, 0x12, 0x16, 0x0a, 0x06, 0x6e,
0x75, 0x6d, 0x62, 0x65, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x06, 0x6e, 0x75, 0x6d,
0x62, 0x65, 0x72, 0x12, 0x28, 0x0a, 0x04, 0x74, 0x79, 0x70, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28,
0x0e, 0x32, 0x14, 0x2e, 0x65, 0x78, 0x2e, 0x50, 0x65, 0x72, 0x73, 0x6f, 0x6e, 0x2e, 0x50, 0x68,
0x6f, 0x6e, 0x65, 0x54, 0x79, 0x70, 0x65, 0x52, 0x04, 0x74, 0x79, 0x70, 0x65, 0x22, 0x2b, 0x0a,
0x09, 0x50, 0x68, 0x6f, 0x6e, 0x65, 0x54, 0x79, 0x70, 0x65, 0x12, 0x0a, 0x0a, 0x06, 0x4d, 0x4f,
0x42, 0x49, 0x4c, 0x45, 0x10, 0x00, 0x12, 0x08, 0x0a, 0x04, 0x48, 0x4f, 0x4d, 0x45, 0x10, 0x01,
0x12, 0x08, 0x0a, 0x04, 0x57, 0x4f, 0x52, 0x4b, 0x10, 0x02, 0x22, 0x31, 0x0a, 0x0b, 0x41, 0x64,
0x64, 0x72, 0x65, 0x73, 0x73, 0x42, 0x6f, 0x6f, 0x6b, 0x12, 0x22, 0x0a, 0x06, 0x70, 0x65, 0x6f,
0x70, 0x6c, 0x65, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x0a, 0x2e, 0x65, 0x78, 0x2e, 0x50,
0x65, 0x72, 0x73, 0x6f, 0x6e, 0x52, 0x06, 0x70, 0x65, 0x6f, 0x70, 0x6c, 0x65, 0x62, 0x06, 0x70,
0x72, 0x6f, 0x74, 0x6f, 0x33,
}
var (
file_addressbook_proto_rawDescOnce sync.Once
file_addressbook_proto_rawDescData = file_addressbook_proto_rawDesc
)
func file_addressbook_proto_rawDescGZIP() []byte {
file_addressbook_proto_rawDescOnce.Do(func() {
file_addressbook_proto_rawDescData = protoimpl.X.CompressGZIP(file_addressbook_proto_rawDescData)
})
return file_addressbook_proto_rawDescData
}
var file_addressbook_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
var file_addressbook_proto_msgTypes = make([]protoimpl.MessageInfo, 3)
var file_addressbook_proto_goTypes = []interface{}{
(Person_PhoneType)(0), // 0: ex.Person.PhoneType
(*Person)(nil), // 1: ex.Person
(*AddressBook)(nil), // 2: ex.AddressBook
(*Person_PhoneNumber)(nil), // 3: ex.Person.PhoneNumber
(*timestamp.Timestamp)(nil), // 4: google.protobuf.Timestamp
}
var file_addressbook_proto_depIdxs = []int32{
3, // 0: ex.Person.phones:type_name -> ex.Person.PhoneNumber
4, // 1: ex.Person.last_updated:type_name -> google.protobuf.Timestamp
1, // 2: ex.AddressBook.people:type_name -> ex.Person
0, // 3: ex.Person.PhoneNumber.type:type_name -> ex.Person.PhoneType
4, // [4:4] is the sub-list for method output_type
4, // [4:4] is the sub-list for method input_type
4, // [4:4] is the sub-list for extension type_name
4, // [4:4] is the sub-list for extension extendee
0, // [0:4] is the sub-list for field type_name
}
func init() { file_addressbook_proto_init() }
func file_addressbook_proto_init() {
if File_addressbook_proto != nil {
return
}
if !protoimpl.UnsafeEnabled {
file_addressbook_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*Person); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_addressbook_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*AddressBook); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
file_addressbook_proto_msgTypes[2].Exporter = func(v interface{}, i int) interface{} {
switch v := v.(*Person_PhoneNumber); i {
case 0:
return &v.state
case 1:
return &v.sizeCache
case 2:
return &v.unknownFields
default:
return nil
}
}
}
type x struct{}
out := protoimpl.TypeBuilder{
File: protoimpl.DescBuilder{
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: file_addressbook_proto_rawDesc,
NumEnums: 1,
NumMessages: 3,
NumExtensions: 0,
NumServices: 0,
},
GoTypes: file_addressbook_proto_goTypes,
DependencyIndexes: file_addressbook_proto_depIdxs,
EnumInfos: file_addressbook_proto_enumTypes,
MessageInfos: file_addressbook_proto_msgTypes,
}.Build()
File_addressbook_proto = out.File
file_addressbook_proto_rawDesc = nil
file_addressbook_proto_goTypes = nil
file_addressbook_proto_depIdxs = nil
}
其中在proto文件中的package对应于go语言中的package
使用
package main
import (
"awesomeProject1/ex"
"fmt"
"github.com/golang/protobuf/proto"
)
func main(){
p1:=&ex.Person{}
p1.Name="Jack"
p1.Email="me@123.com"
p1.Id=3
book:= ex.AddressBook{}
book.People=append(book.People,p1)
fmt.Println("book: ",book)
buffer,_:=proto.Marshal(&book)
fmt.Println("序列化book: ",buffer)
var data ex.AddressBook
proto.Unmarshal(buffer,&data)
fmt.Println("反序列化book: ",data)
}
book: {{{} [] [] } 0 [] [name:“Jack” id:3 email:“me@123.com”]}
序列化book: [10 20 10 4 74 97 99 107 16 3 26 10 109 101 64 49 50 51 46 99 111 109]
反序列化book: {{{} [] [] 0xc000030d38} 0 [] [name:“Jack” id:3 email:“me@123.com”]}
##附 proto数据类型对照(go)
proto go
----------------
double float64
float float32
int32 int32 使用变长编码,对于负值的效率很低,如果你的域有可能有负值,请使用sint64替代
uint32 uint32 使用变长编码
uint64 uint64 使用变长编码
sint32 int32 使用变长编码,这些编码在负值时比int32高效的多
sint64 int64 使用变长编码,有符号的整型值。编码时比通常的int64高效。
fixed32 uint32 总是4个字节,如果数值总是比总是比228大的话,这个类型会比uint32高效。
fixed64 uint64 总是8个字节,如果数值总是比总是比256大的话,这个类型会比uint64高效。
sfixed32 int32 总是4个字节
sfixed64 int64 总是8个字节
bool bool
string string 一个字符串必须是UTF-8编码或者7-bit ASCII编码的文本。
bytes []byte 可能包含任意顺序的字节数据。