文章目录
标题 | 版本号 | 作者 | 技术栈版本号 | IDE | 官网 | |
---|---|---|---|---|---|---|
Protocol Buffers(protobuf)学习_v0.0.1_持续更新。。。 | v0.0.1 | 飞豺 | 8416837 | pb3.10.1 golang1.13.4 | LiteIDE | protobuf golang |
尊重版权,转载请注明出处:https://blog.csdn.net/cc007cc009/article/details/103107725
概念
- protocol buffers 是一种语言无关、平台无关、可扩展的序列化结构数据的方法,它可用于(数据)通信协议、数据存储等。它的同行是风行一时的xml以及json.二进制。
- 用途较广。如区块链相关开发,本章亦有相关实践,详情见:调用区块链的gRPC;消息队列,如腾讯开源的TubeMQ;
- 体积较小,节省资源。
原理
数据结构
- 数字为key,更节省资源;
- 定义简单,维护成本低;
- 能被编译,开发更高效;
操作步骤
- 第一步,定义消息结构体,即.proto文件;
- 第二步,使用protobuf编译器编译.proto文件,编译环境为集成开发工具或者命令行终端;
- 第三步,使用C++/Java、c sharp、nodejs以及golang等对应的protobuf API读写消息,本文使用golang开发gRPC;
开发环境的准备
Win
- 下载 protoc-3.10.1-win64.zip
- 解压,把它的根目录配置到环境变量配置:Path
- 检验。
protoc --version
- 新建robot.proto
syntax = "proto3"; // 版本号
message Robot { // 类结构体
int32 id = 999999; // 数字作为key,省空间。不一定要连续。
string name = 2;
string level = 3;
}
- 编译。
protoc --go_out=. --python_out=. robot.proto # 编译成Golang、Python模型
执行上个命令,报错了--go_out: protoc-gen-go: Plugin failed with status code 1.
看来缺少插件,需安装golang和插件。
- 安装golang
接上文问题。下载 https://studygolang.com/dl/golang/go1.13.4.windows-amd64.msi,因为开发机是Win OS.Linux安装见区块链文档。安装完毕,发现已经自动配置了环境变量:,
配置环境变量,GOROOT,即golang根目录,也即msi文件的安装目录;GOPATH,即工作空间;注意,win10环境,GOPATH配置到当前用户;
检验一下↓
- 安装插件
go get -u github.com/golang/protobuf/protoc-gen-go # 使用go安装protoc-gen-go插件,该插件编译protobuf时有用到。因网络问题有可能失败
下载完毕后,可以在C:\Users\Administrator\go\bin
里面找到protoc-gen-go.exe
,为什么能找到,——因为这是默认的GOPATH目录。但是自定义的GOPATH里面没找到这个文件,说明自定义配置不OK.——这样,插件也装好了。
- 重新编译proto文件,成功。
↑生成了两个文件,go、py.
Ubuntu环境
sudo apt-get install autoconf automake libtool curl make g++ unzip
sudo apt-get install autoconf # 安装自动配置
git clone https://github.com/google/protobuf.git # 下载源码
git clone https://gitee.com/githubmirror/protobuf # 国内镜像更快
cd protobuf
git submodule update --init --recursive
bash autogen.sh # 找不到这个文件
./configure # 找不到这个文件
sudo apt install golang-go # 安装golang
go version # 查看是否成功
# 设置GOPATH
make
make check
sudo make install
make报错:
# 显示
apt-cache showpkg libcurl3-gnutls
# 安装:
sudo apt-get install libcurl3-gnutls=7.47.0-1ubuntu2
还有没安装go也会报命令错误
gopath配置:
sudo gedit ~/.profile
export GOROOT="/usr/lib/go" # 引号内设置为你自己的go安装目录
export GOBIN=$GOROOT/bin
export GOPATH="/home/project/gopath" # 引号内设置为自己的go项目的工作区间
export PATH=$PATH:$GOPATH/bin # 原路径后用冒号连接新路径
source ~/.profile
解决下载慢:
export GOPROXY=https://mirrors.aliyun.com/goproxy/
安装protoc:
go get -u github.com/golang/protobuf/protoc-gen-go
apt install protobuf-compiler
apt-cache madison protobuf-compiler # 查看存在的版本号
版本号不合适,重新安装:
wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz
tar -xzvf protobuf-2.5.0.tar.gz
pushd protobuf-2.5.0 && sudo ./configure --prefix=/usr/local && sudo make && sudo make install && popd
安装成功后:
# 报错:protoc: error while loading shared libraries: libprotoc.so.8: cannot open shared object file: No such file or directory
export LD_LIBRARY_PATH=/usr/local/lib
相关工具
GO IDE
golang LiteIDE
# native compiler windows 386
GOROOT=G:\Program Files (x86)\golang # go安装目录
GOBIN=G:\Program Files (x86)\golang\bin
#GOARCH=386
#GOOS=windows
#CGO_ENABLED=1
PATH=c:\mingw32\bin;%GOROOT%\bin;%PATH%
#LITEIDE_GDB=gdb
LITEIDE_MAKE=mingw32-make
LITEIDE_TERM=%COMSPEC%
LITEIDE_TERMARGS=
LITEIDE_EXEC=%COMSPEC%
LITEIDE_EXECOPT=/C
保存成功的响应日志:
23:18:34 LiteEnv: reset system environment for "go env"
23:18:34 GolangCode: gocode set lib-path "C:\Users\cc\go"
23:18:34 LiteBuild: go environment changed
23:18:34 GolangPackage: Found go bin at C:\Go\bin\go.exe
23:18:34 GolangPackage: GOROOT=C:\Go
23:18:34 GolangPackage: GOPATH=C:\Users\cc\go
23:18:34 GolangCode: go environment changed
23:18:34 GolangCode: Found gocode at C:/dev/tools/ide/liteidex36.2.windows-qt5.9.5/liteide/bin/gocode.exe
点击 -> 工具 => 管理GOPATH.选择Win环境变量配置的值。
- 新建。点击新建项目,选择GOPATH、模板。如↓
- 下述代码数据操作部分运行成功的保证是:成功从github.com下载了依赖、有数据库
// ccgotest project main.go
package main
import (
"database/sql"
"fmt"
"log"
_ "github.com/go-sql-driver/mysql"
)
func main() {
var i int = 1
i += 2
j := i + 3
for {
if j < i {
break
}
fmt.Println("Hello World!")
j--
fmt.Println("j === ", j, ",i === ", i)
}
// DB测试
db, err := sql.Open("mysql", "root:pass@tcp(localhost:3306)/eladmin?charset=utf8")
if err != nil {
panic(err)
}
// DB操作 - start
username1 := "test"
stmt, err := db.Prepare("SELECT id FROM user WHERE username = ?")
if err != nil {
log.Println("数据库错误")
log.Fatal(err)
}
rows, err := stmt.Query(username1)
fmt.Println(err, rows)
// process rows
for rows.Next() {
// var username string
// var id int32
var value string
if err := rows.Scan(&value); err == nil {
fmt.Println("结果")
fmt.Println(value)
}
//fmt.Printf("name:%s ,id:is %d\n", name, id)
}
// DB操作 - end
defer db.Close()
}
实践
Demo
Java版 - IDEA版 - 也可以使用命令行
- 安装相关插件:Protobuf Support、Protobuf Highlight(编辑器高亮,选装),重启.
- 配置。pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.cc</groupId>
<artifactId>cc-test-protobuf</artifactId>
<version>1.0-SNAPSHOT</version>
<name>cc-test-protobuf</name>
<!-- FIXME change it to the project's website -->
<url>http://www.cc.com</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.4.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<extensions>
<extension>
<groupId>kr.motd.maven</groupId>
<artifactId>os-maven-plugin</artifactId>
<version>1.4.1.Final</version>
</extension>
</extensions>
<plugins>
<plugin>
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<version>0.5.0</version>
<configuration>
<protocArtifact>
com.google.protobuf:protoc:3.1.0:exe:${os.detected.classifier}
</protocArtifact>
<pluginId>grpc-java</pluginId>
</configuration>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>compile-custom</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
<pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
<plugins>
<!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>3.1.0</version>
</plugin>
<!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.0.2</version>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.1</version>
</plugin>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
</plugin>
<plugin>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.2</version>
</plugin>
<plugin>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.8.2</version>
</plugin>
<!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
<plugin>
<artifactId>maven-site-plugin</artifactId>
<version>3.7.1</version>
</plugin>
<plugin>
<artifactId>maven-project-info-reports-plugin</artifactId>
<version>3.0.0</version>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
- 模型。新建person.proto
syntax = "proto3"; // 版本号。与proto2差别不大
option java_package = "com.cc.model";
option java_outer_classname = "PersonModel";
message Person { // 类结构体
int32 id = 1; // 结构体属性
string name = 2; // 数字作为key,省空间。不一定要连续
string email = 3;
}
- 编译
注意:com.google.protobuf:protoc:3.1.0:exe
如果下载失败,则拷贝到本地Maven库;
操作系统种类变量:${os.detected.classifier}
:windows-x86_64等种类
并没编译成功,报错:\src\main\proto does not exist.
修正文件目录为proto
,再次编译。
这次编译OK.
,手动拷贝类到java目录 - 测试
test.java
@Test
public void testProtobuf() throws InvalidProtocolBufferException {
PersonModel.Person.Builder builder = PersonModel.Person.newBuilder();
builder.setId(1)
.setName("Shanks")
.setEmail("k999999@kof.com");
PersonModel.Person person = builder.build();
System.out.println("before:");
System.out.println(person);
System.out.println("===Person Byte:===");
for (byte b : person.toByteArray()) {
System.out.print(b);
}
System.out.println();
System.out.println("================");
byte[] byteArray = person.toByteArray();
PersonModel.Person p2 = PersonModel.Person.parseFrom(byteArray);
System.err.println("after id:" + p2.getId());
System.err.println("after name:" + p2.getName());
System.err.println("after email:" + p2.getEmail());
assertTrue(true);
}
output log
before:
id: 1
name: "Shanks"
email: "k999999@kof.com"
===Person Byte:===
8118683104971101071152615107575757575757641071111024699111109
================
after id:1
after name:Shanks
after email:k999999@kof.com
gRPC
golang版
基础包安装及编译错误解决
- 基础命令
go get -u github.com/golang/protobuf/proto # golang protobuf 库 ,上文已装过
go get -u github.com/golang/protobuf/protoc-gen-go # protoc --go_out 工具
go get google.golang.org/grpc # 安装grpc # 默认gopath路径:C:\Users\Administrator\go\src\github.com,不含\src\github.com
# 如果下不下来,请使用下述命令
cd $GOPATH/google.golang.org
git clone https://github.com/grpc/grpc-go # 选择go版本的grpc,也可选择码云极速下载https://gitee.com/mirrors/grpc,但是这个不是go版本的,注意甄别
go install google.golang.org/grpc-go
# 将grpc-go改名为grpc
git clone https://github.com/grpc/grpc # 这个不是go语言的版本,不要选
- golang.org/x/net 安装方法
mkdir -p $GOPATH/src/golang.org/x/
$cd $GOPATH/src/golang.org/x/
$git clone https://github.com/golang/net.git net
$go install net
- 安装之后,build,之前几个报错解决了,但是又有了新的报错:
..\..\golang.org\x\net\idna\idna10.0.0.go:25:2: cannot find package "golang.org/x/text/secure/bidirule" in any of:
解决办法:
cd golang.org
git clone https://github.com/golang/text # 很简单,因为找不到这个包,那就下载它
cd google.golang.org
git clone https://github.com/googleapis/go-genproto
mv go-genproto/ genproto/
编码
- 编写参数:helloword.proto
syntax = "proto3";
option objc_class_prefix = "HLW";
package proto;
// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {}
}
// The request message containing the user's name.
message HelloRequest {
string name = 1;
}
// The response message containing the greetings
message HelloReply {
string message = 1;
}
- 编译
protoc --go_out=plugins=grpc:. helloworld.proto
,生成*.pb.proto文件以备后用。 - 客户端:
package main
import (
"log"
"os"
pb "test/grpc-go-test/proto"
"golang.org/x/net/context"
"google.golang.org/grpc"
)
const (
address = ":8999"
)
func main() {
conn, err := grpc.Dial(address, grpc.WithInsecure())
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
c := pb.NewGreeterClient(conn) // 构造器
name := "Yuzhi"
if len(os.Args) > 1 {
name = os.Args[1]
}
r, err := c.SayHello(context.Background(), &pb.HelloRequest{Name: name})
if err != nil {
log.Fatalf("could not greet: %v", err)
}
log.Println(r.Message)
}
- 服务端
package main
import (
"log"
"net"
pb "test/grpc-go-test/proto"
"golang.org/x/net/context"
"google.golang.org/grpc"
)
const (
PORT = ":8999" // 本地地址
)
type server struct{}
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
log.Println("request: ", in.Name)
return &pb.HelloReply{Message: "Hello123 " + in.Name}, nil
}
func main() {
lis, err := net.Listen("tcp", PORT)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
pb.RegisterGreeterServer(s, &server{})
log.Println("gRPC服务已开启")
s.Serve(lis)
}
- 分别启动服务端、客户端,点击LiteIDE的BR按钮运行,调用成功。
// server
2019/11/26 15:08:16 rpc服务已经开启
2019/11/26 15:09:03 request: Yuzhi
// client
E:/gopath/gopath/src/test/grpc-client-test/grpc-client-test.exe [E:/gopath/gopath/src/test/grpc-client-test]
2019/11/26 15:09:03 Hello123 Yuzhi
调用区块链的gRPC
编码
proto(节选)
syntax = "proto3";
import "google/api/annotations.proto";
import "chainedbft.proto";
package proto;
// ...
rpc QueryACL(AclStatus) returns (AclStatus) {
option (google.api.http) = {
post : "/v1/query_acl"
body : "*"
};
}
// ...
// 查询Acl
message AclStatus {
Header header = 1; // 使用插件可以省略数字key
string bcname = 2;
string accountName = 3;
string contractName = 4;
string methodName = 5;
bool confirmed = 6;
Acl acl = 7;
}
// ...
gRPC客户端
首先你要有区块链节点,见区块链教程。
- 代码
package main
import (
"fmt"
"log"
pb "test/grpc-client-xchain/proto"
"golang.org/x/net/context"
"google.golang.org/grpc"
)
const (
address = "ip:37101" // 区块链节点
)
func main() {
conn, err := grpc.Dial(address, grpc.WithInsecure())
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
c := pb.NewXchainClient(conn) // 构造器
// 查询交易
bcname := "xuper"
// bcname := `xuper`
txid := "1130b45b1d1b953ddb1098878a1f87d6cdc4e8cb4ada9231fdeee241e85a451e"
// txid := `1130b45b1d1b953ddb1098878a1f87d6cdc4e8cb4ada9231fdeee241e85a451e`
// txid := `b29e4013948c5f24702e9c36c02548c4ae2162be8cf9cf0c38e18ff278bc0d59`
var txidBytes = []byte(txid)
fmt.Println(`byte数组`, txidBytes)
// r, err := c.PostTx(context.TODO(), &pb.TxStatus{Bcname: bcname, Txid: txidBytes})
r, err := c.QueryTx(context.Background(), &pb.TxStatus{Bcname: bcname, Txid: txidBytes})
if err != nil {
log.Fatalf("could not greet: %v", err)
}
log.Println(r)
// 查询余额
address := "XC1234560123456789@xuper"
// address := "Umu6Yk6mxiyZFXQhw8zo1r3KpM9nH9d8r"
r1, err1 := c.GetBalance(context.Background(), &pb.AddressStatus{Address: address})
if err1 != nil {
log.Fatalf("could not greet1: %v", err1)
}
log.Println(r1)
// 查询ACL
accountName := `XC1234560123456789@xuper`
r2, err2 := c.QueryACL(context.Background(), &pb.AclStatus{Bcname: bcname, AccountName: accountName})
if err2 != nil {
log.Fatalf("could not greet2: %v", err2)
}
log.Println(r2)
}
- 响应成功(节选)
2019/11/26 18:12:19 header:<> confirmed:true acl:<pm:<rule:SIGN_THRESHOLD acceptValue:1 > aksWeight:<key:"Umu6Yk6mxiyZFXQhw8zo1r3KpM9nH9d8r" value:1 > >
成功: 进程退出代码 0.