Protocol Buffer入门

Protocol Buffer
Protocol buffers are a flexible, efficient, automated mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. You can even update your data structure without breaking deployed programs that are compiled against the “old” format.

Google, Developer Guide, developers.google.com/docs/…

上面是Google官方文档对Protocol Buffer(简称PB)的定义，简单翻译下。

• 序列化结构数据的自动化机制，类似XML，但是更小，更快，更简单
• 你只需定义你的数据结构，就可以用PB生成的源码去操作结构数据
• 支持多种语言
• 数据结构向后兼容
  
  

性能

PB的性能，数据大小口碑一直不错，好多公司也都在用。这里有些人们做的Benchmark数据。相对与未压缩的xml或者json来说是有不小的优势的，官方文档的说法是比xml要小3-10倍，快20-100倍。

语法基础

在使用PB之前要先定义你的数据结构，PB有一套数据结构定义的语法，你需要使用这套语法来定义你的结构。定义结构的代码一般会写到一个以.proto为后缀的文件中（文件后缀好像没有强制）

语法

Message

message SearchRequest {
  required string query = 1;
  optional int32 page_number = 2;
  optional int32 result_per_page = 3;
}

Message类似于Java中的对象，C中的struct，是PB中数据封装的基本单元，一般我们在PB中定义的数据结构就是一个个的Message了。

上面的代码就定义了一个message

• message由field组成，类似对象中的成员变量
• 要定义field的类型，类型包括scalar（待会儿说）， enumeration，以及message（支持嵌套哦）
• 要定义field的规则，规则有required（值不能为空）,optional（值可以为空），repeated（重复，用来声明列表）
• 每个field要分配Tag，tag就是每行最后的那写数字，占位符的意思。数据编码后用tag来对应每个field。tag从1开始，而且是动态编码的，1-16占一个字节，16-2047两个字节，最大是229 - 1

Scalar

下图列出了PB所支持的Scalar的类型

enum

PB支持枚举类型，对应Java或者C中的枚举类型,下面是一个枚举的例子

enumColour {
    RED = 0;
    BLACK = 1;
    BLUE = 2;
  }

使用

使用之前需要先安装protoc，protoc用于生成对应语言的代码。除了protoc，还需要编译对应语言的库，并且添加到你的依赖路径中。安装wiki

定义数据结构

首先用pb的语法定义数据结构。

package me.ellios.tutorial;
option java_package = "me.ellios.tutorial";
option java_outer_classname = "AddressBookProtos";
message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;
  repeated PhoneNumber phone = 4;
  enumPhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }
  message PhoneNumber {
    required string number = 1;
    optional PhoneType type = 2 [default= HOME];
  }
}
message AddressBook {
  repeated Person person = 1;
}

• package：声明生成的代码所在的包，貌似是对应cpp的命名空间
• option java_package：针对java，声明java代码的包
• option java_outer_classname：针对java，声明java代码的类

java的例子

执行下面的命令，会生成AddressBookProtos.java文件，将它拷到你的工作目录中

protoc --java_out=. tutorial.proto

下面是一段简单的示例代码，演示怎么创建对象，以及对象的序列化和反序列化。

packageme.ellios.tutorial;
importme.ellios.tutorial.AddressBookProtos.Person;
importme.ellios.tutorial.AddressBookProtos.AddressBook;
importjava.io.*;
importjava.util.Random;
publicclass AddressBookExample {
    publicstatic void main(String[] args) throwsException {
        //要通过builder来构建对象，
        AddressBook.Builder addressBuiler = AddressBook.newBuilder();
        Random random = newRandom();
        for(inti=0; i<10; i++){
            Person.Builder personBuilder = Person.newBuilder();
            personBuilder.setName("ellios"+ i); //设置name，不能为空
            personBuilder.setId(i);//设置id，也不能为空
            personBuilder.setEmail("");//email可以为空
            //创建phone
            Person.PhoneNumber.Builder phoneBuilder = Person.PhoneNumber.newBuilder();
            phoneBuilder.setNumber("132222222222");
            phoneBuilder.setType(Person.PhoneType.values()[random.nextInt(3)]);
            //将phone添加到person中,              .
            personBuilder.addPhone(phoneBuilder.build());
            //将person添加到地址簿的列表中
            addressBuiler.addPerson(personBuilder);
        }
        AddressBook addressBook = addressBuiler.build();
        System.out.println("addressBook : "  + addressBook);
        byte[] serializeBytes = addressBook.toByteArray();
        AddressBook addressBook2 = AddressBook.parseFrom(serializeBytes);
        System.out.println("addressBook2 equal to addressBook. result is " + addressBook.equals(addressBook2));
        addressBook.writeTo(newFileOutputStream("addressBook"));
        AddressBook.Builder addressBuilder3 = AddressBook.newBuilder();
        addressBuilder3.mergeFrom(newFileInputStream("addressBook"));
        System.out.println("addressBook3 equal to addressBook. result is " + addressBook.equals(addressBuilder3.build()));
    }
}

python示例

执行下面的命令，生成tutorial_pb2.py文件,将它拷到工作目录中。

protoc --python_out=. tutorial.proto

下面是简单的示例代码

#! /usr/bin/python
importtutorial_pb2
address_book=tutorial_pb2.AddressBook()
fori inrange(10):
    person=address_book.person.add()
    person.id= i
    person.name='ellios%d'%i
    person.email=''
    phone_number=person.phone.add()
    phone_number.number="13233333333"
    ifi %3 ==0:
        phone_number.type= tutorial_pb2.Person.MOBILE
    elifi %3 ==1:
        phone_number.type= tutorial_pb2.Person.HOME
    elifi %3 ==2:
        phone_number.type= tutorial_pb2.Person.WORK
print'address_book : ', address_book
f=open("address_book","wb")
f.write(address_book.SerializeToString())
f.close()
address_book2=tutorial_pb2.AddressBook()
f=open("address_book","rb")
address_book2.ParseFromString(f.read())
f.close()
print"address_book same with address_book2. the result is %d"%(address_book==address_book2)

cpp示例

执行下面的命令，生成tutorial.pb.h和tutorial.pb.cc文件，将这两个文件拷到工作目录

protoc --cpp_out=. tutorial.proto

下面是一段示例代码

#include <iostream>
#include <fstream>
#include <string>
#include "tutorial.pb.h"
usingnamespace std;
intmain(intargc, char* argv[]) {
  // Verify that the version of the library that we linked against is
  // compatible with the version of the headers we compiled against.
  GOOGLE_PROTOBUF_VERIFY_VERSION;
  tutorial::AddressBook address_book;
  for(inti = 0; i < 10; i++) {
      tutorial::Person* person = address_book.add_person();
      person->set_id(i);
      person->set_name("ellios");
      person->set_email("");
      tutorial::Person::PhoneNumber* phone_number = person->add_phone();
      phone_number->set_number("13333333333");
      phone_number->set_type(tutorial::Person::HOME);
  }
  // Write the new address book back to disk.
  fstream output("address_book", ios::out | ios::trunc | ios::binary);
  if(!address_book.SerializeToOstream(&output)) {
      cerr << "Failed to write address book." << endl;
      return-1;
  }
  cout << "address_book : " << address_book.person_size() << endl;
  tutorial::AddressBook address_book2;
  fstream input("address_book", ios::in | ios::binary);
  address_book2.ParseFromIstream(&input);
  // Optional:  Delete all global objects allocated by libprotobuf.
  google::protobuf::ShutdownProtobufLibrary();
  return0;
}

参考资料

https://developers.google.com/protocol-buffers/docs/proto