CMDB系统一直使用json 格式作为API接口进行数据的收集和渲染。当数据量比较大时，使用http协议对json数据传输时，传输比较耗时。无意间发现google有一款开源软件protocol buffer，通过测试发现的确性能比json提高不少。
protobuf
Google Protocol Buffer( 简称 Protobuf) 是 Google 公司内部的混合语言数据标准，目前已经正在使用的有超过 48,162 种报文格式定义和超过 12,183 个 .proto 文件。他们用于 RPC 系统和持续数据存储系统。
Protocol Buffers 是一种轻便高效的结构化数据存储格式，可以用于结构化数据串行化，或者说序列化。它很适合做数据存储或 RPC 数据交换格式。可用于通讯协议、数据存储等领域的语言无关、平台无关、可扩展的序列化结构数据格式。目前提供了 C++、Java、Python 三种语言的 API。
protobuf 优点
> 跨平台、跨语言
> 向下兼容性好，易于扩展
> 自动生成解析代码
> 序列化后占用空间小
> 特别方便用于RPC和消息存储的场合
RPC通讯模型和ZeroRpc
RPC通讯模型：
一种通过网络从远程计算机程序服务，而不需要了解底层网络技术的协议。
RPC协议假定某些传输协议的存在，如TCP或UDP，为通信程序之间携带信息数据。在OSI网络通信模型中，RPC跨越了传输层和应用层。
ZeroRPC:
ZeroRPC 是基于zeromq、gevent和msgpack 开发的分布式RPC框架zerorpc-python。这个框架简单、易用。
protobuf 安装和配置
I. 下载源码包
wget http://protobuf.googlecode.com/files/protobuf-2.3.0.zip
II.安装软件包
unzip protobuf-2.3.0.zip
cd protobuf-2.3.0
./configure
make
make install
III. 编写hostinfo.proto 描述文件
package hostinfo;
message HostInfo {
required string factername = 1;
optional string hostname = 2;
optional string serverhardtype = 3;
optional string servertag = 4;
optional string servertype = 5;
optional string gatewayIP = 6;
optional string serverproduct = 7;
optional Ram rams = 8;
repeated cpuinfo cpus = 9;
message cpuinfo {
optional string cpuCoreNum = 1;
optional string cpuBit = 2;
optional string cpuClockSpeed = 3;
optional string cpupynum = 4;
optional string cpuName = 5;
optional string band = 6;
optional string model = 7;
}
message Ram {
repeated memory mem = 1;
}
message memory {
optional string Serial_Number = 1;
optional string Manufacturer = 2;
optional string Speed = 3;
optional string Size = 4;
}
}
VI. 用protoc生成访问代码
protoc hostinfo.proto --python_out=./
生成访问代码如下(hostinfo_pb2.py):
# Generated by the protocol buffer compiler. DO NOT EDIT!
from google.protobuf import descriptor
from google.protobuf import message
from google.protobuf import reflection
from google.protobuf import descriptor_pb2
# @@protoc_insertion_point(imports)
DESCRIPTOR = descriptor.FileDescriptor(
name='hostinfo.proto',
package='info',
serialized_pb='\n\x0ehostinfo.proto\x12\x04info\"\xd3\x03\n\x08HostInfo\x12\x10\n\x08hostname\x18\x02 \x01(\t\x12\x16\n\x0eserverhardtype\x18\x03 \x01(\t\x12\x11\n\tservertag\x18\x04 \x01(\t\x12\x12\n\nservertype\x18\x05 \x01(\t\x12\x11\n\tgatewayIP\x18\x06 \x01(\t\x12\x15\n\rserverproduct\x18\x07 \x01(\t\x12 \n\x04rams\x18\x08 \x01(\x0b\x32\x12.info.HostInfo.Ram\x12$\n\x04\x63pus\x18\x01 \x03(\x0b\x32\x16.info.HostInfo.cpuinfo\x1a\x84\x01\n\x07\x63puinfo\x12\x12\n\ncpuCoreNum\x18\x01 \x01(\t\x12\x0e\n\x06\x63puBit\x18\x02 \x01(\t\x12\x15\n\rcpuClockSpeed\x18\x03 \x01(\t\x12\x10\n\x08\x63pupynum\x18\x04 \x01(\t\x12\x0f\n\x07\x63puName\x18\x05 \x01(\t\x12\x0c\n\x04\x62\x61nd\x18\x06 \x01(\t\x12\r\n\x05model\x18\x07 \x01(\t\x1a)\n\x03Ram\x12\"\n\x03mem\x18\x01 \x03(\x0b\x32\x15.info.HostInfo.memory\x1aR\n\x06memory\x12\x15\n\rSerial_Number\x18\x01 \x01(\t\x12\x14\n\x0cManufacturer\x18\x02 \x01(\t\x12\r\n\x05Speed\x18\x03 \x01(\t\x12\x0c\n\x04Size\x18\x04 \x01(\t')
_HOSTINFO_CPUINFO = descriptor.Descriptor(
name='cpuinfo',
full_name='info.HostInfo.cpuinfo',
filename=None,
file=DESCRIPTOR,
containing_type=None,
fields=[
descriptor.FieldDescriptor(
name='cpuCoreNum', full_name='info.HostInfo.cpuinfo.cpuCoreNum', index=0,
number=1, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
descriptor.FieldDescriptor(
name='cpuBit', full_name='info.HostInfo.cpuinfo.cpuBit', index=1,
number=2, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
descriptor.FieldDescriptor(
name='cpuClockSpeed', full_name='info.HostInfo.cpuinfo.cpuClockSpeed', index=2,
number=3, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
descriptor.FieldDescriptor(
name='cpupynum', full_name='info.HostInfo.cpuinfo.cpupynum', index=3,
number=4, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
descriptor.FieldDescriptor(
name='cpuName', full_name='info.HostInfo.cpuinfo.cpuName', index=4,
number=5, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
descriptor.FieldDescriptor(
name='band', full_name='info.HostInfo.cpuinfo.band', index=5,
number=6, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
descriptor.FieldDescriptor(
name='model', full_name='info.HostInfo.cpuinfo.model', index=6,
number=7, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
],
extensions=[
],
nested_types=[],
enum_types=[
],
options=None,
is_extendable=False,
extension_ranges=[],
serialized_start=233,
serialized_end=365,
)
_HOSTINFO_RAM = descriptor.Descriptor(
name='Ram',
full_name='info.HostInfo.Ram',
filename=None,
file=DESCRIPTOR,
containing_type=None,
fields=[
descriptor.FieldDescriptor(
name='mem', full_name='info.HostInfo.Ram.mem', index=0,
number=1, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
],
extensions=[
],
nested_types=[],
enum_types=[
],
options=None,
is_extendable=False,
extension_ranges=[],
serialized_start=367,
serialized_end=408,
)
_HOSTINFO_MEMORY = descriptor.Descriptor(
name='memory',
full_name='info.HostInfo.memory',
filename=None,
file=DESCRIPTOR,
containing_type=None,
fields=[
descriptor.FieldDescriptor(
name='Serial_Number', full_name='info.HostInfo.memory.Serial_Number', index=0,
number=1, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
descriptor.FieldDescriptor(
name='Manufacturer', full_name='info.HostInfo.memory.Manufacturer', index=1,
number=2, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
descriptor.FieldDescriptor(
name='Speed', full_name='info.HostInfo.memory.Speed', index=2,
number=3, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
descriptor.FieldDescriptor(
name='Size', full_name='info.HostInfo.memory.Size', index=3,
number=4, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
],
extensions=[
],
nested_types=[],
enum_types=[
],
options=None,
is_extendable=False,
extension_ranges=[],
serialized_start=410,
serialized_end=492,
)
_HOSTINFO = descriptor.Descriptor(
name='HostInfo',
full_name='info.HostInfo',
filename=None,
file=DESCRIPTOR,
containing_type=None,
fields=[
descriptor.FieldDescriptor(
name='hostname', full_name='info.HostInfo.hostname', index=0,
number=2, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
descriptor.FieldDescriptor(
name='serverhardtype', full_name='info.HostInfo.serverhardtype', index=1,
number=3, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
descriptor.FieldDescriptor(
name='servertag', full_name='info.HostInfo.servertag', index=2,
number=4, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
descriptor.FieldDescriptor(
name='servertype', full_name='info.HostInfo.servertype', index=3,
number=5, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
descriptor.FieldDescriptor(
name='gatewayIP', full_name='info.HostInfo.gatewayIP', index=4,
number=6, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
descriptor.FieldDescriptor(
name='serverproduct', full_name='info.HostInfo.serverproduct', index=5,
number=7, type=9, cpp_type=9, label=1,
has_default_value=False, default_value=unicode("", "utf-8"),
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
descriptor.FieldDescriptor(
name='rams', full_name='info.HostInfo.rams', index=6,
number=8, type=11, cpp_type=10, label=1,
has_default_value=False, default_value=None,
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
descriptor.FieldDescriptor(
name='cpus', full_name='info.HostInfo.cpus', index=7,
number=1, type=11, cpp_type=10, label=3,
has_default_value=False, default_value=[],
message_type=None, enum_type=None, containing_type=None,
is_extension=False, extension_scope=None,
options=None),
],
extensions=[
],
nested_types=[_HOSTINFO_CPUINFO, _HOSTINFO_RAM, _HOSTINFO_MEMORY, ],
enum_types=[
],
options=None,
is_extendable=False,
extension_ranges=[],
serialized_start=25,
serialized_end=492,
)
_HOSTINFO_CPUINFO.containing_type = _HOSTINFO;
_HOSTINFO_RAM.fields_by_name['mem'].message_type = _HOSTINFO_MEMORY
_HOSTINFO_RAM.containing_type = _HOSTINFO;
_HOSTINFO_MEMORY.containing_type = _HOSTINFO;
_HOSTINFO.fields_by_name['rams'].message_type = _HOSTINFO_RAM
_HOSTINFO.fields_by_name['cpus'].message_type = _HOSTINFO_CPUINFO
DESCRIPTOR.message_types_by_name['HostInfo'] = _HOSTINFO
class HostInfo(message.Message):
__metaclass__ = reflection.GeneratedProtocolMessageType
class cpuinfo(message.Message):
__metaclass__ = reflection.GeneratedProtocolMessageType
DESCRIPTOR = _HOSTINFO_CPUINFO
# @@protoc_insertion_point(class_scope:info.HostInfo.cpuinfo)
class Ram(message.Message):
__metaclass__ = reflection.GeneratedProtocolMessageType
DESCRIPTOR = _HOSTINFO_RAM
# @@protoc_insertion_point(class_scope:info.HostInfo.Ram)
class memory(message.Message):
__metaclass__ = reflection.GeneratedProtocolMessageType
DESCRIPTOR = _HOSTINFO_MEMORY
# @@protoc_insertion_point(class_scope:info.HostInfo.memory)
DESCRIPTOR = _HOSTINFO
# @@protoc_insertion_point(class_scope:info.HostInfo)
# @@protoc_insertion_point(module_scope)
V.编写应用代码(RPC client端)
#!/usr/bin/env python
import hostinfo_pb2
import zerorpc
import sys
hostinfo = hostinfo_pb2.HostInfo()
def ServerInfo(hostinfo):
'''这些信息可以通过脚本进行收集'''
hostinfo.hostname = 'cdn.oss.david.com'
hostinfo.serverhardtype = 'IBM System x3550 M2 -[7946I1M]-'
hostinfo.servertag = '99G0665'
hostinfo.servertype = 'physical'
hostinfo.gatewayIP = '113.207.61.1'
hostinfo.serverproduct = 'IBM'
cpu = hostinfo.cpus.add()
cpu.cpuCoreNum = '1'
cpu.cpuBit = '64'
cpu.cpuClockSpeed = '2.00GHz'
cpu.cpupynum = '1'
cpu.cpuName = 'Intel Xeon'
cpu.band = 'Intel'
cpu.model = 'E5504'
memory = hostinfo.rams.mem.add()
memory.Serial_Number = '5F787C83'
memory.Manufacturer = 'Samsung'
memory.Speed = '800 MHz'
memory.Size = '4096 MB'
if __name__ == "__main__":
ServerInfo(hostinfo)
c = zerorpc.Client()
c.connect("tcp://127.0.0.1:4242") #连接RPC 服务端
print c.info(hostinfo.SerializeToString()) #向服务端发送数据
VI. protobuf 基础
字段约束：
required：必须赋值字段，禁止为空
optional：可选字段，可以为空
repeated：集合，可以填充零到多个对象5.Zero RPC安装和配置
I.Zero RPC 安装
yum -y install zeromq
yum install gcc gcc-c++ libuuid-devel python-uuid uuid
wget http://download.zeromq.org/zeromq-2.1.9.tar.gz
./configure
make
make install
II. 安装gevent
pip install gevent
III. 安装zerorpc
pip install zerorpc
VI. 编写ZeroRPC Server
PRC 服务端：
#!/usr/bin/env python
import zerorpc
import hostinfo_pb2
class HostRpc(object):
def info(self, name):
data = hostinfo_pb2.HostInfo()
data.ParseFromString(name)
print data.hostname
print data.serverhardtype
print data.servertag
print data.gatewayIP
print data.serverproduct
return 'success'
s = zerorpc.Server(HostRpc())
s.bind("tcp://0.0.0.0:4242")
s.run()启动RPC Server：
查看RPC 状态:
客户端向服务端传输数据：
客户端：
服务端：
通过以上服务端的输出信息可以看出，客户端可以成功将信息发送至服务端。这只是为测试效果，可以将传输过来的数据通过Mysqldb写入到数据库。
转载于:https://blog.51cto.com/davidbj/1417371
扫一扫
397
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
