monster.fbs
FlatBuffers 使用.
来指定嵌套的名称空间/包。
root_type
声明序列化数据的根表(或结构)。
除了基本类型外,Monster
中包含一个表和一个结构体。
表是在 FlatBuffers 中定义对象的主要方式,由名称(此处为Monster
)和字段列表组成。每个字段都有一个名称、类型和可选的默认值(如果省略,则默认为0
或NULL
)。
表中每个字段都是可选的:它不必出现在线路表示中,您可以选择忽略每个对象的字段。因此,您可以灵活地添加字段,而不必担心数据膨胀。此设计也是 FlatBuffer 的向前和向后兼容机制。请注意:
- 在 schema 文件中只能在表定义的末尾添加新字段。旧数据仍能正确读取,并在读取时为您提供默认值。旧代码只会忽略新字段。如果想要灵活地对 schema 中的字段使用任何顺序,则可以手动分配 id(与 Protocol Buffers 类似),请参见下面的 id 属性。
- 您无法从 schema 中删除不再使用的字段,但是您可以停止将它们写入数据中以达到几乎相同的效果。另外,您可以像上面的示例中那样将它们标记为已弃用,这将防止在生成的 C++中生成访问器,从而强制不再使用该字段。(小心:这可能会破坏代码!)。
- 您可以更改字段名和表名,如果您不介意同步更改代码重命名。
Struct 没有一个字段是可选的(因此也没有默认值),并且不能添加或弃用字段。只能包含标量或其他结构。对于简单的对象,如果您确定不会进行任何更改(如示例中Vec3
非常明确)。结构使用的内存比表少,访问速度更快(它们始终以内联方式存储在其父对象中,并且不使用虚拟表)。
union
与枚举的许多属性相同,但是使用表名而不是常量的名称。声明的一个联合字段中可以包含对任何这些类型的引用,并且还会生成带有后缀_type
的隐藏字段,其中包含相应的枚举值,使您能够在运行时知道要转换为哪种类型。
联合体是一种能够将多种消息类型作为 FlatBuffer 发送的好方法。请注意,由于联合字段实际上是两个字段,因此它必须始终是表的一部分,本身不能成为 FlatBuffer 的根。
下面代码生成的结构体在MyGame::Sample
命名空间中。
namespace MyGame.Sample;
enum Color:byte { Red = 0, Green, Blue = 2 }
union Equipment { Weapon } // Optionally add more tables.
struct Vec3 {
x:float;
y:float;
z:float;
}
table Monster {
pos:Vec3;
mana:short = 150;
hp:short = 100;
name:string;
friendly:bool = false (deprecated);
inventory:[ubyte];
color:Color = Blue;
weapons:[Weapon];
equipped:Equipment;
path:[Vec3];
}
table Weapon {
name:string;
damage:short;
}
root_type Monster;
sample_binary.cpp
FlatBuffers 中有内部定义的 Vector 和 String。标量可以直接保存,其他成员都需要独立构建后再加入所隶属的结构中。
FlatBufferBuilder 是用于容纳创建 FlatBuffer 所需数据的帮助程序类。
FlatBufferBuilder::FlatBufferBuilder
要序列化数据,通常调用生成代码中的一个Create*()
函数,该函数依次调用StartTable/PushElement/AddElement/EndTable
序列或内置的 CreateString/CreateVector 函数。这样做是以深度优先顺序在根部建立一棵树。Finish() 将缓冲区打包以备传输。
FlatBufferBuilder::CreateString 在缓冲区中存储一个字符串,该字符串可以包含任何二进制数据。返回字符串开始处的缓冲区中的偏移量。
// Build up a serialized buffer algorithmically:
flatbuffers::FlatBufferBuilder builder;
// First, lets serialize some weapons for the Monster: A 'sword' and an 'axe'.
auto weapon_one_name = builder.CreateString("Sword");
short weapon_one_damage = 3;
auto weapon_two_name = builder.CreateString("Axe");
short weapon_two_damage = 5;
CreateWeapon 构建出 Weapon 对象,返回对象的偏移。weapons_vector
为偏移构成的数组。
FlatBufferBuilder::CreateVector 将偏移数组存储起来。
// Use the `CreateWeapon` shortcut to create Weapons with all fields set.
auto sword = CreateWeapon(builder, weapon_one_name, weapon_one_damage);
auto axe = CreateWeapon(builder, weapon_two_name, weapon_two_damage);
// Create a FlatBuffer's `vector` from the `std::vector`.
std::vector<flatbuffers::Offset<Weapon>> weapons_vector;
weapons_vector.push_back(sword);
weapons_vector.push_back(axe);
auto weapons = builder.CreateVector(weapons_vector);
接受数组的 FlatBufferBuilder::CreateVector 将数组值存储起来并返回数组偏移inventory
。
FlatBufferBuilder::Finish() 通过写入根偏移量完成缓冲区的序列化。
// Second, serialize the rest of the objects needed by the Monster.
auto position = Vec3(1.0f, 2.0f, 3.0f);
auto name = builder.CreateString("MyMonster");
unsigned char inv_data[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };
auto inventory = builder.CreateVector(inv_data, 10);
CreateMonster 通过 MonsterBuilder 来构造出 Monster 对象。
// Shortcut for creating monster with all fields set:
auto orc = CreateMonster(builder, &position, 150, 80, name, inventory,
Color_Red, weapons, Equipment_Weapon, axe.Union());
builder.Finish(orc); // Serialize the root of the object.
解析缓冲区。
FlatBufferBuilder::GetBufferPointer 获取序列化的缓冲区(在调用 Finish() 之后)。
GetMonster 调用 GetRoot 获得 Monster 对象的指针。
// We now have a FlatBuffer we can store on disk or send over a network.
// ** file/network code goes here :) **
// access builder.GetBufferPointer() for builder.GetSize() bytes
// Instead, we're going to access it right away (as if we just received it).
// Get access to the root:
auto monster = GetMonster(builder.GetBufferPointer());
Monster::hp 和 Monster::mana 通过 Table::GetField 获得字段值。
Monster::name 通过 Table::GetPointer 获得 String 对象的指针。
Monster::pos 调用 Table::GetStruct 得到 Vec3 结构体的指针。
// Get and test some scalar types from the FlatBuffer.
assert(monster->hp() == 80);
assert(monster->mana() == 150); // default
assert(monster->name()->str() == "MyMonster");
// Get and test a field of the FlatBuffer's `struct`.
auto pos = monster->pos();
assert(pos);
assert(pos->z() == 3.0f);
(void)pos;
Monster::inventory 通过 Table::GetPointer 获取 Vector 对象的指针。
Vector::Get 获取指定索引的元素。
Monster::weapons 通过 Table::GetPointer 获取 Vector 对象的指针weps
。后者记录了每个 Weapon 对象的偏移,从而能够调用 Weapon::name 和 Weapon::damage。
// Get a test an element from the `inventory` FlatBuffer's `vector`.
auto inv = monster->inventory();
assert(inv);
assert(inv->Get(9) == 9);
(void)inv;
// Get and test the `weapons` FlatBuffers's `vector`.
std::string expected_weapon_names[] = { "Sword", "Axe" };
short expected_weapon_damages[] = { 3, 5 };
auto weps = monster->weapons();
for (unsigned int i = 0; i < weps->size(); i++) {
assert(weps->Get(i)->name()->str() == expected_weapon_names[i]);
assert(weps->Get(i)->damage() == expected_weapon_damages[i]);
}
(void)expected_weapon_names;
(void)expected_weapon_damages;
Monster::equipped_type 通过 Table::GetField 获得 Equipment 枚举值。
// Get and test the `Equipment` union (`equipped` field).
assert(monster->equipped_type() == Equipment_Weapon);
auto equipped = static_cast<const Weapon *>(monster->equipped());
assert(equipped->name()->str() == "Axe");
assert(equipped->damage() == 5);
(void)equipped;
printf("The FlatBuffer was successfully created and verified!\n");
FlatBufferBuilder::CreateString
FlatBufferBuilder::NotNested 检查构建过程中是否存在嵌套。
FlatBufferBuilder::PreAlign 向buf_
填0,预先进行对齐。
FlatBufferBuilder::PushBytes 将数据填入buf_
中。
FlatBufferBuilder::PushElement 把长度值追加进去。
FlatBufferBuilder::GetSize 返回当前大小。
/// @brief Store a string in the buffer, which can contain any binary data.
/// @param[in] str A const char pointer to the data to be stored as a string.
/// @param[in] len The number of bytes that should be stored from `str`.
/// @return Returns the offset in the buffer where the string starts.
Offset<String> CreateString(const char *str, size_t len) {
NotNested();
PreAlign<uoffset_t>(len + 1); // Always 0-terminated.
buf_.fill(1);
PushBytes(reinterpret_cast<const uint8_t *>(str), len);
PushElement(static_cast<uoffset_t>(len));
return Offset<String>(GetSize());
}
CreateWeapon
创建一个 WeaponBuilder,由后者进行构造。
WeaponBuilder::add_name 记录name
的偏移。
WeaponBuilder::add_damage 保存damage
的值。
WeaponBuilder::Finish 生成 vtable。
inline flatbuffers::Offset<Weapon> CreateWeapon(
flatbuffers::FlatBufferBuilder &_fbb,
flatbuffers::Offset<flatbuffers::String> name = 0,
int16_t damage = 0) {
WeaponBuilder builder_(_fbb);
builder_.add_name(name);
builder_.add_damage(damage);
return builder_.Finish();
}
WeaponBuilder
FlatBufferBuilder::AddOffset 插入偏移量。
Offset 用于 uoffset_t 的包装器,可进行安全的模板特化。
FlatBufferBuilder::EndTable 完成了一个序列化对象,方法是生成 vtable(如果是表),将其与现有vtable进行比较,然后写入生成的 vtable 偏移量。
struct WeaponBuilder {
typedef Weapon Table;
flatbuffers::FlatBufferBuilder &fbb_;
flatbuffers::uoffset_t start_;
void add_name(flatbuffers::Offset<flatbuffers::String> name) {
fbb_.AddOffset(Weapon::VT_NAME, name);
}
void add_damage(int16_t damage) {
fbb_.AddElement<int16_t>(Weapon::VT_DAMAGE, damage, 0);
}
explicit WeaponBuilder(flatbuffers::FlatBufferBuilder &_fbb)
: fbb_(_fbb) {
start_ = fbb_.StartTable();
}
flatbuffers::Offset<Weapon> Finish() {
const auto end = fbb_.EndTable(start_);
auto o = flatbuffers::Offset<Weapon>(end);
return o;
}
};
Weapon
WeaponT 为表的对象结构体(T 结尾符),该结构体可以直接进行数据操作。MNN 中采用这种结构体和 Pack 函数完成序列化。
struct Weapon FLATBUFFERS_FINAL_CLASS : private flatbuffers::Table {
typedef WeaponT NativeTableType;
typedef WeaponBuilder Builder;
static const flatbuffers::TypeTable *MiniReflectTypeTable() {
return WeaponTypeTable();
}
enum FlatBuffersVTableOffset FLATBUFFERS_VTABLE_UNDERLYING_TYPE {
VT_NAME = 4,
VT_DAMAGE = 6
};
const flatbuffers::String *name() const {
return GetPointer<const flatbuffers::String *>(VT_NAME);
}
flatbuffers::String *mutable_name() {
return GetPointer<flatbuffers::String *>(VT_NAME);
}
int16_t damage() const {
return GetField<int16_t>(VT_DAMAGE, 0);
}
bool mutate_damage(int16_t _damage) {
return SetField<int16_t>(VT_DAMAGE, _damage, 0);
}
bool Verify(flatbuffers::Verifier &verifier) const {
return VerifyTableStart(verifier) &&
VerifyOffset(verifier, VT_NAME) &&
verifier.VerifyString(name()) &&
VerifyField<int16_t>(verifier, VT_DAMAGE) &&
verifier.EndTable();
}
WeaponT *UnPack(const flatbuffers::resolver_function_t *_resolver = nullptr) const;
void UnPackTo(WeaponT *_o, const flatbuffers::resolver_function_t *_resolver = nullptr) const;
static flatbuffers::Offset<Weapon> Pack(flatbuffers::FlatBufferBuilder &_fbb, const WeaponT* _o, const flatbuffers::rehasher_function_t *_rehasher = nullptr);
};
Offset
用于 voffset_t 的包装器,可进行安全的模板特化。
// Wrapper for uoffset_t to allow safe template specialization.
// Value is allowed to be 0 to indicate a null object (see e.g. AddOffset).
template<typename T> struct Offset {
uoffset_t o;
Offset() : o(0) {}
Offset(uoffset_t _o) : o(_o) {}
Offset<void> Union() const { return Offset<void>(o); }
bool IsNull() const { return !o; }
};
FlatBufferBuilder::AddOffset
FlatBufferBuilder::AddElement
FlatBufferBuilder::ReferTo
voffset_t
template<typename T> void AddOffset(voffset_t field, Offset<T> off) {
if (off.IsNull()) return; // Don't store.
AddElement(field, ReferTo(off.o), static_cast<uoffset_t>(0));
}
FlatBufferBuilder::AddElement
插入字段和值。
IsTheSameAs 判断两个标量值是否相等。
FlatBufferBuilder::PushElement 将单个对齐的标量写入缓冲区并返回缓冲区大小。
FlatBufferBuilder::TrackField 将 FieldLoc 结构体插入到buf_
中。
// Like PushElement, but additionally tracks the field this represents.
template<typename T> void AddElement(voffset_t field, T e, T def) {
// We don't serialize values equal to the default.
if (IsTheSameAs(e, def) && !force_defaults_) return;
auto off = PushElement(e);
TrackField(field, off);
}
FlatBufferBuilder::TrackField
构造一个 FieldLoc 结构体。
vector_downward::scratch_push_small 将数据插入向量的最低地址中。
// When writing fields, we track where they are, so we can create correct
// vtables later.
void TrackField(voffset_t field, uoffset_t off) {
FieldLoc fl = { off, field };
buf_.scratch_push_small(fl);
num_field_loc++;
max_voffset_ = (std::max)(max_voffset_, field);
}
FlatBufferBuilder::CreateVector
调用使用数组指针的版本。
/// @brief Serialize a `std::vector` into a FlatBuffer `vector`.
/// @tparam T The data type of the `std::vector` elements.
/// @param v A const reference to the `std::vector` to serialize into the
/// buffer as a `vector`.
/// @return Returns a typed `Offset` into the serialized data indicating
/// where the vector is stored.
template<typename T> Offset<Vector<T>> CreateVector(const std::vector<T> &v) {
return CreateVector(data(v), v.size());
}
FlatBufferBuilder::ReferTo
// Offsets initially are relative to the end of the buffer (downwards).
// This function converts them to be relative to the current location
// in the buffer (when stored here), pointing upwards.
uoffset_t ReferTo(uoffset_t off) {
// Align to ensure GetSize() below is correct.
Align(sizeof(uoffset_t));
// Offset must refer to something already in buffer.
FLATBUFFERS_ASSERT(off && off <= GetSize());
return GetSize() - off + static_cast<uoffset_t>(sizeof(uoffset_t));
}
FlatBufferBuilder::CreateVector
FlatBufferBuilder::EndVector 插入数组长度并返回偏移。
template<typename T>
Offset<Vector<Offset<T>>> CreateVector(const Offset<T> *v, size_t len) {
StartVector(len, sizeof(Offset<T>));
for (auto i = len; i > 0;) { PushElement(v[--i]); }
return Offset<Vector<Offset<T>>>(EndVector(len));
}
FlatBufferBuilder::StartVector
void StartVector(size_t len, size_t elemsize) {
NotNested();
nested = true;
PreAlign<uoffset_t>(len * elemsize);
PreAlign(len * elemsize, elemsize); // Just in case elemsize > uoffset_t.
}
FlatBufferBuilder::PushElement
访问其中的偏移值。
template<typename T> uoffset_t PushElement(Offset<T> off) {
// Special case for offsets: see ReferTo below.
return PushElement(ReferTo(off.o));
}
FlatBufferBuilder::PushElement
AssertScalarT 检查T
类型是否为标量。
Align 对buf_
进行填充。
EndianScalar 得到小端值。
FlatBufferBuilder::GetSize 返回buf_
的大小。
// Write a single aligned scalar to the buffer
template<typename T> uoffset_t PushElement(T element) {
AssertScalarT<T>();
T litle_endian_element = EndianScalar(element);
Align(sizeof(T));
buf_.push_small(litle_endian_element);
return GetSize();
}
FlatBufferBuilder::CreateVector
data 对于空向量返回非 null 以避免未定义的行为。因为最终返回的指针被传递给 memcpy,所以需要它。
/// @brief Serialize a `std::vector` into a FlatBuffer `vector`.
/// @tparam T The data type of the `std::vector` elements.
/// @param v A const reference to the `std::vector` to serialize into the
/// buffer as a `vector`.
/// @return Returns a typed `Offset` into the serialized data indicating
/// where the vector is stored.
template<typename T> Offset<Vector<T>> CreateVector(const std::vector<T> &v) {
return CreateVector(data(v), v.size());
}
FlatBufferBuilder::CreateVector
FlatBufferBuilder::StartVector
以倒序方式插入
template<typename T>
Offset<Vector<Offset<T>>> CreateVector(const Offset<T> *v, size_t len) {
StartVector(len, sizeof(Offset<T>));
for (auto i = len; i > 0;) { PushElement(v[--i]); }
return Offset<Vector<Offset<T>>>(EndVector(len));
}
CreateMonster
inline flatbuffers::Offset<Monster> CreateMonster(
flatbuffers::FlatBufferBuilder &_fbb,
const MyGame::Sample::Vec3 *pos = 0,
int16_t mana = 150,
int16_t hp = 100,
flatbuffers::Offset<flatbuffers::String> name = 0,
flatbuffers::Offset<flatbuffers::Vector<uint8_t>> inventory = 0,
MyGame::Sample::Color color = MyGame::Sample::Color_Blue,
flatbuffers::Offset<flatbuffers::Vector<flatbuffers::Offset<MyGame::Sample::Weapon>>> weapons = 0,
MyGame::Sample::Equipment equipped_type = MyGame::Sample::Equipment_NONE,
flatbuffers::Offset<void> equipped = 0,
flatbuffers::Offset<flatbuffers::Vector<const MyGame::Sample::Vec3 *>> path = 0) {
MonsterBuilder builder_(_fbb);
builder_.add_path(path);
builder_.add_equipped(equipped);
builder_.add_weapons(weapons);
builder_.add_inventory(inventory);
builder_.add_name(name);
builder_.add_pos(pos);
builder_.add_hp(hp);
builder_.add_mana(mana);
builder_.add_equipped_type(equipped_type);
builder_.add_color(color);
return builder_.Finish();
}
MonsterBuilder
struct MonsterBuilder {
typedef Monster Table;
flatbuffers::FlatBufferBuilder &fbb_;
flatbuffers::uoffset_t start_;
void add_pos(const MyGame::Sample::Vec3 *pos) {
fbb_.AddStruct(Monster::VT_POS, pos);
}
void add_mana(int16_t mana) {
fbb_.AddElement<int16_t>(Monster::VT_MANA, mana, 150);
}
void add_hp(int16_t hp) {
fbb_.AddElement<int16_t>(Monster::VT_HP, hp, 100);
}
void add_name(flatbuffers::Offset<flatbuffers::String> name) {
fbb_.AddOffset(Monster::VT_NAME, name);
}
void add_inventory(flatbuffers::Offset<flatbuffers::Vector<uint8_t>> inventory) {
fbb_.AddOffset(Monster::VT_INVENTORY, inventory);
}
void add_color(MyGame::Sample::Color color) {
fbb_.AddElement<int8_t>(Monster::VT_COLOR, static_cast<int8_t>(color), 2);
}
void add_weapons(flatbuffers::Offset<flatbuffers::Vector<flatbuffers::Offset<MyGame::Sample::Weapon>>> weapons) {
fbb_.AddOffset(Monster::VT_WEAPONS, weapons);
}
void add_equipped_type(MyGame::Sample::Equipment equipped_type) {
fbb_.AddElement<uint8_t>(Monster::VT_EQUIPPED_TYPE, static_cast<uint8_t>(equipped_type), 0);
}
void add_equipped(flatbuffers::Offset<void> equipped) {
fbb_.AddOffset(Monster::VT_EQUIPPED, equipped);
}
void add_path(flatbuffers::Offset<flatbuffers::Vector<const MyGame::Sample::Vec3 *>> path) {
fbb_.AddOffset(Monster::VT_PATH, path);
}
explicit MonsterBuilder(flatbuffers::FlatBufferBuilder &_fbb)
: fbb_(_fbb) {
start_ = fbb_.StartTable();
}
flatbuffers::Offset<Monster> Finish() {
const auto end = fbb_.EndTable(start_);
auto o = flatbuffers::Offset<Monster>(end);
return o;
}
};
参考资料:
- Tutorial
- Difference between char and signed char in c++?
- Can I reinterpret std::vector as a std::vector without copying?
- FlatBuffers介绍及初步试用
- Writing a schema
- android FlatBuffers剖析
- FlatBuffers学习总结
- Flatbuffers使用解析
- FlatBuffer Internals
- FlatBuffers white paper
- how to feed data from binary string ubyte vector of flatbuffer in python?
- 深入浅出 FlatBuffers 之 Encode
- 数据序列化组件PB与FB对比
- FlatBuffer代码阅读 - 1
- Applied SOAP: Implementing .NET XML Web Services by Kenn Scribner, Mark Stiver
- Git 协议版本 2 宣布推出:Git wire protocol 的一次重大更新
- 什么是wire protocol?
- What is wire format
- Class WireFormat
- Wire protocol
- XML wire format
- flatbuffer的数组小例子(C++)
- General rules of passing/returning reference of array (not pointer) to/from a function?
- Return array in a function
- flatbuffer编码的内存的结构
- FlatBuffers反射
- caffe模型文件解析_MNN模型转化的主流程
- Why cast an unused function parameter value to void?