java flatbuffers_使用flatbuffers

问题

张三是个java程序员,他写产生数据的程序.李四是个python程序员,他要用python处理张三产生的数据.最直观常用的方法就是张三用java把产生的数据保存成csv或者xml文件,然后李四用python读取csv或xml文件.这没有问题.但现在有一种性能更好的方法,flatbuffers.

作用

可以把flatbuffers理解成一个可执行文件flatc.这个可执行文件可以把表示数据格式的fbs文件转化成c++,java,python...代码.张三擅长使用java,他就用flatc把fbs文件转化成java代码.用转化好的java代码把数据存成flatbuffers的格式.李四用flatc把同样的fbs文件转化成自己熟悉的python代码,李四可以用这份python代码把flatbuffers格式文件中的数据读出来.这就实现了不同语言间传数据.当然这只是一个简单的使用场景.flatbuffers的作用远不只是这些.

使用例子

先写一个表示数据格式的文件school.fbs

namespace school;

table School {

name:string;

students:[Student];

}

table Student {

number:short;

name:string;

}

root_type School;

具体的语法规则看参考资料,只简单的解释下.上面代码定义了一个根结点为School的数据.里面的student有2个成员变量number,name. School也有2个成员变量,一个是name代码school的名字.一个是Student的数组.

然后执行下面命令:

flatc.exe -p school.fbs

会生成一堆python代码.再用生成的python创建一个flatbuffers文件,代码如下：

# coding:utf-8

from __future__ import print_function

from __future__ import division

import flatbuffers

from school import School

from school import Student

builder = flatbuffers.Builder(1024)

stu1 = builder.CreateString('zhang3')

stu2 = builder.CreateString('li4')

Student.StudentStart(builder)

Student.StudentAddName(builder, stu1)

Student.StudentAddNumber(builder, 3)

zhang3 = Student.StudentEnd(builder)

Student.StudentStart(builder)

Student.StudentAddName(builder, stu2)

Student.StudentAddNumber(builder, 4)

li4 = Student.StudentEnd(builder)

School.SchoolStartStudentsVector(builder, 2)

builder.PrependUOffsetTRelative(li4)

builder.PrependUOffsetTRelative(zhang3)

students = builder.EndVector(2)

School.SchoolStart(builder)

School.SchoolAddStudents(builder, students)

school_result = School.SchoolEnd(builder)

builder.Finish(school_result)

buf = builder.Output()

new_school = School.School.GetRootAsSchool(buf, 0)

print(new_school.Students(0).Name())

print(new_school.Students(0).Number())

print(new_school.Students(1).Name())

print(new_school.Students(1).Number())

with open("result.bin", mode="wb") as f:

f.write(buf)

这段代码的作用是新建2个student，一个zhang3,一个li4,然后把student数据保存到School里,最后生成文件result.bin. result.bin就是flatbuffers格式的文件,本质下就是二进制文件.

再执行下面命令:

flatc.exe -j school.fbs

生成一堆java代码.可以用java代码把result.bin文件中的数据读出来.代码如下:

package com.company;

import school.School;

import java.io.*;

import java.nio.ByteBuffer;

import java.nio.file.Files;

import java.nio.file.Path;

import java.nio.file.Paths;

public class Main {

private void process() {

Path path = Paths.get("D:\\yourpath\\result.bin");

try {

byte[] data = Files.readAllBytes(path);

ByteBuffer buf = java.nio.ByteBuffer.wrap(data);

School school = School.getRootAsSchool(buf);

System.out.println(school.students(0).name());

System.out.println(school.students(0).number());

System.out.println(school.students(1).name());

System.out.println(school.students(1).number());

} catch (IOException e) {

System.out.println("IOException");

}

}

public static void main(String[] args) {

Main m = new Main();

m.process();

}

}

另一个名字

google在tensorflow lite中把result.bin这类二进制文件起一个高大上的名字——tflite

参考资料