Android 上的数据格式 FlatBuffers

原文：http://gold.xitu.io/entry/55dd1e3b60b27e6cd500d266

迪鲁宾 分享：Android 工程师 @ 阿里巴巴

FlatBuffers 是一个由谷歌提出的跨平台的数据格式，最大的好处在于不需要反序列化，直接读出数据，减少了decode的计算和内存使用。专门为游戏应用设计，目前已经被 Facebook 使用在 Facebook 应用中。

原文链接

本文已经翻译成中文 《 Android 上的序列化库 FlatBuffers 介绍》，欢迎参加 「掘金翻译计划」，翻译优质的技术文章。

JSON - probably everyone knows this lightweight data format used in almost all modern servers. It weights less, is more human-readable and in general is more dev-friendly than old-fashined, horrible xml. JSON is language-independend data format but parsing data and transforming it to e.g. Java objects costs us time and memory resources.
Several days ago Facebook announced big performance improvement in data handling in its Android app. It was connected with dropping JSON format and replacing it with FlatBuffers in almost entire app. Please check this articleto get some basic knowledge about FlatBuffers and results of transition to it from JSON.

While the results are very promising, at the first glance the implementation isn’t too obvious. Also facebook didn’t say too much. That’s why in this post I’d like to show how we can start our work with FlatBuffers.

FlatBuffers

In short, FlatBuffers is a cross-platform serialization library from Google, created specifically for game development and, as Facebook showed, to follow the 16ms rule of smooth and responsive UI in Android.

But hey, before you throw everything to migrate all your data to FlatBuffers, just make sure that you need this. Sometimes the impact on performance will be imperceptible and sometimes data safety will be more important than a tens of milliseconds difference in computation speed.

What makes FlatBuffers so effective?

• Serialized data is accessed without parsing because of flat binary buffer, even for hierarchical data. Thanks to this we don’t need to initialize parsers (what means to build complicated field mappings) and parse this data, which also takes time.
• FlatBuffers data doesn’t need to allocate more memory than it’s used by buffer itself. We don’t need to allocate extra objects for whole hierarchy of parsed data like it’s done in JSON.

For the real numbers just check again facebook article about migrating to FlatBuffers or Google documentation itself.

Implementation

This article will cover the simplest way of using FlatBuffers in Android app:

• JSON data is converted to FlatBuffer format somewhere outside the app (e.g. bin ary file is delivered as a file or returned directly from API)
• Data model (Java classes) is generated by hand, with flatc (FlatBuffer compiler)
• There are some limitations for JSON file (null fields cannot be used, Date format is parsed as a String)

Probably in the future we’ll prepare more complex solution.

FlatBuffers compiler

At the beginning we have to get flatc - FlatBuffers compiler. It can be built from source code hosted in Google’s flatbuffers repository. Let’s download/clone it. Whole build process is described on FlatBuffers Building documentation. If you are Mac user all you have to do is:

1. Open downloaded source code on \{extract directory}\build\XcodeFlatBuffers.xcodeproj
2. Run flatc scheme (should be selected by default) by pressing Play button or ⌘ + R
3. flatc executable will appear in project root directory.

Now we’re able to use schema compiler which among the others can generate model classes for given schema (in Java, C#, Python, GO and C++) or convert JSON to FlatBuffer binary file.

Schema file

Now we have to prepare schema file which defines data structures we want to de-/serialize. This schema will be used with flatc to create Java models and to transform JSON into Flatbuffer binary file.

Here is a part of our JSON file:

{
  "repos": [
    {
      "id": 27149168,
      "name": "acai",
      "full_name": "google/acai",
      "owner": {
        "login": "google",
        "id": 1342004,
        ...
        "type": "Organization",
        "site_admin": false
      },
      "private": false,
      "html_url": "https://github.com/google/acai",
      "description": "Testing library for JUnit4 and Guice.",
      ...
      "watchers": 21,
      "default_branch": "master"
    },
    ...
  ]
}

Full version is available here. It’s a bit modified version of data which can be taken from Github API call: https://api.github.com/users/google/repos.

Writing a FlatBuffer schema is very well documented, so I won’t delve into this. Also in our case schema won’t be very complicated. All we have to do is to create 3 tables: ReposList, Repo and User, and define root_type. Here is the important part of this schema:

table ReposList {
    repos : [Repo];
}

table Repo {
    id : long;
    name : string;
    full_name : string;
    owner : User;
    //...
    labels_url : string (deprecated);
    releases_url : string (deprecated);
}

table User {
    login : string;
    id : long;
    avatar_url : string;
    gravatar_id : string;
    //...
    site_admin : bool;
}

root_type ReposList;

Full schema file is available here.

FlatBuffers data file

Great, now all we have to do is to convert repos_json.json to FlatBuffers binary file and generate Java models which will be able to represent our data in Java-friendly style (all files required in this operation are available in our repository):

$ ./flatc -j -b repos_schema.fbs repos_json.json

If everything goes well, here is a list of generated files:

• repos_json.bin (we’ll rename it to repos_flat.bin)
• Repos/Repo.java
• Repos/ReposList.java
• Repos/User.java

Android app

Now let’s create our example app to check how FlatBuffers format works in practice. Here is the screenshot of it:

ProgressBar will be used only to show how incorrect data handling (in UI thread) can affect smoothness of user interface.

app/build.gradle file of our app will look like this:

apply plugin: 'com.android.application'
apply plugin: 'com.jakewharton.hugo'

android {
    compileSdkVersion 22
    buildToolsVersion "23.0.0 rc2"

    defaultConfig {
        applicationId "frogermcs.io.flatbuffs"
        minSdkVersion 15
        targetSdkVersion 22
        versionCode 1
        versionName "1.0"
    }
    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
        }
    }
}

dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])
    compile 'com.android.support:appcompat-v7:22.2.1'
    compile 'com.google.code.gson:gson:2.3.1'
    compile 'com.jakewharton:butterknife:7.0.1'
    compile 'io.reactivex:rxjava:1.0.10'
    compile 'io.reactivex:rxandroid:1.0.0'
}

Of course it’s not necessary to use Rx or ButterKnife in our example, but why not to make this app a bit nicer