Using a TensorFlow Lite model in your mobile app requires multiple considerations: you must choose a pre-trained or custom model, convert the model to a TensorFLow Lite format, and finally, integrate the model in your app.
1. Choose a model
Depending on the use case, you can choose one of the popular open-sourced models, such as InceptionV3 or MobileNets, and re-train these models with a custom data set or even build your own custom model.
Use a pre-trained model
MobileNets is a family of mobile-first computer vision models for TensorFlow designed to effectively maximize accuracy, while taking into consideration the restricted resources for on-device or embedded applications. MobileNets are small, low-latency, low-power models parameterized to meet the resource constraints for a variety of uses. They can be used for classification, detection, embeddings, and segmentation—similar to other popular large scale models, such asInception. Google provides 16 pre-trained ImageNet classification checkpoints for MobileNets that can be used in mobile projects of all sizes.
Inception-v3 is an image recognition model that achieves fairly high accuracy recognizing general objects with 1000 classes, for example, "Zebra", "Dalmatian", and "Dishwasher". The model extracts general features from input images using a convolutional neural network and classifies them based on those features with fully-connected and softmax layers.
On Device Smart Reply is an on-device model that provides one-touch replies for incoming text messages by suggesting contextually relevant messages. The model is built specifically for memory constrained devices, such as watches and phones, and has been successfully used in Smart Replies on Android Wear. Currently, this model is Android-specific.
These pre-trained models are available for download
Re-train Inception-V3 or MobileNet for a custom data set
These pre-trained models were trained on the ImageNet data set which contains 1000 predefined classes. If these classes are not sufficient for your use case, the model will need to be re-trained. This technique is called transfer learningand starts with a model that has been already trained on a problem, then retrains the model on a similar problem. Deep learning from scratch can take days, but transfer learning is fairly quick. In order to do this, you need to generate a custom data set labeled with the relevant classes.
The TensorFlow for Poets codelab walks through the re-training process step-by-step. The code supports both floating point and quantized inference.
Train a custom model
A developer may choose to train a custom model using Tensorflow (see the Tutorials for examples of building and training models). If you have already written a model, the first step is to export this to a tf.GraphDef
file. This is required because some formats do not store the model structure outside the code, and we must communicate with other parts of the framework. See Exporting the Inference Graph to create .pb file for the custom model.
TensorFlow Lite currently supports a subset of TensorFlow operators. Refer to the TensorFlow Lite & TensorFlow Compatibility Guide for supported operators and their usage. This set of operators will continue to grow in future Tensorflow Lite releases.
2. Convert the model format
The model generated (or downloaded) in the previous step is a standard Tensorflow model and you should now have a .pb or .pbtxt tf.GraphDef
file. Models generated with transfer learning (re-training) or custom models must be converted—but, we must first freeze the graph to convert the model to the Tensorflow Lite format. This process uses several model formats:
tf.GraphDef
(.pb) —A protobuf that represents the TensorFlow training or computation graph. It contains operators, tensors, and variables definitions.(包括了运算、向量、变量定义。是图和数据 .pb=meta+data,将变量及其取值以常量constant的形式存储,同时去掉一些不必要的节点)- CheckPoint (.ckpt) —Serialized variables from a TensorFlow graph. Since this does not contain a graph structure, it cannot be interpreted by itself.( 将 计算图结构meta 与 变量取值data 分开)
FrozenGraphDef
—A subclass ofGraphDef
that does not contain variables. AGraphDef
can be converted to aFrozenGraphDef
by taking a CheckPoint and aGraphDef
, and converting each variable into a constant using the value retrieved from the CheckPoint.SavedModel
—AGraphDef
and CheckPoint with a signature that labels input and output arguments to a model. AGraphDef
and CheckPoint can be extracted from aSavedModel
.- TensorFlow Lite model (.tflite) —A serialized FlatBuffer that contains TensorFlow Lite operators and tensors for the TensorFlow Lite interpreter, similar to a
FrozenGraphDef
.