tensorflow lite（一）

模型文件格式：

模型的主结构，modle结构体
/tensorflow$ vi contrib/lite/schema/schema.fbs +420
table Model {
  // Version of the schema.
  version:uint;

  // A list of all operator codes used in this model. This is
  // kept in order because operators carry an index into this
  // vector.
  operator_codes:[OperatorCode];	整个模型的所有算子

  // All the subgraphs of the model. The 0th is assumed to be the main
  // model.
  subgraphs:[SubGraph];	所有子图，子图中第一个元素是主图

  // A description of the model.
  description:string;

  // Buffers of the model
  buffers:[Buffer];	数据存储区域，主要存储模型的权重信息

}

是不是每个模型都会定义一个这样的table Modle，好像是也好像不是
子图是里面最重要的部分

// The root type, defining a model.
table SubGraph {
  // A list of all tensors used in this model.
  tensors:[Tensor];	子图中的各个tensor

  // Indices of the input tensors.
  inputs:[int];	和output对应，用来索引的维护着子图中 Tensor 与输入输出之间的对应关系

  // Indices of the output tensors.
  outputs:[int];

  // All operators, in execution order.
  operators:[Operator];	定义子图中的算子

  // Name of subgraph (used for debugging).
  name:string;
}

tensor结构

table Tensor {
  // The tensor shape. The meaning of each entry is operator-specific but
  // builtin ops use: [batch size, height, width, number of channels] (That's
  // Tensorflow's NHWC).
  shape:[int];	维度
  type:TensorType;	数据类型
  // An index that refers to the buffers table at the root of the model. Or,
  // if there is no data buffer associated (i.e. intermediate results), then
  // this is 0 (which refers to an always existent empty buffer).
  //
  // The data_buffer itself is an opaque container, with the assumption that the
  // target device is little-endian. In addition, all builtin operators assume
  // the memory is ordered such that if `shape` is [4, 3, 2], then index
  // [i, j, k] maps to data_buffer[i*3*2 + j*2 + k].
  buffer:uint;	以索引的形式给出了这个tensor需要用到的子图的哪个buffer
  name:string;  // For debugging and importing back into tensorflow.
  quantization:QuantizationParameters;  // Optional.	这个是新加的吗？
}

SubGraph中最重要的结构体

// An operator takes tensors as inputs and outputs. The type of operation being
// performed is determined by an index into the list of valid OperatorCodes,
// while the specifics of each operations is configured using builtin_options
// or custom_options.
table Operator {
  // Index into the operator_codes array. Using an integer here avoids
  // complicate map lookups.
  opcode_index:uint;	以索引的方式指明Operator对应的是哪个算子

  // Optional input and output tensors are indicated by -1.
  inputs:[int];	inputs 和 outputs 则是 Tensor 的索引值，指明该 Operator 的输入输出信息。
  outputs:[int];	

  builtin_options:BuiltinOptions;
  custom_options:[ubyte];
  custom_options_format:CustomOptionsFormat;
}

解析器

终端设备会通过mmap将模型文件映射到客户端内存中，包含tensor、opertor、buffer。同时创建只读区域和分配可写buffer区域

kernel：包含了集体执行计算的代码，模型中的tensor会被加载成TFliteTensor的格式（context.h文件中）

// An tensor in the interpreter system which is a wrapper around a buffer of
// data including a dimensionality (or NULL if not currently defined).
typedef struct {
  // The data type specification for data stored in `data`. This affects
  // what member of `data` union should be used.
  TfLiteType type;
  // A union of data pointers. The appropriate type should be used for a typed
  // tensor based on `type`.
  TfLitePtrUnion data;
  // A pointer to a structure representing the dimensionality interpretation
  // that the buffer should have. NOTE: the product of elements of `dims`
  // and the element datatype size should be equal to `bytes` below.
  TfLiteIntArray* dims;
  // Quantization information.
  TfLiteQuantizationParams params;
  // How memory is mapped
  //  kTfLiteMmapRo: Memory mapped read only.
  //  i.e. weights
  //  kTfLiteArenaRw: Arena allocated read write memory
  //  (i.e. temporaries, outputs).
  TfLiteAllocationType allocation_type;
  // The number of bytes required to store the data of this Tensor. I.e.
  // (bytes of each element) * dims[0] * ... * dims[n-1].  For example, if
  // type is kTfLiteFloat32 and dims = {3, 2} then
  // bytes = sizeof(float) * 3 * 2 = 4 * 3 * 2 = 24.
  size_t bytes;

  // An opaque pointer to a tflite::MMapAllocation
  const void* allocation;

  // Null-terminated name of this tensor.
  const char* name;
} TfLiteTensor;

并集中给TfLiteContext管理

typedef struct TfLiteContext {
  // Number of tensors in the context.
  int tensors_size;

  // The execution plan contains a list of the node indices in execution
  // order. execution_plan->size is the current number of nodes. And,
  // execution_plan->data[0] is the first node that needs to be run.
  // TfLiteDelegates can traverse the current execution plan by iterating
  // through each member of this array and using GetNodeAndRegistration() to
  // access details about a node. i.e.
  // TfLiteIntArray* execution_plan;
  // TF_LITE_ENSURE_STATUS(context->GetExecutionPlan(context, &execution_plan));
  // for (int exec_index = 0; exec_index < execution_plan->size; exec_index++) {
  //    int node_index = execution_plan->data[exec_index];
  //    TfLiteNode* node;
  //    TfLiteRegistration* reg;
  //    context->GetNodeAndRegistration(context, node_index, &node, &reg);
  // }
  // WARNING: This is an experimental interface that is subject to change.
  TfLiteStatus (*GetExecutionPlan)(struct TfLiteContext* context,
                                   TfLiteIntArray** execution_plan);

  // An tensor of tensors in the interpreter context (of length `tensors_size`)
  TfLiteTensor* tensors;

  // opaque full context ptr (an opaque c++ data structure)
  void* impl_;

  // Request memory pointer be resized. Updates dimensions on the tensor.
  // NOTE: ResizeTensor takes ownership of newSize.
  TfLiteStatus (*ResizeTensor)(struct TfLiteContext*, TfLiteTensor* tensor,
                               TfLiteIntArray* new_size);
  // Request that a error be reported with format string msg.
  void (*ReportError)(struct TfLiteContext*, const char* msg, ...);

  // Add `tensors_to_add` tensors, preserving pre-existing Tensor entries.  If
  // non-null, the value pointed to by `first_new_tensor_index` will be set to
  // the index of the first new tensor.
  TfLiteStatus (*AddTensors)(struct TfLiteContext*, int tensors_to_add,
                             int* first_new_tensor_index);

  // Get a Tensor node by node_index.
  // WARNING: This is an experimental interface that is subject to change.
  TfLiteStatus (*GetNodeAndRegistration)(struct TfLiteContext*, int node_index,
                                         TfLiteNode** node,
                                         TfLiteRegistration** registration);

  // Replace ops with delegate.
  TfLiteStatus (*ReplaceSubgraphsWithDelegateKernels)(
      struct TfLiteContext*, TfLiteRegistration registration,
      const TfLiteIntArray* nodes_to_replace);

  // TODO(ahentz): we should create a more general mechanism for this sort of
  // library-global objects.
  void* gemm_context;
} TfLiteContext;

每个 Tensor 的指针都指向了内存中的只读 Buffer 区域或是一开始新分配的可写入 Buffer 区域

模型中的 Operator 被重新加载为 TfLiteNode

// A structure representing an instance of a node.
// This structure only exhibits the inputs, outputs and user defined data, not
// other features like the type.
typedef struct {
  // Inputs to this node expressed as indices into the simulator's tensors.
  TfLiteIntArray* inputs;

  // Outputs to this node expressed as indices into the simulator's tensors.
  TfLiteIntArray* outputs;

  // Temporary tensors uses during the computations. This usually contains no
  // tensors, but ops are allowed to change that if they need scratch space of
  // any sort.
  TfLiteIntArray* temporaries;

  // Opaque data provided by the node implementer through `Registration.init`.
  void* user_data;

  // Opaque data provided to the node if the node is a builtin. This is usually
  // a structure defined in builtin_op_data.h
  void* builtin_data;

  // Custom initial data. This is the opaque data provided in the flatbuffer.
  // WARNING: This is an experimental interface that is subject to change.
  const void* custom_initial_data;
  int custom_initial_data_size;
} TfLiteNode;

它包含输入输出的 Tensor 索引值。这些 Node 对应的操作符存储于 TfLiteRegistration 中

typedef struct _TfLiteRegistration {
  // Initializes the op from serialized data.
  // If a built-in op:
  //   `buffer` is the op's params data (TfLiteLSTMParams*).
  //   `length` is zero.
  // If custom op:
  //   `buffer` is the op's `custom_options`.
  //   `length` is the size of the buffer.
  //
  // Returns a type-punned (i.e. void*) opaque data (e.g. a primitive pointer
  // or an instance of a struct).
  //
  // The returned pointer will be stored with the node in the `user_data` field,
  // accessible within prepare and invoke functions below.
  // NOTE: if the data is already in the desired format, simply implement this
  // function to return `nullptr` and implement the free function to be a no-op.
  void* (*init)(TfLiteContext* context, const char* buffer, size_t length);

  // The pointer `buffer` is the data previously returned by an init invocation.
  void (*free)(TfLiteContext* context, void* buffer);

  // prepare is called when the inputs this node depends on have been resized.
  // context->ResizeTensor() can be called to request output tensors to be
  // resized.
  //
  // Returns kTfLiteOk on success.
  TfLiteStatus (*prepare)(TfLiteContext* context, TfLiteNode* node);

  // Execute the node (should read node->inputs and output to node->outputs).
  // Returns kTfLiteOk on success.
  TfLiteStatus (*invoke)(TfLiteContext* context, TfLiteNode* node);

  // Builtin codes. If this kernel refers to a builtin this is the code
  // of the builtin. This is so we can do marshaling to other frameworks like
  // NN API. Note, it is the responsibility of the registration binder to
  // set this properly.
  int32_t builtin_code;

  // Custom op name. If the op is a builtin, this will be null.
  // WARNING: This is an experimental interface that is subject to change.
  const char* custom_name;
} TfLiteRegistration;

它包含了指向 Kernel 的函数指针。OpResolver 负责维护函数指针的对应关系。

// Abstract interface that returns TfLiteRegistrations given op codes or custom
// op names. This is the mechanism that ops being referenced in the flatbuffer
// model are mapped to executable function pointers (TfLiteRegistrations).
class OpResolver {
 public:
  // Finds the op registration for a builtin operator by enum code.
  virtual TfLiteRegistration* FindOp(tflite::BuiltinOperator op) const = 0;
  // Finds the op registration of a custom operator by op name.
  virtual TfLiteRegistration* FindOp(const char* op) const = 0;
  virtual ~OpResolver() {}
};

更多关于解析器的可以看：

• lite/context.h

• lite/model.h

• lite/interpreter.h

• lite/kernels/register.h

或者这个教程也不错，虽然很短：https://www.bilibili.com/video/av24219725/