TensorFlow-Slim Data

TF-Slim provides a data loading library for facilitating the reading of data from various formats. TF-Slim's data modules are composed of several layers of abstraction to make it flexible enough to support multiple file storage types, such as TFRecords or Text files, data encoding and features naming schemes.

TF-Slim提供了一个数据加载库，以便于读取各种格式的数据。TF-Slim的数据模块由多个抽象层组成，以使其足够灵活，以支持多个文件存储类型，例如TFRecords或文本文件、数据编码和特征命名方案。

Overview

The task of loading data has two main components: (1) specification of how a dataset is represented so it can be read and interpreted and (2) instruction for providing the data to consumers of the dataset.

Secondly, one must specify instructions for how the data is actually provided and housed in memory. For example, if the data is sharded over many sources, should it be read in parallel from these sources? Should it be read serially? Should the data be shuffled in memory?

加载数据的任务有两个主要组成部分：

1）说明数据集是如何表示的，以便它可以被读取和解释，

2）向数据集的使用者提供数据的指令。
其次，必须指定如何实际提供数据并存储在内存中的指令。例如，如果数据在许多源上被划分，那么应该从这些源并行读取数据吗？应该连续读取？数据应该在内存中洗牌吗？

Dataset Specification

TF-Slim defines a dataset to be a set of files (that may or may not be encoded) representing a finite set of samples, and which can be read to provide a predefined set of entities or items. For example, a dataset might be stored over thousands of files or a single file. The files might store the data in clear text or some advanced encoding scheme. It might provide a single item, like an image, or several items, like an image, a class label and a scene label.

TF-Slim将数据集定义为表示一组有限样本的一组文件（可以被编码或可能不被编码），并且可以被读取以提供预定义的实体或项目集。例如，数据集可以存储在数千个文件或单个文件上。文件可以以明文或一些高级编码方案存储数据。它可以提供一个单一的项目，像一个图像，或者几个项目，像一个图像，一个类标签和一个场景标签。

More concretely, TF-Slim's dataset is a tuple that encapsulates the following elements of a dataset specification:

更具体地说，TF-Slim的数据集是封装数据集规范的以下元素的元组：

• data_sources: A list of file paths that together make up the dataset  数据源：文件路径列表
• reader: A TensorFlow Reader appropriate for the file type in data_sources. 适合数据集格式的Tensorflow -reader
• decoder: A TF-Slim data_decoder class which is used to decode the content of the read dataset files. 解码器：解码数据集文件内容。
• num_samples: The number of samples in the dataset. 数据集样例数量（如：训练数据集，验证数据集，测试数据集）
• items_to_descriptions: A map from the items provided by the dataset to descriptions of each. 从数据集提供的项目到每个描述的映射。

In a nutshell, a dataset is read by (a) opening the files specified by data_sources using the given reader class (b) decoding the files using the given decoder and (c) allowing the user to request a list of items to be returned as Tensors.

简言之，

（a）通过使用给定的阅读器类打开数据集指定的文件读取数据集

（b）使用给定解码器解码文件

（c）允许用户请求要返回的项目列表作为张量。

Data Decoders

A data_decoder is a class which is given some (possibly serialized/encoded) data and returns a list of Tensors. In particular, a given data decoder is able to decode a predefined list of items and can return a subset or all of them, when requested:

数据解码器是给定一些（可能是序列化的/编码的）数据并返回张量列表的类。特别地，给定的数据解码器能够解码预定义的项目列表，并且当请求时可以返回子集或所有这些子集。

# Load the data
my_encoded_data = ...
data_decoder = MyDataDecoder()

# Decode the inputs and labels:
decoded_input, decoded_labels = data_decoder.Decode(data, ['input', 'labels'])

# Decode just the inputs:
decoded_input = data_decoder.Decode(data, ['input'])

# Check which items a data decoder knows how to decode:
for item in data_decoder.list_items():
  print(item)

Example: TFExampleDecoder

The tfexample_decoder.py is a data decoder which decodes serialized TFExample protocol buffers. A TFExample protocol buffer is a map from keys (strings) to either a tf.FixedLenFeature or tf.VarLenFeature. Consequently, to decode a TFExample, one must provide a mapping from one or more TFExample fields to each of the items that the tfexample_decoder can provide. For example, a dataset of TFExamples might store images in various formats and each TFExample might contain an encoding key and a format key which can be used to decode the image using the appropriate decoder (jpg, png, etc).

tfexample_decoder.py 是一个数据解码器，它对序列化的TFExample的协议缓冲区进行解码。其是TFExample协议缓冲区是从键（字符串）到 tf.FixedLenFeature或tf.VarLenFeature.的映射。

因此，为了解码TFExample，必须提供从一个或多个TFExample字段到tfexample_decoder解码器所能提供的每个项目的映射。例如，TFExamples的数据集可以存储各种格式的图像，并且每个TF示例可以包含编码密钥和格式密钥，该格式密钥可以用于使用适当的解码器（JPG、PNG等）解码图像。

To make this possible, the tfexample_decoder is constructed by specifying the a map of TFExample keys to either tf.FixedLenFeature or tf.VarLenFeature as well as a set of ItemHandlers. An ItemHandler provides a mapping from TFExample keys to the item being provided. Because a tfexample_decoder might return multiple items, one often constructs a tfexample_decoder using multiple ItemHandlers.

为了使这成为可能，tfexample_decoder解码器是通过指定TFExample键的映射到tf.FixedLenFeature或tf.VarLenFeature 以及一组ItemHandlers程序来构造的。一个ItemHandler提供了从TFExample键到所提供的项目的映射。因为tfexample_decoder解码器可能返回多个项，所以常常使用多个ItemHandlers构造tfexample_decoder 解码器。
 

tfexample_decoder provides some predefined ItemHandlers which take care of the common cases of mapping TFExamples to images, Tensors and SparseTensors. For example, the following specification might be used to decode a dataset of images:

tfexample_decoder解码器提供一些预定义的ItemHandlers，它负责将TFExamples映射到图像、张量和SparseTensors的常见情况。例如，下面的规范可以用来解码图像数据集：

keys_to_features = {
    'image/encoded': tf.FixedLenFeature((), tf.string, default_value=''),
    'image/format': tf.FixedLenFeature((), tf.string, default_value='raw'),
    'image/class/label': tf.FixedLenFeature(
        [1], tf.int64, default_value=tf.zeros([1], dtype=tf.int64)),
}

items_to_handlers = {
    'image': tfexample_decoder.Image(
      image_key = 'image/encoded',
      format_key = 'image/format',
      shape=[28, 28],
      channels=1),
    'label': tfexample_decoder.Tensor('image/class/label'),
}

decoder = tfexample_decoder.TFExampleDecoder(
    keys_to_features, items_to_handlers)

Notice that the TFExample is parsed using three keys: image/encoded, image/format and image/class/label. Additionally, the first two keys are mapped to a single item named 'image'. As defined, this data_decoder provides two items named 'image' and 'label'.

   样例代码有三个键值： image/encoded, image/format and image/class/label，此外，前两个键值映射单个项目'image',如代码所示，这个data_decoder提供了两个项目：‘image’和‘label'

Data Provision 数据提供

A data_provider is a class which provides Tensors for each item requested: 

my_data_provider = ...
image, class_label, bounding_box = my_data_provider.get(
    ['image', 'label', 'bb'])

通过get语句获取数据。对应相应的项目名。

The dataset_data_provider is a data_provider that provides data from a given dataset specification:

dataset = GetDataset(...)
data_provider = dataset_data_provider.DatasetDataProvider(
    dataset, common_queue_capacity=32, common_queue_min=8)

数据提供函数。其可能控制以下设置。

The dataset_data_provider enables control over several elements of data provision:

• How many concurrent readers are used.  并行读取数据的reader的个数
• Whether the data is shuffled as its loaded into its queue 在入队前是否打乱
• Whether to take a single pass over the data or read data indefinitely. 是否对数据进行单次传递或无限期读取数据