TF2 基础 ：from_tensor_slices,shuffle,batch

tf

主要总结整理TF2的使用过程总的基础的的知识点

图片数据的切片，混合，打包

测试数据集为：

import tensorflow as tf
import numpy as np
 

np.random.seed(10)#固定每次的随机数
features, labels = (np.random.sample((4, 2, 2)),  # 模拟6组数据，每组数据3个特征
                    np.random.sample((4, 1)))  # 模拟6组数据，每组数据对应一个标签，注意两者的维数必须匹配

print((features))  #  打印feature数据
print((labels))  #  打印标签数据

得到测试用的数据集为：

feture:
[[[0.77132064 0.02075195]
  [0.63364823 0.74880388]]

 [[0.49850701 0.22479665]
  [0.19806286 0.76053071]]

 [[0.16911084 0.08833981]
  [0.68535982 0.95339335]]

 [[0.00394827 0.51219226]
  [0.81262096 0.61252607]]]

lables:
[[0.72175532]
 [0.29187607]
 [0.91777412]
 [0.71457578]]

我们用features表示4张大小为[2,2]的图片数据，labels表示4张图片的标签数据，例如属于哪一类图片

from_tensor_slices

执行以下操作对数据集进行切片操作

data = tf.data.Dataset.from_tensor_slices((features, labels))

输出的结果是包含每一张图片的数据信息和特征信息的节点，4张图片对应的是4个Tensor节点

----------from_tensor_slices--------------

(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=array([[0.77132064, 0.02075195],
       [0.63364823, 0.74880388]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.72175532])>)
       
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.49850701, 0.22479665],
       [0.19806286, 0.76053071]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.29187607])>)
       
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.16911084, 0.08833981],
       [0.68535982, 0.95339335]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.91777412])>)
       
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.00394827, 0.51219226],
       [0.81262096, 0.61252607]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.71457578])>)

shuffle

对from_tensor_slices处理的数据，进行混合，混合就是打乱原数组之间的顺序，数组的数据大小和内容并没有改变;
混合的数据越大，混合程度越高

shuffle_data =data.shuffle(4)#4表示每次混合的buffer size，因为我们只有四个数据，直接混合所有数据

混合后的结果为：
可以跟from_tensor_slices数据进行比较下，原来的数据顺序为[0,1,2,3],混合后为[3,2,1,0]

----------shuffle--------------
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.00394827, 0.51219226],
       [0.81262096, 0.61252607]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.71457578])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.169121084, 0.08833981],
       [0.68535982, 0.95339335]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.91777412])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.49850701, 0.22479665],
       [0.19806286, 0.76053071]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.29187607])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.77132064, 0.02075195],
       [0.63364823, 0.74880388]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.72175532])>)

batch##

batch_data =data.batch(1)

对shuffle处理后的数据进行打包，如果为1，则数据内容和格式跟shuffle的数据相同，相当于没有处理

----------batch(1)--------------
(<tf.Tensor: shape=(1, 2, 2), dtype=float64, numpy=
array([[[0.77132064, 0.02075195],
        [0.63364823, 0.74880388]]])>, <tf.Tensor: shape=(1, 1), dtype=float64, numpy=array([[0.72175532]])>)

(<tf.Tensor: shape=(1, 2, 2), dtype=float64, numpy=
array([[[0.00394827, 0.51219226],
        [0.81262096, 0.61252607]]])>, <tf.Tensor: shape=(1, 1), dtype=float64, numpy=array([[0.71457578]])>)

(<tf.Tensor: shape=(1, 2, 2), dtype=float64, numpy=
array([[[0.16911084, 0.08833981],
        [0.68535982, 0.95339335]]])>, <tf.Tensor: shape=(1, 1), dtype=float64, numpy=array([[0.91777412]])>)

(<tf.Tensor: shape=(1, 2, 2), dtype=float64, numpy=
array([[[0.49850701, 0.22479665],
        [0.19806286, 0.76053071]]])>, <tf.Tensor: shape=(1, 1), dtype=float64, numpy=array([[0.29187607]])>)

如果batch(2)，则每个节点信息中包含两张图片数据，以此类推

(<tf.Tensor: shape=(2, 2, 2), dtype=float64, numpy=
array([[[0.00394827, 0.51219226],
        [0.81262096, 0.61252607]],

       [[0.77132064, 0.02075195],
        [0.63364823, 0.74880388]]])>, <tf.Tensor: shape=(2, 1), dtype=float64, numpy=
array([[0.71457578],
       [0.72175532]])>)


(<tf.Tensor: shape=(2, 2, 2), dtype=float64, numpy=
array([[[0.49850701, 0.22479665],
        [0.19806286, 0.76053071]],

       [[0.16911084, 0.08833981],
        [0.68535982, 0.95339335]]])>, <tf.Tensor: shape=(2, 1), dtype=float64, numpy=
array([[0.29187607],
       [0.91777412]])>)

总结一下：

1. from_tensor_slices对原图片数据进行切片，有多上张图片就切成多少个数据
2. shuffle,对切好后的数据进行混合，交换下顺序
3. batch对数据进行打包，便于批量化处理；batch后的节点数为(数据集大小/batch大小)，两个数字可能不是整除，会导致一个batch大小可能小于等于batch size

完整代码：shuffle_and_batch.py

import tensorflow as tf
import numpy as np
 
from    tensorflow.keras import datasets

np.random.seed(10)#固定每次的随机数
features, labels = (np.random.sample((4, 2, 2)),  # 模拟6组数据，每组数据3个特征
                    np.random.sample((4, 1)))  # 模拟6组数据，每组数据对应一个标签，注意两者的维数必须匹配

print((features))  #  打印feature数据
print((labels))  #  打印标签数据

print('----------from_tensor_slices--------------') 

for element in features: 
  print(element) 

#切片转换，将数据转化成tesor节点数据
data = tf.data.Dataset.from_tensor_slices((features, labels))

for element in data: 
  print(element) 

print('----------shuffle--------------') 

#shuffle_data = tf.data.Dataset.from_tensor_slices((features,labels)).shuffle(1000).batch(128)
shuffle_data =data.shuffle(4)

for element in shuffle_data: 
  print(element) 
  
  
print('----------batch--------------') 
batch_data =shuffle_data.batch(2)
  
for element in batch_data: 
  print(element) 
  
  
batch_data.repeat(2)

output:

runfile('G:/GitHub/tensorflow/Spyder/TF2/shuffle_and_batch.py', wdir='G:/GitHub/tensorflow/Spyder/TF2')
[[[0.77132064 0.02075195]
  [0.63364823 0.74880388]]

 [[0.49850701 0.22479665]
  [0.19806286 0.76053071]]

 [[0.16911084 0.08833981]
  [0.68535982 0.95339335]]

 [[0.00394827 0.51219226]
  [0.81262096 0.61252607]]]
[[0.72175532]
 [0.29187607]
 [0.91777412]
 [0.71457578]]
----------from_tensor_slices--------------
[[0.77132064 0.02075195]
 [0.63364823 0.74880388]]
[[0.49850701 0.22479665]
 [0.19806286 0.76053071]]
[[0.16911084 0.08833981]
 [0.68535982 0.95339335]]
[[0.00394827 0.51219226]
 [0.81262096 0.61252607]]
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.77132064, 0.02075195],
       [0.63364823, 0.74880388]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.72175532])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.49850701, 0.22479665],
       [0.19806286, 0.76053071]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.29187607])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.16911084, 0.08833981],
       [0.68535982, 0.95339335]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.91777412])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.00394827, 0.51219226],
       [0.81262096, 0.61252607]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.71457578])>)
----------shuffle--------------
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.49850701, 0.22479665],
       [0.19806286, 0.76053071]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.29187607])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.16911084, 0.08833981],
       [0.68535982, 0.95339335]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.91777412])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.77132064, 0.02075195],
       [0.63364823, 0.74880388]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.72175532])>)
(<tf.Tensor: shape=(2, 2), dtype=float64, numpy=
array([[0.00394827, 0.51219226],
       [0.81262096, 0.61252607]])>, <tf.Tensor: shape=(1,), dtype=float64, numpy=array([0.71457578])>)
----------batch--------------
(<tf.Tensor: shape=(2, 2, 2), dtype=float64, numpy=
array([[[0.77132064, 0.02075195],
        [0.63364823, 0.74880388]],

       [[0.16911084, 0.08833981],
        [0.68535982, 0.95339335]]])>, <tf.Tensor: shape=(2, 1), dtype=float64, numpy=
array([[0.72175532],
       [0.91777412]])>)
(<tf.Tensor: shape=(2, 2, 2), dtype=float64, numpy=
array([[[0.00394827, 0.51219226],
        [0.81262096, 0.61252607]],

       [[0.49850701, 0.22479665],
        [0.19806286, 0.76053071]]])>, <tf.Tensor: shape=(2, 1), dtype=float64, numpy=
array([[0.71457578],
       [0.29187607]])>)