使用Python解析MNIST数据集

最新推荐文章于 2023-06-05 17:31:04 发布

LeadAI学院

最新推荐文章于 2023-06-05 17:31:04 发布

阅读量1.2k

点赞数

五月两场 | NVIDIA DLI 深度学习入门课程

5月19日/5月26日一天密集式学习快速带你入门阅读全文 >

正文共948个字（不含代码），2张图，预计阅读时间15分钟。

前言

最近在学习Keras，要使用到LeCun大神的MNIST手写数字数据集，直接从官网上下载了4个压缩包：

MNIST数据集

解压后发现里面每个压缩包里有一个idx-ubyte文件，没有图片文件在里面。回去仔细看了一下官网后发现原来这是IDX文件格式，是一种用来存储向量与多维度矩阵的文件格式。

IDX文件格式

官网上的介绍如下:

THE IDX FILE FORMAT

the IDX file format is a simple format for vectors and multidimensional matrices of various numerical types.

The basic format is

1magic number
2size in dimension 0
3size in dimension 1
4size in dimension 2
5.....
6size in dimension N
7data

The magic number is an integer (MSB first). The first 2 bytes are always 0.

The third byte codes the type of the data:

10x08: unsigned byte
20x09: signed byte
30x0B: short (2 bytes)
40x0C: int (4 bytes)
50x0D: float (4 bytes)
60x0E: double (8 bytes)

The 4-th byte codes the number of dimensions of the vector/matrix: 1 for vectors, 2 for matrices....

The sizes in each dimension are 4-byte integers (MSB first, high endian, like in most non-Intel processors).

The data is stored like in a C array, i.e. the index in the last dimension changes the fastest.

解析脚本

根据以上解析规则，我使用了Python里的struct模块对文件进行读写（如果不熟悉struct模块的可以看我的另一篇博客文章《Python中对字节流/二进制流的操作:struct模块简易使用教程》）。IDX文件的解析通用接口如下：

 1# 解析idx1格式
 2def decode_idx1_ubyte(idx1_ubyte_file):
 3"""
 4解析idx1文件的通用函数
 5:param idx1_ubyte_file: idx1文件路径
 6:return: np.array类型对象
 7"""
 8return data
 9def decode_idx3_ubyte(idx3_ubyte_file):
10"""
11解析idx3文件的通用函数
12:param idx3_ubyte_file: idx3文件路径
13:return: np.array类型对象
14"""
15return data

针对MNIST数据集的解析脚本如下：

  1# encoding: utf-8
  2"""
  3@author: monitor1379 
  4@contact: yy4f5da2@hotmail.com
  5@site: www.monitor1379.com
  6@version: 1.0
  7@license: Apache Licence
  8@file: mnist_decoder.py
  9@time: 2016/8/16 20:03
 10对MNIST手写数字数据文件转换为bmp图片文件格式。
 11数据集下载地址为http://yann.lecun.com/exdb/mnist。
 12相关格式转换见官网以及代码注释。
 13========================
 14关于IDX文件格式的解析规则：
 15========================
 16THE IDX FILE FORMAT
 17the IDX file format is a simple format for vectors and  multidimensional matrices of various numerical types.
 18The basic format is
 19magic number
 20size in dimension 0
 21size in dimension 1
 22size in dimension 2
 23.....
 24size in dimension N
 25data
 26The magic number is an integer (MSB first). The first 2 bytes are always 0.
 27The third byte codes the type of the data:
 280x08: unsigned byte
 290x09: signed byte
 300x0B: short (2 bytes)
 310x0C: int (4 bytes)
 320x0D: float (4 bytes)
 330x0E: double (8 bytes)
 34The 4-th byte codes the number of dimensions of the vector/matrix: 1 for vectors, 2 for matrices....
 35The sizes in each dimension are 4-byte integers (MSB first, high endian, like in most non-Intel processors).
 36The data is stored like in a C array, i.e. the index in the last dimension changes the fastest.
 37"""
 38import numpy as np
 39import struct
 40import matplotlib.pyplot as plt
 41# 训练集文件
 42train_images_idx3_ubyte_file = '../../data/mnist/bin/train-images.idx3-ubyte'
 43# 训练集标签文件
 44train_labels_idx1_ubyte_file = '../../data/mnist/bin/train-labels.idx1-ubyte'
 45# 测试集文件
 46test_images_idx3_ubyte_file = '../../data/mnist/bin/t10k-images.idx3-ubyte'
 47# 测试集标签文件
 48test_labels_idx1_ubyte_file = '../../data/mnist/bin/t10k-labels.idx1-ubyte'
 49def decode_idx3_ubyte(idx3_ubyte_file):
 50"""
 51解析idx3文件的通用函数
 52:param idx3_ubyte_file: idx3文件路径
 53:return: 数据集
 54"""
 55# 读取二进制数据
 56bin_data = open(idx3_ubyte_file, 'rb').read()
 57# 解析文件头信息，依次为魔数、图片数量、每张图片高、每张图片宽
 58offset = 0
 59fmt_header = '>iiii'
 60magic_number, num_images, num_rows, num_cols = struct.unpack_from(fmt_header, bin_data, offset)
 61print '魔数:%d, 图片数量: %d张, 图片大小: %d*%d' % (magic_number, num_images, num_rows, num_cols)
 62# 解析数据集
 63image_size = num_rows * num_cols
 64offset += struct.calcsize(fmt_header)
 65fmt_image = '>' + str(image_size) + 'B'
 66images = np.empty((num_images, num_rows, num_cols))
 67for i in range(num_images):
 68if (i + 1) % 10000 == 0:
 69print '已解析 %d' % (i + 1) + '张'
 70images[i] = np.array(struct.unpack_from(fmt_image, bin_data, offset)).reshape((num_rows, num_cols))
 71offset += struct.calcsize(fmt_image)
 72return images
 73def decode_idx1_ubyte(idx1_ubyte_file):
 74"""
 75解析idx1文件的通用函数
 76:param idx1_ubyte_file: idx1文件路径
 77:return: 数据集
 78"""
 79# 读取二进制数据
 80bin_data = open(idx1_ubyte_file, 'rb').read()
 81# 解析文件头信息，依次为魔数和标签数
 82offset = 0
 83fmt_header = '>ii'
 84magic_number, num_images = struct.unpack_from(fmt_header, bin_data, offset)
 85print '魔数:%d, 图片数量: %d张' % (magic_number, num_images)
 86# 解析数据集
 87offset += struct.calcsize(fmt_header)
 88fmt_image = '>B'
 89labels = np.empty(num_images)
 90for i in range(num_images):
 91if (i + 1) % 10000 == 0:
 92    print '已解析 %d' % (i + 1) + '张'
 93labels[i] = struct.unpack_from(fmt_image, bin_data, offset)[0]
 94offset += struct.calcsize(fmt_image)
 95return labels
 96def load_train_images(idx_ubyte_file=train_images_idx3_ubyte_file):
 97"""
 98TRAINING SET IMAGE FILE (train-images-idx3-ubyte):
 99[offset] [type]          [value]          [description]
1000000     32 bit integer  0x00000803(2051) magic number
1010004     32 bit integer  60000            number of images
1020008     32 bit integer  28               number of rows
103 0012     32 bit integer  28               number of columns
104 0016     unsigned byte   ??               pixel
105 0017     unsigned byte   ??               pixel
106 ........
107 xxxx     unsigned byte   ??               pixel
108 Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black).
109 :param idx_ubyte_file: idx文件路径
110 :return: n*row*col维np.array对象，n为图片数量
111 """
112 return decode_idx3_ubyte(idx_ubyte_file)
113 def load_train_labels(idx_ubyte_file=train_labels_idx1_ubyte_file):
114 """
115 TRAINING SET LABEL FILE (train-labels-idx1-ubyte):
116 [offset] [type]          [value]          [description]
117 0000     32 bit integer  0x00000801(2049) magic number (MSB first)
118 0004     32 bit integer  60000            number of items
119 0008     unsigned byte   ??               label
120 0009     unsigned byte   ??               label
121 ........
122 xxxx     unsigned byte   ??               label
123 The labels values are 0 to 9.
124 :param idx_ubyte_file: idx文件路径
125 :return: n*1维np.array对象，n为图片数量
126 """
127 return decode_idx1_ubyte(idx_ubyte_file)
128 def load_test_images(idx_ubyte_file=test_images_idx3_ubyte_file):
129 """
130 TEST SET IMAGE FILE (t10k-images-idx3-ubyte):
131 [offset] [type]          [value]          [description]
132 0000     32 bit integer  0x00000803(2051) magic number
133 0004     32 bit integer  10000            number of images
134 0008     32 bit integer  28               number of rows
135 0012     32 bit integer  28               number of columns
136 0016     unsigned byte   ??               pixel
137 0017     unsigned byte   ??               pixel
138 ........
139 xxxx     unsigned byte   ??               pixel
140 Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black).
141 :param idx_ubyte_file: idx文件路径
142 :return: n*row*col维np.array对象，n为图片数量
143 """
144 return decode_idx3_ubyte(idx_ubyte_file)
145 def load_test_labels(idx_ubyte_file=test_labels_idx1_ubyte_file):
146 """
147 TEST SET LABEL FILE (t10k-labels-idx1-ubyte):
148 [offset] [type]          [value]          [description]
149 0000     32 bit integer  0x00000801(2049) magic number (MSB first)
150 0004     32 bit integer  10000            number of items
151 0008     unsigned byte   ??               label
152 0009     unsigned byte   ??               label
153 ........
154 xxxx     unsigned byte   ??               label
155 The labels values are 0 to 9.
156 :param idx_ubyte_file: idx文件路径
157 :return: n*1维np.array对象，n为图片数量
158 """
159 return decode_idx1_ubyte(idx_ubyte_file)
160 def run():
161 train_images = load_train_images()
162 train_labels = load_train_labels()
163 # test_images = load_test_images()
164 # test_labels = load_test_labels()
165 # 查看前十个数据及其标签以读取是否正确
166for i in range(10):
167print train_labels[i]
168plt.imshow(train_images[i], cmap='gray')
169plt.show()
170print 'done'
171if __name__ == '__main__':
172run()