python有什么库可以 从fig文件中读取数据_python读取mnist文件

从 http://yann.lecun.com/exdb/mnist/ 可以下载原始的文件。

train-images-idx3-ubyte.gz: training set images (9912422 bytes)

train-labels-idx1-ubyte.gz:

training set labels (28881 bytes)

t10k-images-idx3-ubyte.gz:

test set images (1648877 bytes)

t10k-labels-idx1-ubyte.gz:

test set labels (4542 bytes)

The training set contains 60000 examples, and the test set 10000 examples.

The first 5000 examples of the test set are taken from the original

NIST training set. The last 5000 are taken from the original NIST test

set. The first 5000 are cleaner and easier than the last 5000.

TRAINING SET LABEL FILE (train-labels-idx1-ubyte):

[offset] [type]

[value] [description]

0000 32 bit integer 0x00000801(2049)

magic number (MSB first)

0004 32 bit integer 60000

number of items

0008 unsigned byte ??

label

0009 unsigned byte ??

label

........

xxxx unsigned byte ??

label

The labels values are 0 to 9.

TRAINING SET IMAGE FILE (train-images-idx3-ubyte):

[offset] [type]

[value] [description]

0000 32 bit integer 0x00000803(2051)

magic number

0004 32 bit integer 60000

number of images

0008 32 bit integer 28

number of rows

0012 32 bit integer 28

number of columns

0016 unsigned byte ??

pixel

0017 unsigned byte ??

pixel

........

xxxx unsigned byte ??

pixel

Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background

(white), 255 means foreground (black).

TEST SET LABEL FILE (t10k-labels-idx1-ubyte):

[offset] [type]

[value] [description]

0000 32 bit integer 0x00000801(2049)

magic number (MSB first)

0004 32 bit integer 10000

number of items

0008 unsigned byte ??

label

0009 unsigned byte ??

label

........

xxxx unsigned byte ??

label

The labels values are 0 to 9.

TEST SET IMAGE FILE (t10k-images-idx3-ubyte):

[offset] [type]

[value] [description]

0000 32 bit integer 0x00000803(2051)

magic number

0004 32 bit integer 10000

number of images

0008 32 bit integer 28

number of rows

0012 32 bit integer 28

number of columns

0016 unsigned byte ??

pixel

0017 unsigned byte ??

pixel

........

xxxx unsigned byte ??

pixel

Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background

(white), 255 means foreground (black).

THE IDX FILE FORMAT

the IDX file format is a simple format for vectors and multidimensional

matrices of various numerical types.

The basic format is

magic number

size in dimension 0

size in dimension 1

size in dimension 2

.....

size in dimension N

data

The magic number is an integer (MSB first). The first 2 bytes are always

0.

The third byte codes the type of the data:

0x08: unsigned byte

0x09: signed byte

0x0B: short (2 bytes)

0x0C: int (4 bytes)

0x0D: float (4 bytes)

0x0E: double (8 bytes)

The 4-th byte codes the number of dimensions of the vector/matrix: 1

for vectors, 2 for matrices....

The sizes in each dimension are 4-byte integers (MSB first, high endian,

like in most non-Intel processors).

The data is stored like in a C array, i.e. the index in the last dimension

changes the fastest.

python 读取 mnist 文件其实就是 python 怎么读取 binnary file。mnist 的结构如下，选取 train-images-idx3-ubyte

TRAINING SET IMAGE FILE (train-images-idx3-ubyte):

[offset] [type] [value] [description]

0000 32 bit integer 0x00000803(2051) magic number

0004 32 bit integer 60000 number of images

0008 32 bit integer 28 number of rows

0012 32 bit integer 28 number of columns

0016 unsigned byte ?? pixel

0017 unsigned byte ?? pixel

........

xxxx unsigned byte ?? pixel

也就是之前我们要读取4个 32 bit integer. 试过很多方法，觉得最方便的，至少对我来说还是使用 struct.unpack_from()

filename= 'train-images.idx3-ubyte'

binfile= open(filename ,'rb')

buf= binfile.read()

先使用二进制方式把文件都读进来

index= 0

magic, numImages , numRows , numColumns= struct.unpack_from('>IIII' , buf , index)

index+= struct.calcsize('>IIII')

然后使用struc.unpack_from

'>IIII'是说使用大端法读取4个unsinged int32

然后读取一个图片测试是否读取成功

im= struct.unpack_from('>784B' ,buf, index)

index+= struct.calcsize('>784B')

im= np.array(im)

im= im.reshape(28,28)

fig= plt.figure()

plotwindow= fig.add_subplot(111)

plt.imshow(im , cmap='gray')

plt.show()

'>784B'的意思就是用大端法读取784个unsigned byte

完整代码如下，读取其中第一个图像：

importnumpy as np #python 3.7importstructimportmatplotlib.pyplot as plt

filename= 'train-images.idx3-ubyte'binfile= open(filename, 'rb')

buf=binfile.read()

index=0

magic, numImages, numRows, numColumns= struct.unpack_from('>IIII', buf, index)

index+= struct.calcsize('>IIII')

im= struct.unpack_from('>784B', buf, index)

index+= struct.calcsize('>784B')

im=np.array(im)

im= im.reshape(28, 28)

fig=plt.figure()

plotwindow= fig.add_subplot(111)

plt.imshow(im, cmap='gray')

plt.show()###### https://www.cnblogs.com/x1957/archive/2012/06/02/2531503.html###

另外一个实例，读取全部图像：

## from https://www.jianshu.com/p/84f72791806f#encoding: utf-8

"""@author: monitor1379

@contact: yy4f5da2@hotmail.com

@site: www.monitor1379.com

@version: 1.0

@license: Apache Licence

@file: mnist_decoder.py

@time: 2016/8/16 20:03

对MNIST手写数字数据文件转换为bmp图片文件格式。

数据集下载地址为http://yann.lecun.com/exdb/mnist。

相关格式转换见官网以及代码注释。

========================

关于IDX文件格式的解析规则：

========================

THE IDX FILE FORMAT

the IDX file format is a simple format for vectors and multidimensional matrices of various numerical types.

The basic format is

magic number

size in dimension 0

size in dimension 1

size in dimension 2

.....

size in dimension N

data

The magic number is an integer (MSB first). The first 2 bytes are always 0.

The third byte codes the type of the data:

0x08: unsigned byte

0x09: signed byte

0x0B: short (2 bytes)

0x0C: int (4 bytes)

0x0D: float (4 bytes)

0x0E: double (8 bytes)

The 4-th byte codes the number of dimensions of the vector/matrix: 1 for vectors, 2 for matrices....

The sizes in each dimension are 4-byte integers (MSB first, high endian, like in most non-Intel processors).

The data is stored like in a C array, i.e. the index in the last dimension changes the fastest."""

importnumpy as npimportstructimportmatplotlib.pyplot as plt#训练集文件

train_images_idx3_ubyte_file = 'train-images.idx3-ubyte'

#训练集标签文件

train_labels_idx1_ubyte_file = 'train-labels.idx1-ubyte'

#测试集文件

test_images_idx3_ubyte_file = 't10k-images.idx3-ubyte'

#测试集标签文件

test_labels_idx1_ubyte_file = 't10k-labels.idx1-ubyte'

defdecode_idx3_ubyte(idx3_ubyte_file):"""解析idx3文件的通用函数

:param idx3_ubyte_file: idx3文件路径

:return: 数据集"""

#读取二进制数据

bin_data = open(idx3_ubyte_file, 'rb').read()#解析文件头信息，依次为魔数、图片数量、每张图片高、每张图片宽

offset =0

fmt_header= '>iiii'magic_number, num_images, num_rows, num_cols=struct.unpack_from(fmt_header, bin_data, offset)print('魔数:%d, 图片数量: %d张, 图片大小: %d*%d' %(magic_number, num_images, num_rows, num_cols))#解析数据集

image_size = num_rows *num_cols

offset+=struct.calcsize(fmt_header)

fmt_image= '>' + str(image_size) + 'B'images=np.empty((num_images, num_rows, num_cols))for i inrange(num_images):if (i + 1) % 10000 ==0:print('已解析 %d' % (i + 1) + '张')

images[i]=np.array(struct.unpack_from(fmt_image, bin_data, offset)).reshape((num_rows, num_cols))

offset+=struct.calcsize(fmt_image)returnimagesdefdecode_idx1_ubyte(idx1_ubyte_file):"""解析idx1文件的通用函数

:param idx1_ubyte_file: idx1文件路径

:return: 数据集"""

#读取二进制数据

bin_data = open(idx1_ubyte_file, 'rb').read()#解析文件头信息，依次为魔数和标签数

offset =0

fmt_header= '>ii'magic_number, num_images=struct.unpack_from(fmt_header, bin_data, offset)print('魔数:%d, 图片数量: %d张' %(magic_number, num_images))#解析数据集

offset +=struct.calcsize(fmt_header)

fmt_image= '>B'labels=np.empty(num_images)for i inrange(num_images):if (i + 1) % 10000 ==0:print('已解析 %d' % (i + 1) + '张')

labels[i]=struct.unpack_from(fmt_image, bin_data, offset)[0]

offset+=struct.calcsize(fmt_image)returnlabelsdef load_train_images(idx_ubyte_file=train_images_idx3_ubyte_file):"""TRAINING SET IMAGE FILE (train-images-idx3-ubyte):

[offset] [type] [value] [description]

0000 32 bit integer 0x00000803(2051) magic number

0004 32 bit integer 60000 number of images

0008 32 bit integer 28 number of rows

0012 32 bit integer 28 number of columns

0016 unsigned byte ?? pixel

0017 unsigned byte ?? pixel

........

xxxx unsigned byte ?? pixel

Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black).

:param idx_ubyte_file: idx文件路径

:return: n*row*col维np.array对象，n为图片数量"""

returndecode_idx3_ubyte(idx_ubyte_file)def load_train_labels(idx_ubyte_file=train_labels_idx1_ubyte_file):"""TRAINING SET LABEL FILE (train-labels-idx1-ubyte):

[offset] [type] [value] [description]

0000 32 bit integer 0x00000801(2049) magic number (MSB first)

0004 32 bit integer 60000 number of items

0008 unsigned byte ?? label

0009 unsigned byte ?? label

........

xxxx unsigned byte ?? label

The labels values are 0 to 9.

:param idx_ubyte_file: idx文件路径

:return: n*1维np.array对象，n为图片数量"""

returndecode_idx1_ubyte(idx_ubyte_file)def load_test_images(idx_ubyte_file=test_images_idx3_ubyte_file):"""TEST SET IMAGE FILE (t10k-images-idx3-ubyte):

[offset] [type] [value] [description]

0000 32 bit integer 0x00000803(2051) magic number

0004 32 bit integer 10000 number of images

0008 32 bit integer 28 number of rows

0012 32 bit integer 28 number of columns

0016 unsigned byte ?? pixel

0017 unsigned byte ?? pixel

........

xxxx unsigned byte ?? pixel

Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black).

:param idx_ubyte_file: idx文件路径

:return: n*row*col维np.array对象，n为图片数量"""

returndecode_idx3_ubyte(idx_ubyte_file)def load_test_labels(idx_ubyte_file=test_labels_idx1_ubyte_file):"""TEST SET LABEL FILE (t10k-labels-idx1-ubyte):

[offset] [type] [value] [description]

0000 32 bit integer 0x00000801(2049) magic number (MSB first)

0004 32 bit integer 10000 number of items

0008 unsigned byte ?? label

0009 unsigned byte ?? label

........

xxxx unsigned byte ?? label

The labels values are 0 to 9.

:param idx_ubyte_file: idx文件路径

:return: n*1维np.array对象，n为图片数量"""

returndecode_idx1_ubyte(idx_ubyte_file)defrun():

train_images=load_train_images()

train_labels=load_train_labels()#test_images = load_test_images()

#test_labels = load_test_labels()

#查看前十个数据及其标签以读取是否正确

for i in range(3):print(train_labels[i])

plt.imshow(train_images[i], cmap='gray')

plt.show()print('done')if __name__ == '__main__':

run()

另外一个实例：

## https://www.e-learn.cn/content/wangluowenzhang/615391

importosimportnumpy as npimportmatplotlib.pyplot as pltdefload_data(data_path):'''函数功能：导出MNIST数据

输入: data_path 传入数据所在路径（解压后的数据）

输出: train_data 输出data

train_label 输出label'''f_data= open(os.path.join(data_path, 'train-images.idx3-ubyte'))

loaded_data= np.fromfile(file=f_data, dtype=np.uint8)#前16个字符为说明符，需要跳过

train_data = loaded_data[16:].reshape((-1, 784)).astype(np.float)

f_label= open(os.path.join(data_path, 'train-labels.idx1-ubyte'))

loaded_label= np.fromfile(file=f_label, dtype=np.uint8)#前8个字符为说明符，需要跳过

train_label = loaded_label[8:].reshape((-1)).astype(np.float)returntrain_data, train_labelif __name__ == '__main__':

train_data, train_label= load_data('./') ## path of files

#把下载好的minst数据集 放在xxxxxxx/minst文件夹里面；填入路径即可

print(np.shape(train_data))#(60000, 784)

print(np.shape(train_label))#(60000,)

for i in range(5):

img= train_data[i].reshape(28, 28)#变成二维图片

plt.imshow(img)

plt.show()print(train_label[i])#输出了前五个图片 及其标签

From:

REF：