神经网络训练数据准备---数据标签在numpy和tensor之间的转换及处理---Pytorch实现

目录

1.从txt文档中读取训练数据和测试数据

2.从txt文件中读取标签数据

3.将训练数据和测试数据转化为tensor形式，为后面训练做准备，因为训练数据在pytorch要求一般为tensor格式

4.将标签转化为float型(此处是因为做的是回归任务)，如果想要做分类任务的话需要将标签转化为long型！

5.将已经转化为tensor格式的(train_datas, train_labels)和(test_datas, test_labels)转化为Data.DataLoader可以使用的格式

6.利用Data.DataLoader来对数据集进行迭代

7.训练流程

---

1.从txt文档中读取训练数据和测试数据

fp = open('E:\\桌面\\test_new_3\\data\\train.txt', 'r', encoding='utf-8')
string = fp.read()  # string是一行字符串，该字符串包含文件所有内容
fp.close()
row_list = string.splitlines()  # splitlines默认参数是‘\n’
data_list = [[float(i) for i in row.strip().split(" ")] for row in row_list]
shuffle(data_list) ##打乱数据
data_train = np.array(data_list)     # data=data.to_numpy()
fp = open('E:\\桌面\\test_new_3\\data\\test.txt', 'r', encoding='utf-8')
string = fp.read()  # string是一行字符串，该字符串包含文件所有内容
fp.close()
row_list = string.splitlines()  # splitlines默认参数是‘\n’
data_list = [[float(i) for i in row.strip().split(" ")] for row in row_list]
shuffle(data_list) ##打乱数据
data_test = np.array(data_list)     # data=data.to_numpy()

​

2.从txt文件中读取标签数据

fp = open('E:\\桌面\\test_new_3\\label\\1004.txt', 'r', encoding='utf-8')
line = fp.readline()
l = line.split(",")
label1 = [[float(i)] for i in l]
label1 = np.array(label1)
fp = open('E:\\桌面\\test_new_3\\label\\355.txt', 'r', encoding='utf-8')
line = fp.readline()
l = line.split(",")
label2 = [[float(i)] for i in l]
label2 = np.array(label2)
# label1 = np.loadtxt('./284.txt', dtype=np.float32, delimiter=' ')
# print("原始红外矩阵大小为：", label1)
# label2 = np.loadtxt('./71.txt', dtype=np.float32, delimiter=' ')
# print("原始红外矩阵大小为：", label2)
train_datas = data_train
train_labels = label1
test_datas = data_test
test_labels = label2

3.将训练数据和测试数据转化为tensor形式，为后面训练做准备，因为训练数据在pytorch要求一般为tensor格式

##这几步的作用就是将数据转化为tensor的格式，为后面训练做准备（因为训练的时候数据要是tensor格式）
##同时，train_datas = train_datas / 1.0 ，这里的作用就是将其强制转化为浮点类型
train_datas = train_datas / 1.0
train_datas = torch.from_numpy(train_datas).float()
test_datas = test_datas / 1.0
test_datas = torch.from_numpy(test_datas).float()

4.将标签转化为float型(此处是因为做的是回归任务)，如果想要做分类任务的话需要将标签转化为long型！

train_labels = torch.from_numpy(train_labels).float()
test_labels = torch.from_numpy(test_labels).float()

5.将已经转化为tensor格式的(train_datas, train_labels)和(test_datas, test_labels)转化为Data.DataLoader可以使用的格式

torch_dataset_train = Data.TensorDataset(train_datas, train_labels)
torch_dataset_test = Data.TensorDataset(test_datas, test_labels)
print(train_datas.size())                 # (60000, 28, 28)
print(train_labels.size())
print(test_datas.size())
print(test_labels.size())

6.利用Data.DataLoader来对数据集进行迭代

train_loader = Data.DataLoader(dataset=torch_dataset_train, batch_size=BATCH_SIZE, shuffle=True)

7.训练流程

a = []
for epoch in range(EPOCH):
    for step, (b_x, b_y) in enumerate(train_loader):   # train_loader的用法！
        # print(b_x.size())
        b_x = b_x.view(-1, 3)
        # print(b_x.size())
        output = mlp(b_x)            # logistic output
        loss = loss_func(output, b_y)   # cross entropy loss
        optimizer.zero_grad()           # clear gradients for this training step
        loss.backward()                 # backpropagation, compute gradients
        optimizer.step()                # apply gradients
​
        if step % 50 == 0:
            test_output = mlp(test_x.view(-1, 3))
            accuracy = (((abs(test_output.detach().numpy()-test_y.numpy()) < 0.1)).sum())/len(test_y.numpy())
            print('Epoch: ', round(epoch), '| train loss: %.4f' % loss.data.numpy(), '| test accuracy:%.2f' % accuracy)
            a.append(accuracy)
​

End···