Tips for DNN

一、原始的网络

读取数据时数据已归一化

import numpy as np
import tensorflow as tf
from tensorflow import keras
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD,Adam

from keras.datasets import mnist
from keras.utils import np_utils

def load_data():
    (x_train,y_train),(x_test,y_test)=mnist.load_data()
    number=10000
    #将图片压成1维的
    x_train = x_train[0:number]
    y_train = y_train[0:number]
    x_train = x_train.reshape(number,28 * 28).astype(np.float32)
    y_train=np_utils.to_categorical(y_train,10)

    x_test = x_test[0:number]
    y_test = y_test[0:number]
    x_test = x_test.reshape(number, 28 * 28).astype(np.float32)
    y_test = np_utils.to_categorical(y_test, 10)

    # 将像素归一化
    x_train=x_train/255
    x_test=x_test/255

    return (x_train,y_train),(x_test,y_test)

(x_train,y_train),(x_test,y_test)=load_data()

model=Sequential()
model.add(Dense(input_dim=28*28,units=633,activation='sigmoid'))
model.add(Dense(units=633,activation='sigmoid'))
model.add(Dense(units=633,activation='sigmoid'))
model.add(Dense(units=10,activation='softmax'))

model.compile(loss='mse',optimizer=SGD(lr=0.1),metrics=['accuracy'])

model.fit(x_train,y_train,batch_size=100,epochs=20)

result=model.evaluate(x_test,y_test)
print('Test Acc:',result[1])

正确率：

Train Acc: 0.132

Test Acc: 0.113

二、改进技巧

训练集上结果差，没有训练好：

1、修改loss function为 cross entropy

Train Acc: 0.849

Test Acc: 0.840

 

2、batch size 100改为10000，改大后会从SGD不断靠近GD，陷入局部最优解，性能变差，GPU会加速

Train Acc: 0.107

Test Acc: 0.113

 

3、batch size 100改为1，改小后GPU加速能力减弱，为1时相当于不加速

最终结果和100近似，但训练时很慢（前提是用GPU）

 

4改变网络结构：：再加10个layers,train不起来

原因？？？好像是sigmoid影响

 

5、修改activate function 。sigmoid改为relu

+10层：0.998、0.953

不加10层:

6 、数据没有归一化

train不起来

7、Adam

收敛速度变快

8、testing set上加噪声，训练集和测试集分布不一致，

Train Acc:1.0

Test Acc:0.494

 

训练集好，测试集差，（过拟合）dropout

Train Acc:0.98

Test Acc:0.61

训练集正确率下降，测试集上升