- fit和fit_generator的区别
首先Keras中的fit()函数传入的x_train和y_train是被完整的加载进内存的,当然用起来很方便,但是如果我们数据量很大,那么是不可能将所有数据载入内存的,必将导致内存泄漏,这时候我们可以用fit_generator函数来进行训练。
下面是fit传参的例子:
history = model.fit(x_train, y_train, epochs=10,batch_size=32,
validation_split=0.2)
这里需要给出epochs和batch_size,epoch是这个数据集要被轮多少次,batch_size是指这个数据集被分成多少个batch进行处理。
最后可以给出交叉验证集的大小,这里的0.2是指在训练集上占比20%。
fit_generator函数必须传入一个生成器,我们的训练数据也是通过生成器产生的,下面给出一个简单的生成器函数:
batch_size = 128
def generator():
while 1:
row = np.random.randint(0,len(x_train),size=batch_size)
x = np.zeros((batch_size,x_train.shape[-1]))
y = np.zeros((batch_size,))
x = x_train[row]
y = y_train[row]
yield x,y
这里的生成器函数我产生的是一个batch_size为128大小的数据,这只是一个demo。如果我在生成器里没有规定batch_size的大小,就是每次产生一个数据,那么在用fit_generator时候里面的参数steps_per_epoch是不一样的。这里的坑我困惑了好久,虽然不是什么大问题。
下面是fit_generator函数的传参:
history = model.fit_generator(generator(),epochs=epochs,steps_per_epoch=len(x_train)//(batch_size*epochs))
2、batch_size和steps_per_epoch的区别
首先batch_size = 数据集大小/steps_per_epoch的,如果我们在生成函数里设置了batch_size的大小,那么在fit_generator传参的时候,,steps_per_epoch=len(x_train)//(batch_size*epochs)
我得完整demo代码:
from keras.datasets import imdb
from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras import layers
import numpy as np
import random
from sklearn.metrics import f1_score,accuracy_score
max_features = 10000
maxlen = 500
batch_size = 32
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)
x_train = pad_sequences(x_train,maxlen=maxlen)
x_test = pad_sequences(x_test,maxlen=maxlen)
def generator():
while 1:
row = np.random.randint(0,len(x_train),size=batch_size)
x = np.zeros((batch_size,x_train.shape[-1]))
y = np.zeros((batch_size,))
x = x_train[row]
y = y_train[row]
yield x,y
# generator()
model = Sequential()
model.add(layers.Embedding(max_features,32,input_length=maxlen))
model.add(layers.GRU(64,return_sequences=True))
model.add(layers.GRU(32))
# model.add(layers.Flatten())
# model.add(layers.Dense(32,activation='relu'))
model.add(layers.Dense(1,activation='sigmoid'))
model.compile(optimizer='rmsprop',loss='binary_crossentropy',metrics=['acc'])
print(model.summary())
# history = model.fit(x_train, y_train, epochs=1,batch_size=32, validation_split=0.2)
history = model.fit_generator(generator(),epochs=1,steps_per_epoch=len(x_train)//(batch_size))
print(model.evaluate(x_test,y_test))
y = model.predict_classes(x_test)
print(accuracy_score(y_test,y))