本章的主要内容
The need for transfer learning
Building Convolutional Neural Network (CNN) models from scratch:
Building a basic CNN model
Improving our CNN model with regularization
Improving our CNN model with image augmentation
Leveraging transfer learning with pretrained CNN models:
Using a pretrained model as a feature extractor
Improving our pretrained model with image augmentation
Improving our pretrained model with fine-tuning
Model performance evaluation
The code for this chapter will be available in the Chapter 5 folder in the GitHub repository at https://github.com/dipanjanS/handson-transfer-learning-with-python which you can refer to as needed to follow along with the chapter.
The need for transfer learning
需要注意的是迁移学习不总是应用在深度学习相关领域,并且它在深度学习之前很早就出现了。
我们的想法是:通常像专家那样使用图像分类的预训练模型,以减少数据样本少的约束情况来解决我们的问题。
Formulating our real-world problem
The dataset we will be using comes from the Dogs vs. Cats challenge (https://www.kaggle.com/c/dogs-vscats/data),whereas our primary objective is to build a model that can successfully recognize and categorize images into either a cat or a dog.
To start, download the train.zip file from the dataset page and store it in your local system. Once downloaded, unzip it into a folder. This folder will contain 25,000 images of dogs and cats; that is, 12,500 images per category
Building our dataset
Datasets Builder.ipynb:we have 3,000 images for training, 1,000 images for validation, and 1,000 images for our test dataset。
Cat datasets: (1500,) (500,) (500,)
Dog datasets: (1500,) (500,) (500,)
Test Images | Validation Images | Training Images |
---|---|---|
1000 | 1000 | 3000 |
Formulating our approach
We will start by building simple CNN models from scratch, then try to improve using techniques such as regularization and image augmentation. Then, we will try and leverage pretrained models to unleash the true power of transfer learning!
Building CNN models from scratch
1. from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout
2. from keras.models import Sequential
3. from keras import optimizers
4.
5. model = Sequential()
6.
7. model.add(Conv2D(16, kernel_size=(3, 3), activation='relu',
8. input_shape=input_shape))
9. model.add(MaxPooling2D(pool_size=(2, 2)))
10.
11. model.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))
12. model.add(MaxPooling2D(pool_size=(2, 2)))
13.
14. model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))
15. model.add(MaxPooling2D(pool_size=(2, 2)))
16.
17. model.add(Flatten())
18. model.add(Dense(512, activation='relu'))
19. model.add(Dense(1, activation='sigmoid'))
20.
21.
22. model.compile(loss='binary_crossentropy',
23. optimizer=optimizers.RMSprop(),
24. metrics=['accuracy'])
25.
26. model.summary()
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_1 (Conv2D) (None, 148, 148, 16) 448
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 74, 74, 16) 0
_________________________________________________________________
conv2d_2 (Conv2D) (None, 72, 72, 64) 9280
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 36, 36, 64) 0
_________________________________________________________________
conv2d_3 (Conv2D) (None, 34, 34, 128) 73856
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 17, 17, 128) 0
_________________________________________________________________
flatten_1 (Flatten) (None, 36992) 0
_________________________________________________________________
dense_1 (Dense) (None, 512) 18940416
_________________________________________________________________
dense_2 (Dense) (None, 1) 513
=================================================================
Total params: 19,024,513
Trainable params: 19,024,513
Non-trainable params: 0
_________________________________________________________________
发现过拟合,使用了dropout
1. model = Sequential()
2. # convolutional and pooling layers
3. model.add(Conv2D(16, kernel_size=(3, 3), activation='relu',
4. input_shape=input_shape))
5. model.add(MaxPooling2D(pool_size=(2, 2)))
6. model.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))
7. model.add(MaxPooling2D(pool_size=(2, 2)))
8. model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))
9. model.add(MaxPooling2D(pool_size=(2, 2)))
10. model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))
11. model.add(MaxPooling2D(pool_size=(2, 2)))
12. model.add(Flatten())
13. model.add(Dense(512, activation='relu'))
14. model.add(Dropout(0.3))
15. model.add(Dense(512, activation='relu'))
16. model.add(Dropout(0.3))
17. model.add(Dense(1, activation='sigmoid'))
18. model.compile(loss='binary_crossentropy',
19. optimizer=optimizers.RMSprop(),
20. metrics=['accuracy'])
21. model.compile(loss='binary_crossentropy',
22. optimizer=optimizers.RMSprop(),
23. metrics=['accuracy'])
24.
25. model.summary()
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_4 (Conv2D) (None, 148, 148, 16) 448
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 74, 74, 16) 0
_________________________________________________________________
conv2d_5 (Conv2D) (None, 72, 72, 64) 9280
_________________________________________________________________
max_pooling2d_5 (MaxPooling2 (None, 36, 36, 64) 0
_________________________________________________________________
conv2d_6 (Conv2D) (None, 34, 34, 128) 73856
_________________________________________________________________
max_pooling2d_6 (MaxPooling2 (None, 17, 17, 128) 0
_________________________________________________________________
conv2d_7 (Conv2D) (None, 15, 15, 128) 147584
_________________________________________________________________
max_pooling2d_7 (MaxPooling2 (None, 7, 7, 128) 0
_________________________________________________________________
flatten_2 (Flatten) (None, 6272) 0
_________________________________________________________________
dense_3 (Dense) (None, 512) 3211776
_________________________________________________________________
dropout_1 (Dropout) (None, 512) 0
_________________________________________________________________
dense_4 (Dense) (None, 512) 262656
_________________________________________________________________
dropout_2 (Dropout) (None, 512) 0
_________________________________________________________________
dense_5 (Dense) (None, 1) 513
=================================================================
Total params: 3,706,113
Trainable params: 3,706,113
Non-trainable params: 0
_________________________________________________________________
稍微有所改善,validation ACC 在78%左右。
模型过拟合的一个重要原因是少量的数据使得每次epoch的训练中看到的是相同的训练样本,所以需要用到data augmentation。
train_datagen = ImageDataGenerator(rescale=1./255, zoom_range=0.3, rotation_range=50,
width_shift_range=0.2, height_shift_range=0.2, shear_range=0.2,
horizontal_flip=True, fill_mode='nearest')
val_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow(train_imgs, train_labels_enc, batch_size=30)
val_generator = val_datagen.flow(validation_imgs, validation_labels_enc, batch_size=20)
Leveraging transfer learning with pretrained CNN models
Pretrained models are used in the following two popular ways when building new models or reusing them:
Using a pretrained model as a feature extractor
Fine-tuning the pretrained model
Understanding the VGG-16 model
For the last model, we will apply fine-tuning to the VGG model, where we will unfreeze the last two blocks (Block 4 and Block 5) so that their weights get updated in each epoch (per batch of data) as we train our own model.
Pretrained CNN model as a feature extractor
可以看到我们已经删除了与该分类器相关的VGG-16模型分类器的最后部分,因次我们将建立自己的分类器并利用VGG作为特征提取器。
from keras.applications import vgg16
from keras.models import Model
import keras
vgg = vgg16.VGG16(include_top=False, weights='imagenet',
input_shape=input_shape)
output = vgg.layers[-1].output
output = keras.layers.Flatten()(output)
vgg_model = Model(vgg.input, output)
vgg_model.trainable = False
for layer in vgg_model.layers:
layer.trainable = False
vgg_model.summary()
这里需要注意的是,vgg = vgg16.VGG16(include_top=False, weights=‘imagenet’,input_shape=input_shape) 会自动去github download vgg的模型文件vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5,由于文件较大,受网速和墙的因素可能会中断,因此可以提前将文件下载好放到指定目录,这里特别指出需要放置的路径:
- 如果是Linux环境
cd ~/.keras/models - 如果是windows环境
C:\Users\XXX\.keras\models\
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_2 (InputLayer) (None, 150, 150, 3) 0
_________________________________________________________________
block1_conv1 (Conv2D) (None, 150, 150, 64) 1792
_________________________________________________________________
block1_conv2 (Conv2D) (None, 150, 150, 64) 36928
_________________________________________________________________
block1_pool (MaxPooling2D) (None, 75, 75, 64) 0
_________________________________________________________________
block2_conv1 (Conv2D) (None, 75, 75, 128) 73856
_________________________________________________________________
block2_conv2 (Conv2D) (None, 75, 75, 128) 147584
_________________________________________________________________
block2_pool (MaxPooling2D) (None, 37, 37, 128) 0
_________________________________________________________________
block3_conv1 (Conv2D) (None, 37, 37, 256) 295168
_________________________________________________________________
block3_conv2 (Conv2D) (None, 37, 37, 256) 590080
_________________________________________________________________
block3_conv3 (Conv2D) (None, 37, 37, 256) 590080
_________________________________________________________________
block3_pool (MaxPooling2D) (None, 18, 18, 256) 0
_________________________________________________________________
block4_conv1 (Conv2D) (None, 18, 18, 512) 1180160
_________________________________________________________________
block4_conv2 (Conv2D) (None, 18, 18, 512) 2359808
_________________________________________________________________
block4_conv3 (Conv2D) (None, 18, 18, 512) 2359808
_________________________________________________________________
block4_pool (MaxPooling2D) (None, 9, 9, 512) 0
_________________________________________________________________
block5_conv1 (Conv2D) (None, 9, 9, 512) 2359808
_________________________________________________________________
block5_conv2 (Conv2D) (None, 9, 9, 512) 2359808
_________________________________________________________________
block5_conv3 (Conv2D) (None, 9, 9, 512) 2359808
_________________________________________________________________
block5_pool (MaxPooling2D) (None, 4, 4, 512) 0
_________________________________________________________________
flatten_1 (Flatten) (None, 8192) 0
=================================================================
Total params: 14,714,688
Trainable params: 0
Non-trainable params: 14,714,688
_________________________________________________________________
为了确认VGG-16各层是否真正被冻结,使用如下代码:
import pandas as pd
pd.set_option('max_colwidth', -1)
layers = [(layer, layer.name, layer.trainable) for layer in vgg_model.layers]
pd.DataFrame(layers, columns=['Layer Type', 'Layer Name', 'Layer Trainable'])
下面先提取bottleneck层的特征。
def get_bottleneck_features(model, input_imgs):
features = model.predict(input_imgs, verbose=0)
return features
train_features_vgg = get_bottleneck_features(vgg_model, train_imgs_scaled)
validation_features_vgg = get_bottleneck_features(vgg_model, validation_imgs_scaled)
print('Train Bottleneck Features:', train_features_vgg.shape,
'\tValidation Bottleneck Features:', validation_features_vgg.shape)
Train Bottleneck Features: (3000, 8192) Validation Bottleneck Features: (1000, 8192)
开始构建模型。
1. from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout, InputLayer
2. from keras.models import Sequential
3. from keras import optimizers
4.
5. input_shape = vgg_model.output_shape[1]
6.
7. model = Sequential()
8. model.add(InputLayer(input_shape=(input_shape,)))
9. model.add(Dense(512, activation='relu', input_dim=input_shape))
10. model.add(Dropout(0.3))
11. model.add(Dense(512, activation='relu'))
12. model.add(Dropout(0.3))
13. model.add(Dense(1, activation='sigmoid'))
14.
15. model.compile(loss='binary_crossentropy',
16. optimizer=optimizers.RMSprop(lr=1e-4),
17. metrics=['accuracy'])
18.
19. model.summary()
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense_1 (Dense) (None, 512) 4194816
_________________________________________________________________
dropout_1 (Dropout) (None, 512) 0
_________________________________________________________________
dense_2 (Dense) (None, 512) 262656
_________________________________________________________________
dropout_2 (Dropout) (None, 512) 0
_________________________________________________________________
dense_3 (Dense) (None, 1) 513
=================================================================
Total params: 4,457,985
Trainable params: 4,457,985
Non-trainable params: 0
开始模型训练
1. history = model.fit(x=train_features_vgg, y=train_labels_enc,
2. validation_data=(validation_features_vgg, validation_labels_enc),
3. batch_size=batch_size,
4. epochs=epochs,
5. verbose=1)
Pretrained CNN model as a feature extractor with image augmentation
1. train_datagen = ImageDataGenerator(rescale=1./255, zoom_range=0.3, rotation_range=50, width_shift_range=0.2, height_shift_range=0.2, shear_range=0.2, horizontal_flip=True, fill_mode='nearest')
2.
3. val_datagen = ImageDataGenerator(rescale=1./255)
4.
5. train_generator = train_datagen.flow(train_imgs, train_labels_enc, batch_size=30)
6. val_generator = val_datagen.flow(validation_imgs, validation_labels_enc, batch_size=20)
这次没有像上次那样提取瓶颈特征,因为我们将对数据生成器进行训练;因此,我们将vgg_model对象作为输入传递到我们自己的模型.
1. from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout, InputLayer
2. from keras.models import Sequential
3. from keras import optimizers
4.
5. model = Sequential()
6. model.add(vgg_model)
7. model.add(Dense(512, activation='relu', input_dim=input_shape))
8. model.add(Dropout(0.3))
9. model.add(Dense(512, activation='relu'))
10. model.add(Dropout(0.3))
11. model.add(Dense(1, activation='sigmoid'))
12.
13. model.compile(loss='binary_crossentropy',
14. optimizer=optimizers.RMSprop(lr=2e-5),
15. metrics=['accuracy'])
16.
17. model.summary()
Pretrained CNN model with finetuning and image augmentation
冻结前三层,4-5层解冻
1. vgg_model.trainable = True
2.
3. set_trainable = False
4. for layer in vgg_model.layers:
5. if layer.name in ['block5_conv1', 'block4_conv1']:
6. set_trainable = True
7. if set_trainable:
8. layer.trainable = True
9. else:
10. layer.trainable = False
11.
12. print("Trainable layers:", vgg_model.trainable_weights)
1. layers = [(layer, layer.name, layer.trainable) for layer in vgg_model.layers]
2. pd.DataFrame(layers, columns=['Layer Type', 'Layer Name', 'Layer Trainable'])
1. train_datagen = ImageDataGenerator(rescale=1./255, zoom_range=0.3, rotation_range=50, width_shift_range=0.2, height_shift_range=0.2, shear_range=0.2,
2. horizontal_flip=True, fill_mode='nearest')
3.
4. val_datagen = ImageDataGenerator(rescale=1./255)
5.
6. train_generator = train_datagen.flow(train_imgs, train_labels_enc, batch_size=30)
7. val_generator = val_datagen.flow(validation_imgs, validation_labels_enc, batch_size=20)
8.
9. from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout, InputLayer
10. from keras.models import Sequential
11. from keras import optimizers
12.
13. model = Sequential()
14. model.add(vgg_model)
15. model.add(Dense(512, activation='relu', input_dim=input_shape))
16. model.add(Dropout(0.3))
17. model.add(Dense(512, activation='relu'))
18. model.add(Dropout(0.3))
19. model.add(Dense(1, activation='sigmoid'))
20.
21. model.compile(loss='binary_crossentropy',
22. optimizer=optimizers.RMSprop(lr=1e-5),
23. metrics=['accuracy'])
24.
25. model.summary()
Evaluating our deep learning models
Model predictions on a sample test image(单个样本的测试)
取了其中一张图片,使用了前面5个模型:
Basic CNN
CNN with Img Augmentation
Pre-trained CNN (Transfer Learning)
Pre-trained CNN with Img Augmentation (Transfer Learning)
Pre-trained CNN with Fine-tuning & Img Augmentation (Transfer Learning)
3对2错
Evaluation model performance on test data
1. basic_cnn = load_model('cats_dogs_basic_cnn.h5')
2. img_aug_cnn = load_model('cats_dogs_cnn_img_aug.h5')
3. tl_cnn = load_model('cats_dogs_tlearn_basic_cnn.h5')
4. tl_img_aug_cnn = load_model('cats_dogs_tlearn_img_aug_cnn.h5')
5. tl_img_aug_finetune_cnn = load_model('cats_dogs_tlearn_finetune_img_aug_cnn.h5')
6. predictions = basic_cnn.predict_classes(test_imgs_scaled, verbose=0)
7. predictions = num2class_label_transformer(predictions)
8.
9. meu.display_model_performance_metrics(true_labels=test_labels, predicted_labels=predictions,
10. classes=list(set(test_labels)))
11.
12. predictions = img_aug_cnn.predict_classes(test_imgs_scaled, verbose=0)
13. predictions = num2class_label_transformer(predictions)
14.
15. meu.display_model_performance_metrics(true_labels=test_labels, predicted_labels=predictions,
16. classes=list(set(test_labels)))
17.
18. test_bottleneck_features = get_bottleneck_features(vgg_model, test_imgs_scaled)
19.
20. predictions = tl_cnn.predict_classes(test_bottleneck_features, verbose=0)
21. predictions = num2class_label_transformer(predictions)
22.
23. meu.display_model_performance_metrics(true_labels=test_labels, predicted_labels=predictions,
24. classes=list(set(test_labels)))
25.
26. predictions = tl_img_aug_cnn.predict_classes(test_imgs_scaled, verbose=0)
27. predictions = num2class_label_transformer(predictions)
28.
29. meu.display_model_performance_metrics(true_labels=test_labels, predicted_labels=predictions, classes=list(set(test_labels)))
30.
31. predictions = tl_img_aug_finetune_cnn.predict_classes(test_imgs_scaled, verbose=0)
32. predictions = num2class_label_transformer(predictions)
33.
34. meu.display_model_performance_metrics(true_labels=test_labels, predicted_labels=predictions,
35. classes=list(set(test_labels)))
最差的一个模型和最好的一个模型,分别是basic_cnn和Pre-trained CNN with Fine-tuning & Img Augmentation。