机器学习参数设置与预训练模型设置

使用tensorlayer时,出现了大量相关的参数设置,通用的参数设置如下:

task = 'dcgan'
flags = tf.app.flags
flags.DEFINE_string('task','dcgan','this task name')
flags.DEFINE_integer("epoch", 200, "Epoch to train [100]")
flags.DEFINE_float("learning_rate", 0.0002, "Learning rate of for adam [0.0002]")
flags.DEFINE_float("beta1", 0.5, "Momentum term of adam [0.5]")
flags.DEFINE_float("weight_decay", 1e-5, "Weight decay for l2 loss")
flags.DEFINE_float("pool_size", 50, 'size of image buffer that stores previously generated images, default: 50')
flags.DEFINE_integer("train_size", 3000, "The size of train images [np.inf]")
flags.DEFINE_integer("batch_size", 1, "The number of batch images [1] if we use InstanceNormLayer !")
flags.DEFINE_integer("image_size", 256, "The size of image to use (will be center cropped) [256]")
flags.DEFINE_integer("gf_dim", 32, "Size of generator filters in first layer")
flags.DEFINE_integer("df_dim", 64, "Size of discriminator filters in first layer")
# flags.DEFINE_integer("class_embedding_size", 5, "Size of class embedding")
flags.DEFINE_integer("output_size", 256, "The size of the output images to produce [64]")
flags.DEFINE_integer("sample_size", 64, "The number of sample images [64]")
flags.DEFINE_integer("c_dim", 3, "Dimension of image color. [3]")
flags.DEFINE_integer("sample_step", 500, "The interval of generating sample. [500]")
flags.DEFINE_integer("save_step", 200, "The interval of saveing checkpoints. [200]")
flags.DEFINE_string("dataset_dir", "spring2snow", "The name of dataset [horse2zebra, apple2orange, sunflower2daisy and etc]")
flags.DEFINE_string("checkpoint_dir", "/home/liuwenjie/deep_save/{}/ckpt".format(task), "Directory name to save the checkpoints [checkpoint]")
flags.DEFINE_string("sample_dir", '/home/liuwenjie/deep_save/{}/samples'.format(task), "Directory name to save the image samples [samples]")
flags.DEFINE_string("direction", "forward", "The direction of generator [forward, backward]")
flags.DEFINE_string("test_dir", "/home/liuwenjie/deep_save/{}/test".format(task), "The direction of test")
flags.DEFINE_boolean("is_train", True, "True for training, False for testing [False]")
flags.DEFINE_boolean("is_crop", False, "True for training, False for testing [False]")
# flags.DEFINE_boolean("visualize", False, "True for visualizing, False for nothing [False]")
FLAGS = flags.FLAGS

通过一个网络的学习,我发现大多数网络都需要进行以上的定义,以下一一解读:

1. task:(dcgan_spring2winter_18.3.1)设定任务名称,一般设计成为 {model name}-{data name}-{time},这样每次训练都可以获得唯一确定的ckpt和image,保证多任务不重复

2. n_ epoch:(200)设定task整批数据集运行的轮次

3. learning_rate :(0.002)一般起始的learning rate 设定为0.01,在神经网络的loss停止不下降的时候,lr下降一个数量级

4. beta1 :(0.5)设定在0.9或0.5左右使用,主要是给Adam作为参数

5. weight_decay:(1e-5)下降权重,一般不设置

6. pool_size:(50)

7. train_size:(len<train_dataset)训练数据集,一般都小于训练集,训练时需要len(min(min(train_A,tran_label),train_size))防止错误

8. batch_size:(1-10)一次放入网络的图片数量

9. mf_dim:(64)网络的第一个卷积层使用的卷积核个数

10.output_size:(256)输出图片的数量

11.sample_size:(64)sample的图片的数量

12.c_dim:(3)图片的通道数目

13.sample_step:(500)运行多少步保存一次sample

14.save_step:(10)运行多少次进行一次step的保存工作

15.dataset_dir:(/home/deep_save/{}/datasetdir.format(task))文件存储位置

16.ckpt_dir:(/home/deep_save/{}/ckpt_dir.format(task))ckpt存储位置

17.sample_dir:(/home/deep_save/{}/sample_dir.format(task))sample存储的文件夹

18.direction::(forward,backword)正向传播,反向传播

19.test_dir:(./test)一般将test放在显眼的位置

20.is_train:train或test

21.is_crop:是否需要将输入剪裁

在使用机器学习的过程中,通常需要使用预训练模型

最基础的几个预训练模型都是使用ImageNet进行训练的resnet,cgg16,googleNet.无论是做分类,分割或者是做检测问题,都需要使用这三种的一个,不然都很难获得一个较为良好的训练结果.以tensorlayer中使用vgg16为例,演示一下如何使用tensorlayer逐步对每个层进行参数加载:

#! /usr/bin/python
# -*- coding: utf-8 -*-
"""
VGG-16 for ImageNet.
Introduction
----------------
VGG is a convolutional neural network model proposed by K. Simonyan and A. Zisserman
from the University of Oxford in the paper “Very Deep Convolutional Networks for
Large-Scale Image Recognition”  . The model achieves 92.7% top-5 test accuracy in ImageNet,
which is a dataset of over 14 million images belonging to 1000 classes.
Download Pre-trained Model
----------------------------
- Model weights in this example - vgg16_weights.npz : http://www.cs.toronto.edu/~frossard/post/vgg16/
- Caffe VGG 16 model : https://gist.github.com/ksimonyan/211839e770f7b538e2d8#file-readme-md
- Tool to convert the Caffe models to TensorFlow's : https://github.com/ethereon/caffe-tensorflow
Note
------
- For simplified CNN layer see "Convolutional layer (Simplified)"
in read the docs website.
- When feeding other images to the model be sure to properly resize or crop them
beforehand. Distorted images might end up being misclassified. One way of safely
feeding images of multiple sizes is by doing center cropping, as shown in the
following snippet:
>>> image_h, image_w, _ = np.shape(img)
>>> shorter_side = min(image_h, image_w)
>>> scale = 224. / shorter_side
>>> image_h, image_w = np.ceil([scale * image_h, scale * image_w]).astype('int32')
>>> img = imresize(img, (image_h, image_w))
>>> crop_x = (image_w - 224) / 2
>>> crop_y = (image_h - 224) / 2
>>> img = img[crop_y:crop_y+224,crop_x:crop_x+224,:]
"""

import os
import time

import numpy as np
from scipy.misc import imread, imresize

import tensorflow as tf
import tensorlayer as tl
from tensorlayer.layers import *

# try:
#     from tensorlayer.models.imagenet_classes import *
# except Exception as e:
#     raise Exception(
#         "{} / download the file from: https://github.com/zsdonghao/tensorlayer/tree/master/example/data".format(e)
#     )

def conv_layers(net_in):
    with tf.name_scope('preprocess'):
        # Notice that we include a preprocessing layer that takes the RGB image
        # with pixels values in the range of 0-255 and subtracts the mean image
        # values (calculated over the entire ImageNet training set).
        mean = tf.constant([123.68, 116.779, 103.939], dtype=tf.float32, shape=[1, 1, 1, 3], name='img_mean')
        net_in.outputs = net_in.outputs - mean

    # conv1
    net = Conv2dLayer(net_in, act=tf.nn.relu, shape=[3, 3, 3, 64], strides=[1, 1, 1, 1], padding='SAME', name='conv1_1')
    net = Conv2dLayer(net, act=tf.nn.relu, shape=[3, 3, 64, 64], strides=[1, 1, 1, 1], padding='SAME', name='conv1_2')
    net = PoolLayer(net, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', pool=tf.nn.max_pool, name='pool1')

    # conv2
    net = Conv2dLayer(net, act=tf.nn.relu, shape=[3, 3, 64, 128], strides=[1, 1, 1, 1], padding='SAME', name='conv2_1')
    net = Conv2dLayer(net, act=tf.nn.relu, shape=[3, 3, 128, 128], strides=[1, 1, 1, 1], padding='SAME', name='conv2_2')
    net = PoolLayer(net, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', pool=tf.nn.max_pool, name='pool2')

    # conv3
    net = Conv2dLayer(net, act=tf.nn.relu, shape=[3, 3, 128, 256], strides=[1, 1, 1, 1], padding='SAME', name='conv3_1')
    net = Conv2dLayer(net, act=tf.nn.relu, shape=[3, 3, 256, 256], strides=[1, 1, 1, 1], padding='SAME', name='conv3_2')
    net = Conv2dLayer(net, act=tf.nn.relu, shape=[3, 3, 256, 256], strides=[1, 1, 1, 1], padding='SAME', name='conv3_3')
    net = PoolLayer(net, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', pool=tf.nn.max_pool, name='pool3')

    # conv4
    net = Conv2dLayer(net, act=tf.nn.relu, shape=[3, 3, 256, 512], strides=[1, 1, 1, 1], padding='SAME', name='conv4_1')
    net = Conv2dLayer(net, act=tf.nn.relu, shape=[3, 3, 512, 512], strides=[1, 1, 1, 1], padding='SAME', name='conv4_2')
    net = Conv2dLayer(net, act=tf.nn.relu, shape=[3, 3, 512, 512], strides=[1, 1, 1, 1], padding='SAME', name='conv4_3')
    net = PoolLayer(net, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', pool=tf.nn.max_pool, name='pool4')

    # conv5
    net = Conv2dLayer(net, act=tf.nn.relu, shape=[3, 3, 512, 512], strides=[1, 1, 1, 1], padding='SAME', name='conv5_1')
    net = Conv2dLayer(net, act=tf.nn.relu, shape=[3, 3, 512, 512], strides=[1, 1, 1, 1], padding='SAME', name='conv5_2')
    net = Conv2dLayer(net, act=tf.nn.relu, shape=[3, 3, 512, 512], strides=[1, 1, 1, 1], padding='SAME', name='conv5_3')
    net = PoolLayer(
        net, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', pool=tf.nn.max_pool, name='pool5')
    return net


def conv_layers_simple_api(net_in):
    with tf.name_scope('preprocess'):
        # Notice that we include a preprocessing layer that takes the RGB image
        # with pixels values in the range of 0-255 and subtracts the mean image
        # values (calculated over the entire ImageNet training set).
        mean = tf.constant([123.68, 116.779, 103.939], dtype=tf.float32, shape=[1, 1, 1, 3], name='img_mean')
        net_in.outputs = net_in.outputs - mean

    # conv1
    net = Conv2d(net_in, 64, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv1_1')
    net = Conv2d(net, n_filter=64, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv1_2')
    net = MaxPool2d(net, filter_size=(2, 2), strides=(2, 2), padding='SAME', name='pool1')

    # conv2
    net = Conv2d(net, n_filter=128, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv2_1')
    net = Conv2d(net, n_filter=128, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv2_2')
    net = MaxPool2d(net, filter_size=(2, 2), strides=(2, 2), padding='SAME', name='pool2')

    # conv3
    net = Conv2d(net, n_filter=256, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv3_1')
    net = Conv2d(net, n_filter=256, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv3_2')
    net = Conv2d(net, n_filter=256, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv3_3')
    net = MaxPool2d(net, filter_size=(2, 2), strides=(2, 2), padding='SAME', name='pool3')

    # conv4
    net = Conv2d(net, n_filter=512, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv4_1')
    net = Conv2d(net, n_filter=512, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv4_2')
    net = Conv2d(net, n_filter=512, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv4_3')
    net = MaxPool2d(net, filter_size=(2, 2), strides=(2, 2), padding='SAME', name='pool4')

    # conv5
    net = Conv2d(net, n_filter=512, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv5_1')
    net = Conv2d(net, n_filter=512, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv5_2')
    net = Conv2d(net, n_filter=512, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv5_3')
    net = MaxPool2d(net, filter_size=(2, 2), strides=(2, 2), padding='SAME', name='pool5')
    return net


def fc_layers(net):
    net = FlattenLayer(net, name='flatten')
    net = DenseLayer(net, n_units=4096, act=tf.nn.relu, name='fc1_relu')
    net = DenseLayer(net, n_units=4096, act=tf.nn.relu, name='fc2_relu')
    net = DenseLayer(net, n_units=1000, act=tf.nn.relu, name='fc3_relu')
    return net


sess = tf.InteractiveSession()

x = tf.placeholder(tf.float32, [None, 224, 224, 3])
# y_ = tf.placeholder(tf.int32, shape=[None, ], name='y_')

net_in = InputLayer(x, name='input')
# net_cnn = conv_layers(net_in)               # professional CNN APIs
net_cnn = conv_layers_simple_api(net_in)  # simplified CNN APIs
net = fc_layers(net_cnn)

y = net.outputs
probs = tf.nn.softmax(y)
y1 = net_cnn.outputs
# y_op = tf.argmax(tf.nn.softmax(y), 1)
# cost = tl.cost.cross_entropy(y, y_, name='cost')
# correct_prediction = tf.equal(tf.cast(tf.argmax(y, 1), tf.float32), tf.cast(y_, tf.float32))
# acc = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

tl.layers.initialize_global_variables(sess)
net.print_params()
net.print_layers()

tl.files.maybe_download_and_extract(
    'vgg16_weights.npz', '/home/liuwenjie/premodels', 'http://www.cs.toronto.edu/~frossard/vgg16/', expected_bytes=553436134
)

npz = np.load(os.path.join('/home/liuwenjie/premodels', 'vgg16_weights.npz'))

params = []
for val in sorted(npz.items()):
    print("  Loading params %s" % str(val[1].shape))       
    params.append(val[1])

tl.files.assign_params(sess, params[0:16], net_cnn)

img1 = imread('/home/liuwenjie/liuwenjie/tensorflow_workplace/image2012_1.jpg', mode='RGB')  # test data in github
img1 = imresize(img1, (224, 224))

y = sess.run(probs, feed_dict={x: [img1]})[0]  # 1st time take time to compile
# start_time = time.time()
# prob = sess.run(probs, feed_dict={x: [img1]})[0]
# print("  End time : %.5ss" % (time.time() - start_time))
# preds = (np.argsort(prob)[::-1])[0:5]

# for p in preds:
#     print(p, prob[p])

ys = sess.run(y1,feed_dict={x:[img1]})[0]
print(y)

print(ys)

在这里我复制了网上vgg16的参数加载代码,可以看到,在预处理阶段,使用减去均值的办法进行处理,将图片输入,若只需要卷积层不需要全连接层,则将原来npz的前16层的权重加入net_cnn,因为全连接层的数据占全部参数的70%以上,所以做语义分割的过程中不是很建议使用denselayer层的参数.在使用了这一系列操作之后,使用一张图片测试一下带denselayer的检测图片的结果,若有显示则加载预训练模型成功