Udacity Deep Learning课程作业（三）

最新推荐文章于 2021-02-20 18:56:29 发布
最新推荐文章于 2021-02-20 18:56:29 发布
阅读量949 收藏 1
点赞数
分类专栏： 深度学习
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/draco_mystack/article/details/77418043
版权

第三次作业也比较简单，主要学习如何使用tf的函数处理模型过拟合。

Problem 1

Linear Regression with Regularization

batch_size = 128

graph = tf.Graph()
with graph.as_default():

    # Input data. For the training data, we use a placeholder that will be fed
    # at run time with a training minibatch.
    tf_train_dataset = tf.placeholder(tf.float32,
                                    shape=(batch_size, image_size * image_size))
    tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
    # ==> define parameter of regularization
    lmbda = tf.placeholder(tf.float32)

    tf_valid_dataset = tf.constant(valid_dataset)
    tf_test_dataset = tf.constant(test_dataset)

    # Variables.
    weights = tf.Variable(
    tf.truncated_normal([image_size * image_size, num_labels]))
    biases = tf.Variable(tf.zeros([num_labels]))

    # Training computation.
    logits = tf.matmul(tf_train_dataset, weights) + biases
    loss = tf.reduce_mean(
    tf.nn.softmax_cross_entropy_with_logits(labels=tf_train_labels, logits=logits)) + lmbda * tf.nn.l2_loss(weights)
    # ==> add regularized item

    # Optimizer.
    optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)

    # Predictions for the training, validation, and test data.
    train_prediction = tf.nn.softmax(logits)
    valid_prediction = tf.nn.softmax(
    tf.matmul(tf_valid_dataset, weights) + biases)
    test_prediction = tf.nn.softmax(tf.matmul(tf_test_dataset, weights) + biases)

num_steps = 3001

with tf.Session(graph=graph) as session:
    tf.global_variables_initializer().run()
    print("Initialized")
    for step in range(num_steps):
        # Pick an offset within the training data
        offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
        # Generate a minibatch.
        batch_data = train_dataset[offset:(offset + batch_size), :]
        batch_labels = train_labels[offset:(offset + batch_size), :]

        feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels, lmbda : 1e-3}
        _, l, predictions = session.run(
          [optimizer, loss, train_prediction], feed_dict=feed_dict)
        if (step % 500 == 0):
            print("Minibatch loss at step %d: %f" % (step, l))
            print("Minibatch accuracy: %.1f%%" % accuracy(predictions, batch_labels))
            print("Validation accuracy: %.1f%%" % accuracy(
            valid_prediction.eval(), valid_labels))
    print("Test accuracy: %.1f%%" % accuracy(test_prediction.eval(), test_labels))

结果：

Initialized
Minibatch loss at step 0: 19.376085
Minibatch accuracy: 3.9%
Validation accuracy: 13.8%
Minibatch loss at step 500: 2.657836
Minibatch accuracy: 77.3%
Validation accuracy: 76.4%
Minibatch loss at step 1000: 1.627377
Minibatch accuracy: 79.7%
Validation accuracy: 78.4%
Minibatch loss at step 1500: 1.153279
Minibatch accuracy: 78.9%
Validation accuracy: 80.0%
Minibatch loss at step 2000: 0.969894
Minibatch accuracy: 79.7%
Validation accuracy: 80.5%
Minibatch loss at step 2500: 0.691905
Minibatch accuracy: 86.7%
Validation accuracy: 81.0%
Minibatch loss at step 3000: 0.581506
Minibatch accuracy: 86.7%
Validation accuracy: 82.1%
Test accuracy: 89.0%

Neural Network (1-layer ReLU) with Regularization

hidden_nodes = 1024
batch_size = 128

def computation(dataset, weights, biases):
    weight_sum = tf.add(tf.matmul(dataset, weights[0]), biases[0])
    hidden_layer = tf.nn.relu(weight_sum)
    logits = tf.add(tf.matmul(hidden_layer, weights[1]), biases[1])
    return logits

graph = tf.Graph()
with graph.as_default():

    # Input data. For the training data, we use a placeholder that will be fed
    # at run time with a training minibatch.
    tf_train_dataset = tf.placeholder(tf.float32,
                                    shape=(batch_size, image_size * image_size))
    tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
    lmbda = tf.placeholder(tf.float32) # ==> add placeholder

    tf_valid_dataset = tf.constant(valid_dataset)
    tf_test_dataset = tf.constant(test_dataset)

    # Variables.
    weights = [tf.Variable(tf.truncated_normal([image_size * image_size, hidden_nodes])), 
             tf.Variable(tf.truncated_normal([hidden_nodes, num_labels]))
            ]
    biases = [tf.Variable(tf.zeros([hidden_nodes])),
            tf.Variable(tf.zeros([num_labels]))]

    # Training computation.
    logits = computation(tf_train_dataset, weights, biases)

    loss = tf.reduce_mean(
    tf.nn.softmax_cross_entropy_with_logits(labels=tf_train_labels, logits=logits)) + lmbda * (tf.nn.l2_loss(weights[0]) + tf.nn.l2_loss(weights[1])) # ==> add regularization

    # Optimizer.
    optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)

    # Predictions for the training, validation, and test data.
    train_prediction = tf.nn.softmax(logits)
    valid_prediction = tf.nn.softmax(computation(tf_valid_dataset, weights, biases))
    test_prediction = tf.nn.softmax(computation(tf_test_dataset, weights, biases))


def accuracy(predictions, labels):
  return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))
          / predictions.shape[0])

num_steps = 3001

with tf.Session(graph=graph) as session:
    tf.global_variables_initializer().run()
    print("Initialized")
    for step in range(num_steps):

        offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
        # Generate a minibatch.
        batch_data = train_dataset[offset:(offset + batch_size), :]
        batch_labels = train_labels[offset:(offset + batch_size), :]

        feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels, lmbda : 1e-3}
        _, l, predictions = session.run(
          [optimizer, loss, train_prediction], feed_dict=feed_dict)
        if (step % 500 == 0):
            print("Minibatch loss at step %d: %f" % (step, l))
            print("Minibatch accuracy: %.1f%%" % accuracy(predictions, batch_labels))
            print("Validation accuracy: %.1f%%" % accuracy(
            valid_prediction.eval(), valid_labels))
    print("Test accuracy: %.1f%%" % accuracy(test_prediction.eval(), test_labels))

结果：

Initialized
Minibatch loss at step 0: 617.020447
Minibatch accuracy: 14.1%
Validation accuracy: 34.7%
Minibatch loss at step 500: 206.956284
Minibatch accuracy: 72.7%
Validation accuracy: 77.8%
Minibatch loss at step 1000: 113.745140
Minibatch accuracy: 85.2%
Validation accuracy: 81.3%
Minibatch loss at step 1500: 68.286926
Minibatch accuracy: 81.2%
Validation accuracy: 82.7%
Minibatch loss at step 2000: 41.318298
Minibatch accuracy: 85.9%
Validation accuracy: 84.4%
Minibatch loss at step 2500: 25.188065
Minibatch accuracy: 91.4%
Validation accuracy: 85.7%
Minibatch loss at step 3000: 15.299156
Minibatch accuracy: 93.8%
Validation accuracy: 86.6%
Test accuracy: 93.3%

Problem 2

def accuracy(predictions, labels):
  return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))
          / predictions.shape[0])

num_steps = 3001

with tf.Session(graph=graph) as session:
    tf.global_variables_initializer().run()
    print("Initialized")
    for step in range(num_steps):
        offset = ((step % 10) * batch_size) % (train_labels.shape[0] - batch_size)
        # Generate a minibatch.
        batch_data = train_dataset[offset:(offset + batch_size), :]
        batch_labels = train_labels[offset:(offset + batch_size), :]

        feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels, lmbda : 1e-3}
        _, l, predictions = session.run(
          [optimizer, loss, train_prediction], feed_dict=feed_dict)
        if (step % 500 == 0):
            print("Minibatch loss at step %d: %f" % (step, l))
            print("Minibatch accuracy: %.1f%%" % accuracy(predictions, batch_labels))
            print("Validation accuracy: %.1f%%" % accuracy(
            valid_prediction.eval(), valid_labels))
    print("Test accuracy: %.1f%%" % accuracy(test_prediction.eval(), test_labels))

结果：

Initialized
Minibatch loss at step 0: 21.644571
Minibatch accuracy: 7.0%
Validation accuracy: 13.5%
Minibatch loss at step 500: 1.961034
Minibatch accuracy: 96.9%
Validation accuracy: 70.4%
Minibatch loss at step 1000: 1.173835
Minibatch accuracy: 99.2%
Validation accuracy: 72.0%
Minibatch loss at step 1500: 0.738649
Minibatch accuracy: 99.2%
Validation accuracy: 73.4%
Minibatch loss at step 2000: 0.498035
Minibatch accuracy: 99.2%
Validation accuracy: 74.6%
Minibatch loss at step 2500: 0.356002
Minibatch accuracy: 100.0%
Validation accuracy: 75.4%
Minibatch loss at step 3000: 0.271700
Minibatch accuracy: 100.0%
Validation accuracy: 75.8%
Test accuracy: 84.4%

结论：计算minibatch的offet时，修改为offset = ((step % 10) * batch_size) % (train_labels.shape[0] - batch_size)，限制反复使用相同的10个batch数据（每个batch有128个样本），每次epoch使用一个batch。可以看到在2500次迭代后，minibatch的数据可以达到100%准确率，但在测试集上只有84.4%，远不如之前的93.3%

Problem 3

hidden_nodes = 1024
batch_size = 128

def computation(dataset, weights, biases, is_dropout=False, keep_prob=0.5):
    weight_sum = tf.add(tf.matmul(dataset, weights[0]), biases[0])
    hidden_layer = tf.nn.relu(weight_sum)
    if is_dropout: # ==> add dropout
        hidden_layer = tf.nn.dropout(hidden_layer, keep_prob)
    logits = tf.add(tf.matmul(hidden_layer, weights[1]), biases[1])
    return logits

graph = tf.Graph()
with graph.as_default():

    # Input data. For the training data, we use a placeholder that will be fed
    # at run time with a training minibatch.
    tf_train_dataset = tf.placeholder(tf.float32,
                                    shape=(batch_size, image_size * image_size))
    tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
    lmbda = tf.placeholder(tf.float32) # ==> add placeholder
    keep_prob = tf.placeholder(tf.float32) # ==> add placeholder

    tf_valid_dataset = tf.constant(valid_dataset)
    tf_test_dataset = tf.constant(test_dataset)

    # Variables.
    weights = [tf.Variable(tf.truncated_normal([image_size * image_size, hidden_nodes])), 
             tf.Variable(tf.truncated_normal([hidden_nodes, num_labels]))]
    biases = [tf.Variable(tf.zeros([hidden_nodes])),
            tf.Variable(tf.zeros([num_labels]))]

    # Training computation.
    logits = computation(tf_train_dataset, weights, biases, is_dropout=True, keep_prob=keep_prob)

    loss = tf.reduce_mean(
    tf.nn.softmax_cross_entropy_with_logits(labels=tf_train_labels, logits=logits)) + lmbda * (tf.nn.l2_loss(weights[0]) + tf.nn.l2_loss(weights[1])) # ==> add regularization

    # Optimizer.
    optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)

    # Predictions for the training, validation, and test data.
    train_prediction = tf.nn.softmax(logits)
    valid_prediction = tf.nn.softmax(computation(tf_valid_dataset, weights, biases))
    test_prediction = tf.nn.softmax(computation(tf_test_dataset, weights, biases))
Initialized
Minibatch loss at step 0: 746.116699
Minibatch accuracy: 11.7%
Validation accuracy: 32.7%
Minibatch loss at step 500: 200.717834
Minibatch accuracy: 75.0%
Validation accuracy: 78.4%
Minibatch loss at step 1000: 114.825737
Minibatch accuracy: 80.5%
Validation accuracy: 80.8%
Minibatch loss at step 1500: 68.643456
Minibatch accuracy: 79.7%
Validation accuracy: 82.9%
Minibatch loss at step 2000: 41.425804
Minibatch accuracy: 82.0%
Validation accuracy: 83.5%
Minibatch loss at step 2500: 25.150757
Minibatch accuracy: 86.7%
Validation accuracy: 85.3%
Minibatch loss at step 3000: 15.244875
Minibatch accuracy: 92.2%
Validation accuracy: 86.2%
Test accuracy: 92.7%

结论：只在训练中加入dropout操作，验证和测试时不加入。随机丢弃一部分神经元，以一定概率使得神经元激活函数不参与。貌似只能解决小minibatch的过拟合问题，对于大minibatch时，加入dropout效果迭代3000次得到（92.7%）并不如不加的（93.3%）。

Problem 4

hidden_nodes_1 = 1024
hidden_nodes_2 = 500
hidden_nodes_3 = 100
batch_size = 128

def computation(dataset, weights, biases, is_dropout=False): 
    weight_sum_1 = tf.matmul(dataset, weights[0])+ biases[0]
    hidden_layer_1 = tf.nn.relu(weight_sum_1)
    if is_dropout:
        hidden_layer_1 = tf.nn.dropout(hidden_layer_1, keep_prob=0.7)
    weight_sum_2 = tf.matmul(hidden_layer_1, weights[1]) + biases[1]
    hidden_layer_2 = tf.nn.relu(weight_sum_2)
    if is_dropout:
        hidden_layer_2 = tf.nn.dropout(hidden_layer_2, keep_prob=0.7)
    weight_sum_3 = tf.matmul(hidden_layer_2, weights[2]) + biases[2]
    hidden_layer_3 = tf.nn.relu(weight_sum_3)
    if is_dropout:
        hidden_layer_3 = tf.nn.dropout(hidden_layer_3, keep_prob=0.7)
    outputs = tf.matmul(hidden_layer_3, weights[3]) + biases[3]
    return outputs

graph = tf.Graph()
with graph.as_default():

    # Input data. For the training data, we use a placeholder that will be fed
    # at run time with a training minibatch.
    tf_train_dataset = tf.placeholder(tf.float32,
                                    shape=(batch_size, image_size * image_size))
    tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
    lmbda = tf.placeholder(tf.float32) # ==> add placeholder
#     keep_prob = tf.placeholder(tf.float32) # ==> add placeholder

    tf_valid_dataset = tf.constant(valid_dataset)
    tf_test_dataset = tf.constant(test_dataset)

    global_step = tf.Variable(0) # ==> Add for learning rate decay

    # Variables.
    weights = [tf.Variable(tf.truncated_normal([image_size * image_size, hidden_nodes_1],
                                stddev=np.sqrt(2.0 / (image_size * image_size)))), 
            tf.Variable(tf.truncated_normal([hidden_nodes_1, hidden_nodes_2],
                                stddev=np.sqrt(2.0 / hidden_nodes_1))),
            tf.Variable(tf.truncated_normal([hidden_nodes_2, hidden_nodes_3],
                                stddev=np.sqrt(2.0 / hidden_nodes_2))),
            tf.Variable(tf.truncated_normal([hidden_nodes_3, num_labels],
                                stddev=np.sqrt(2.0 / hidden_nodes_3)))

              ]

    biases = [tf.Variable(tf.zeros([hidden_nodes_1])),
            tf.Variable(tf.zeros([hidden_nodes_2])),
            tf.Variable(tf.zeros([hidden_nodes_3])),
            tf.Variable(tf.zeros([num_labels]))
             ]

    # Training computation.
    logits = computation(tf_train_dataset, weights, biases, is_dropout=True) 

    loss = tf.reduce_mean(
    tf.nn.softmax_cross_entropy_with_logits(labels=tf_train_labels, logits=logits)) \
        + lmbda * (tf.nn.l2_loss(weights[0]) \
        + tf.nn.l2_loss(weights[1]) \
        + tf.nn.l2_loss(weights[2]) \
        + tf.nn.l2_loss(weights[3])) # ==> add regularization

    # Optimizer.
    learning_rate = tf.train.exponential_decay(0.5, global_step, 500, 0.9)
    optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step)
#     optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)

    # Predictions for the training, validation, and test data.
    train_prediction = tf.nn.softmax(logits)
    valid_prediction = tf.nn.softmax(computation(tf_valid_dataset, weights, biases))
    test_prediction = tf.nn.softmax(computation(tf_test_dataset, weights, biases))


def accuracy(predictions, labels):
  return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))
          / predictions.shape[0])

num_steps = 8001

with tf.Session(graph=graph) as session:
    tf.global_variables_initializer().run()
    print("Initialized")
    for step in range(num_steps):
        offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
        # Generate a minibatch.
        batch_data = train_dataset[offset:(offset + batch_size), :]
        batch_labels = train_labels[offset:(offset + batch_size), :]

        feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels, lmbda : 1e-3}
        _, l, predictions = session.run(
          [optimizer, loss, train_prediction], feed_dict=feed_dict)
        if (step % 500 == 0):
            print("Minibatch loss at step %d: %f" % (step, l))
            print("Minibatch accuracy: %.1f%%" % accuracy(predictions, batch_labels))
            print("Validation accuracy: %.1f%%" % accuracy(
            valid_prediction.eval(), valid_labels))
    print("Test accuracy: %.1f%%" % accuracy(test_prediction.eval(), test_labels))

结果：

Initialized
Minibatch loss at step 0: 3.903420
Minibatch accuracy: 8.6%
Validation accuracy: 31.9%
Minibatch loss at step 500: 1.387359
Minibatch accuracy: 84.4%
Validation accuracy: 84.9%
Minibatch loss at step 1000: 0.973619
Minibatch accuracy: 88.3%
Validation accuracy: 86.1%
Minibatch loss at step 1500: 0.967071
Minibatch accuracy: 84.4%
Validation accuracy: 86.5%
Minibatch loss at step 2000: 0.897126
Minibatch accuracy: 82.8%
Validation accuracy: 87.2%
Minibatch loss at step 2500: 0.560037
Minibatch accuracy: 91.4%
Validation accuracy: 87.5%
Minibatch loss at step 3000: 0.477757
Minibatch accuracy: 93.8%
Validation accuracy: 88.1%
Minibatch loss at step 3500: 0.596783
Minibatch accuracy: 86.7%
Validation accuracy: 88.2%
Minibatch loss at step 4000: 0.578859
Minibatch accuracy: 89.8%
Validation accuracy: 88.5%
Minibatch loss at step 4500: 0.515870
Minibatch accuracy: 89.1%
Validation accuracy: 88.8%
Minibatch loss at step 5000: 0.603967
Minibatch accuracy: 85.2%
Validation accuracy: 89.2%
Minibatch loss at step 5500: 0.420070
Minibatch accuracy: 92.2%
Validation accuracy: 89.3%
Minibatch loss at step 6000: 0.486005
Minibatch accuracy: 93.8%
Validation accuracy: 89.4%
Minibatch loss at step 6500: 0.555979
Minibatch accuracy: 90.6%
Validation accuracy: 89.6%
Minibatch loss at step 7000: 0.377530
Minibatch accuracy: 93.0%
Validation accuracy: 89.6%
Minibatch loss at step 7500: 0.470899
Minibatch accuracy: 91.4%
Validation accuracy: 89.6%
Minibatch loss at step 8000: 0.398080
Minibatch accuracy: 90.6%
Validation accuracy: 89.8%
Test accuracy: 95.2%

学习总结：

  • 引入正则化来缓解过拟合：用tf.nn.l2_loss()计算模型参数的L2范数，加入到loss中，具体如下：
loss = tf.reduce_mean(
    tf.nn.softmax_cross_entropy_with_logits(labels=tf_train_labels, logits=logits)) + lmbda * (tf.nn.l2_loss(weights[0]) + tf.nn.l2_loss(weights[1])) # ==> add regularization

# Optimizer.
optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)

  • 用较大的minibatch样本数缓解过拟合：在用较小的minbatch样本训练会出现过拟合，即验证集达到100%准确率，但测试集准确率不高

  • 引入dropout来缓解过拟合：用tf.nn.dropout(keep_prob)在神经网络的训练中引入dropout，指定keep_prob，具体如下：

# tf.nn.dropout(x, keep_prob, noise_shape=None, seed=None, name=None)用法
# 参数：训练样本x，keep_prob为训练时神经元节点保留概率

hidden_layer = tf.nn.dropout(hidden_layer, keep_prob)
  • 用衰减学习率+多层神经网络改善模型性能：
    • 学习率使用指数衰减，前期使用较大学习率来加快得到最优解，越往后epoch使用越小的学习率，避免靠近最优点时出现震荡，更好寻优；
    • 引入多层神经网络，加一层ReLU，将上一层的ReLU输出再进行一次加权组合，W1(feature*hidden_nodes_1)，W2(hidden_nodes_1, hidden_nodes_2)和W3(hidden_nodes_2, num_labels)。
# tf.train.exponential_decay(learning_rate, global_step, decay_steps, decay_rate, staircase=False, name=None)用法
# 参数：初始值learning_rate, global_step, decay_steps, 衰减率decay_rate, staircase=False
# decayed_learning_rate = learning_rate *
#                        decay_rate ^ (global_step / decay_steps)
# (staircase置True时，global_step/decay_steps为整除)
# 每过一次decay_steps轮，衰减率乘一次decay_rate实现衰减

global_step = tf.Variable(0)  # count the number of steps taken.
learning_rate = tf.train.exponential_decay(0.5, global_step, 500, 0.6, staircase=True) 
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step)

第三章作业主要是学会处理神经网络的过拟合：

  • 引入正则化来缓解过拟合：用tf.nn.l2_loss()计算模型参数的L2范数，加入到loss中，具体如下：
loss = tf.reduce_mean(
    tf.nn.softmax_cross_entropy_with_logits(labels=tf_train_labels, logits=logits)) + lmbda * (tf.nn.l2_loss(weights[0]) + tf.nn.l2_loss(weights[1])) # ==> add regularization

# Optimizer.
optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)

  • 用较大的minibatch样本数缓解过拟合：在用较小的minbatch样本训练会出现过拟合，即验证集达到100%准确率，但测试集准确率不高

  • 引入dropout来缓解过拟合：用tf.nn.dropout(keep_prob)在神经网络的训练中引入dropout，指定keep_prob，具体如下：

# tf.nn.dropout(x, keep_prob, noise_shape=None, seed=None, name=None)用法
# 参数：训练样本x，keep_prob为训练时神经元节点保留概率

hidden_layer = tf.nn.dropout(hidden_layer, keep_prob)
  • 用衰减学习率+多层神经网络改善模型性能：
    • 学习率使用指数衰减，前期使用较大学习率来加快得到最优解，越往后epoch使用越小的学习率，避免靠近最优点时出现震荡，更好寻优；
    • 引入多层神经网络，加两层ReLU，将上一层的ReLU输出再进行一次加权组合，W1(feature*hidden_nodes_1)，W2(hidden_nodes_1, hidden_nodes_2), W3(hidden_nodes_2, hidden_nodes_3), 和W4(hidden_nodes_3, num_labels)。
# tf.train.exponential_decay(learning_rate, global_step, decay_steps, decay_rate, staircase=False, name=None)用法
# 参数：初始值learning_rate, global_step, decay_steps, 衰减率decay_rate, staircase=False
# decayed_learning_rate = learning_rate *
#                        decay_rate ^ (global_step / decay_steps)
# (staircase置True时，global_step/decay_steps为整除)
# 每过一次decay_steps轮，衰减率乘一次decay_rate实现衰减

global_step = tf.Variable(0)  # count the number of steps taken.
learning_rate = tf.train.exponential_decay(0.5, global_step, 500, 0.6, staircase=True) 
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step)
def computation(dataset, weights, biases, is_dropout=False): 
    weight_sum_1 = tf.matmul(dataset, weights[0])+ biases[0]
    hidden_layer_1 = tf.nn.relu(weight_sum_1)
    if is_dropout:
        hidden_layer_1 = tf.nn.dropout(hidden_layer_1, keep_prob=0.7)
    weight_sum_2 = tf.matmul(hidden_layer_1, weights[1]) + biases[1]
    hidden_layer_2 = tf.nn.relu(weight_sum_2)
    if is_dropout:
        hidden_layer_2 = tf.nn.dropout(hidden_layer_2, keep_prob=0.7)
    weight_sum_3 = tf.matmul(hidden_layer_2, weights[2]) + biases[2]
    hidden_layer_3 = tf.nn.relu(weight_sum_3)
    if is_dropout:
        hidden_layer_3 = tf.nn.dropout(hidden_layer_3, keep_prob=0.7)
    outputs = tf.matmul(hidden_layer_3, weights[3]) + biases[3]
    return outputs
天才XLM
关注
专栏目录
udacity-deep-learning：Udacity深度学习MOOC作业
02-06
udacity-deep-learning：Udacity深度学习MOOC作业
Python图像处理：使用TensorFlow或Keras进行图像分类
人邮异步社区
12-22 7492
在本节中，我们将重新讨论手写数字分类的问题（使用MNIST数据集），但这次使用的是深度神经网络，即使用两个非常流行的深度学习库TensorFlow和Keras来解决这个问题。TensorFlow（TF）是用于建立深度学习模型的著名的库，它有一个非常庞大且令人惊艳的社区。然而，TensorFlow并不容易学会使用。而Keras是一个基于TensorFlow的高级应用程序接口（API），对底层结构的控制较少，但它比TF更友好且更易于使用。底层库提供了更多的灵活性，因此TF可以相比Keras可以更多地调整，更灵活
python中定义变量为啥要用下划线_python 之中使用单下划线“_”做变量的目的？...
weixin_29839415的博客
01-29 248
Pythonnum_steps = 801def accuracy(predictions, labels):return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))/ predictions.shape[0])with tf.Session(graph=graph) as session:# This is...
CS344 Udacity 答案
B10090411的博客
01-16 1686
这里我只给出student.cu的代码 Ploblem Set1// Homework 1 // Color to Greyscale Conversion //A common way to represent color images is known as RGBA - the color //is specified by how much Red, Green, and Blue i
语义分割python教学_tensorflow2.0 语义分割 评价指标代码（tensor版）
weixin_39742065的博客
12-05 408
np编写的语义分割评价指标不能通过GPU 加速，故改编了一份tensor的代码，在tf2.0测试通过。第一次写博客，将就着看吧def get_hist_t(predictions, labels, num_class):# labels:[b.h,w] tensor# predictions:[b.h,w,c] tensornum_class = predictions.shape[3]bat...
Udacity Deep Learning课程作业（一）
天才XLM的博客
08-17 2128
Udacity的深度学习是Google开设的一门基于TensorFlow完成任务的在线课程，课程短小精悍，包括4章（入门ML/DL，DNN，CNN，RNN）、6个小作业（以ipynb的形式，十分方便友好）和1个大作业（开发实时摄像头应用）。有ML/DL基础的同学，视频很快可以过完，因此课程精华在于其实战项目，很有意思。作为G家的课程，算是TensorFlow比较权威的学习tutorial了。课程链接
Deep-Learning-Nanodegree-Udacity:Udacity的深度学习纳米学位
05-01
这是我在Udacity的Deep Learning Nanodegree的市长项目上的实现的存储库。 神经网络。 数学演示： ： 使用TensorFlow实施 对成本函数和梯度下降的不同初始化的含义。 已审核： 初始化。 统一配送，统一出售。 ...
Udacity_Deep_learning
04-04
该存储库包含与Udacity的有关的材料。 它由一堆用于各种深度学习主题的教程笔记本组成。 在大多数情况下，笔记本会指导您实现诸如卷积网络，循环网络和GAN之类的模型。 还涉及其他主题，例如权重初始化和批次归一化...
udacity_deep_learning
02-26
《深度学习基础与实践——udacity_deep_learning课程详解》 深度学习是现代人工智能领域的核心驱动力，它在图像识别、语音处理、自然语言处理等多个领域展现出强大的能力。本篇文章将基于"udacity_deep_learning"这...
Udacity-Deep_Learning
02-28
Udacity-Deep_Learning 设置存储库以容纳我在从Udacity购买深度学习纳米学位计划时完成的练习和项目。 该课程的课程共分为6个部分，其中包括7个主要项目，以及在整个课程中完成的各种练习。 项目和练习内容如下： ...
Udacity狂欢节：Udacity虚拟现实纳米学位的项目＃1
02-06
Udacity狂欢节入门项目 该项目是的。 版本号 Unity 2017.1.0 GVR Unity SDK v1.60.0
Udacity作业——TensorFlow notMNIST代码及输出结果（Jupyter文件）
09-17
Udacity作业——TensorFlow notMNIST代码及输出结果，Jupyter文件。Jupyter是很好的工具。预览内容可以参见：http://blog.csdn.net/jidebingfeng/article/details/78013799
Udacity深度学习DeepLearning课程作业1—notMnist
学习专栏，期待交流与成长
01-22 6892
前言：Udacity上有一个免费的《深度学习》单项课程。虽然是免费的课程，但保持了Udacity一贯的水准。其课程主要是神经网络的实际应用。因此，该课程能较好的提升实际项目水平。但是，如果需要提升理论水平，则可以同步学习Coursera上hinton的《机器学习》课程，里面有很深入和系统的讲解了深度学习的所有基础理论。该课程的入门作业是notMnist字符的分类。
Udacity Deep Learning课程作业（四）
天才XLM的博客
08-20 891
第四章作业主要是关于卷积神经网络的练习，涉及到tf.nn.conv2d、tf.nn.max_pooling等的使用。相关API的用法tf.nn.conv2dtf.nn.conv2d(input, filter, strides, padding, use_cudnn_on_gpu=None, name=None)： - input和filter都是4D的tensor类型，计算两者的2D卷积，inp
tensorflow session.run()
02-20 1428
1. 简单说一下，session.run(),需要几个参数，fetches，feed_dict,前者是输出，后者是输入。 tf有个特点，是在模型中定义所有的张量，铺出一张图，session.run才会真正计算，这里fetches参数中包含的内容就是需要进行计算的变量。 举个例子： loss = session.run(fetches=[ model.loss], feed_dict={ model.inpu...
干货！！！学习笔记-TensorFlow NN
weili_的博客
06-24 978
1、NN1.1 基础网络的解析       其中，测试和验证集能赋值到 tf.constant() 中，而训练集可以导入 tf.placeholder() 中，训练集只有导入占位符我们才能在随机梯度下降中成批量地进行训练。网络定义的权重矩阵和偏置向量后需要执行初始化，每一层需要一个权重矩阵和一个偏置向量。构建损失函数，并计算训练损失。模型会输出一个预测向量，我们可以比较预测标签和真实标签并使用交叉...
理解LSTM网络
Deep Learning and NLP Farm
11-16 3504
英文原地址：http://colah.github.io/posts/2015-08-Understanding-LSTMs/ 中文原地址：http://www.jianshu.com/p/9dc9f41f0b29 Recurrent Neural Networks 人类并不是每时每刻都从一片空白的大脑开始他们的思考。在你阅读这篇文章时候，你都是基于自己已经拥有的对先前所见词的理
Udacity-A/B TEST
zpxcod007的博客
04-30 2682
这个是之间在Udacity参与数据分析课程的项目之一。完成了有一阵子了，想着发上来以后方便复习。一．项目说明在进行此试验时，优达学城当前的主页上有两个选项：“开始免费试学”和“访问课程资料”。如果学生点击“开始免费试学”，系统将要求他们输入信用卡信息，然后他们将进入付费课程版本的免费试学。14天后，将对他们自动收费，除非他们在此期限结束前取消试用。若学生点击“访问课程材料”，他们将能够观看视频和免...
深度学习的课程和培训你有推荐吗
最新发布
04-03
1. 深度学习入门课程（Deep Learning Specialization）：由吴恩达教授领导的系列课程，涵盖了深度学习的基本概念、神经网络、卷积神经网络、循环神经网络等。 2. TensorFlow官方文档：TensorFlow是目前最流行的深度...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值