TF/02_TensorFlow_Way/06 Working_with_Batch_and_Stochastic_Training

Working with Batch and Stochastic Training

Summary

Here, we introduce the differences between batch and stochastic training and how to implement both in TensorFlow. Stochastic training is defined as training on one observation at once, and batch training is defined as training on a group of observations at once.

Model

In this script, we use generated data. We will generate input data that is distributed as Normal(mean=1, sd=0.1). All the target data will be is the value 10.0 repeated. The model will try to predict the multiplication factor to minimize the loss between the model output and the value 10.0.

Notes

It is important to note that TensorFlow works well with many dimensional matrices, so we can easily implement batch training by adding the batch dimension to our inputs, as illustrated in this script.

Viewing the Difference

Here we plot the loss function of stochastic and batch training on the same graph. Notice how stochastic training is less smooth in the convergence of a solution. This may sound like a bad thing, but it can help explore sample spaces and be less likely to be stuck in local minima.

# 06_batch_stochastic_training.py
# Batch and Stochastic Training
#----------------------------------
#
#  This python function illustrates two different training methods:
#  batch and stochastic training.  For each model, we will use
#  a regression model that predicts one model variable.

import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
from tensorflow.python.framework import ops
ops.reset_default_graph()

# We will implement a regression example in stochastic and batch training

# Stochastic Training:
# Create graph
sess = tf.Session()

# Create data
x_vals = np.random.normal(1, 0.1, 100)
y_vals = np.repeat(10., 100)
x_data = tf.placeholder(shape=[1], dtype=tf.float32)
y_target = tf.placeholder(shape=[1], dtype=tf.float32)

# Create variable (one model parameter = A)
A = tf.Variable(tf.random_normal(shape=[1]))

# Add operation to graph
my_output = tf.multiply(x_data, A)

# Add L2 loss operation to graph
loss = tf.square(my_output - y_target)

# Initialize variables
init = tf.global_variables_initializer()
sess.run(init)

# Create Optimizer
my_opt = tf.train.GradientDescentOptimizer(0.02)
train_step = my_opt.minimize(loss)

loss_stochastic = []
# Run Loop
for i in range(100):
    rand_index = np.random.choice(100)
    rand_x = [x_vals[rand_index]]
    rand_y = [y_vals[rand_index]]
    sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y})
    if (i+1)%5==0:
        print('Step #' + str(i+1) + ' A = ' + str(sess.run(A)))
        temp_loss = sess.run(loss, feed_dict={x_data: rand_x, y_target: rand_y})
        print('Loss = ' + str(temp_loss))
        loss_stochastic.append(temp_loss)


# Batch Training:
# Re-initialize graph
ops.reset_default_graph()
sess = tf.Session()

# Declare batch size
batch_size = 20

# Create data
x_vals = np.random.normal(1, 0.1, 100)
y_vals = np.repeat(10., 100)
x_data = tf.placeholder(shape=[None, 1], dtype=tf.float32)
y_target = tf.placeholder(shape=[None, 1], dtype=tf.float32)

# Create variable (one model parameter = A)
A = tf.Variable(tf.random_normal(shape=[1,1]))

# Add operation to graph
my_output = tf.matmul(x_data, A)

# Add L2 loss operation to graph
loss = tf.reduce_mean(tf.square(my_output - y_target))

# Initialize variables
init = tf.global_variables_initializer()
sess.run(init)

# Create Optimizer
my_opt = tf.train.GradientDescentOptimizer(0.02)
train_step = my_opt.minimize(loss)

loss_batch = []
# Run Loop
for i in range(100):
    rand_index = np.random.choice(100, size=batch_size)
    rand_x = np.transpose([x_vals[rand_index]])
    rand_y = np.transpose([y_vals[rand_index]])
    sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y})
    if (i+1)%5==0:
        print('Step #' + str(i+1) + ' A = ' + str(sess.run(A)))
        temp_loss = sess.run(loss, feed_dict={x_data: rand_x, y_target: rand_y})
        print('Loss = ' + str(temp_loss))
        loss_batch.append(temp_loss)

plt.plot(range(0, 100, 5), loss_stochastic, 'b-', label='Stochastic Loss')
plt.plot(range(0, 100, 5), loss_batch, 'r--', label='Batch Loss, size=20')
plt.legend(loc='upper right', prop={'size': 11})
plt.show()

Step #5 A = [ 4.01994658]
Loss = [ 44.24846268]
Step #10 A = [ 5.09761477]
Loss = [ 29.49569702]
Step #15 A = [ 6.00010538]
Loss = [ 18.27215576]
Step #20 A = [ 6.74236965]
Loss = [ 11.45827484]
Step #25 A = [ 7.33007812]
Loss = [ 5.45985699]
Step #30 A = [ 7.80332994]
Loss = [ 2.90487218]
Step #35 A = [ 8.1608696]
Loss = [ 1.4812237]
Step #40 A = [ 8.5275383]
Loss = [ 1.63392246]
Step #45 A = [ 8.92523861]
Loss = [ 5.08095312]
Step #50 A = [ 9.05187988]
Loss = [ 1.64842927]
Step #55 A = [ 9.13195038]
Loss = [ 2.97699523]
Step #60 A = [ 9.2034111]
Loss = [ 0.16664693]
Step #65 A = [ 9.41993332]
Loss = [ 1.09736943]
Step #70 A = [ 9.55030441]
Loss = [ 0.17578457]
Step #75 A = [ 9.59056473]
Loss = [ 0.10454286]
Step #80 A = [ 9.63950634]
Loss = [ 0.0052873]
Step #85 A = [ 9.59080887]
Loss = [ 0.06288875]
Step #90 A = [ 9.67657375]
Loss = [ 0.4455913]
Step #95 A = [ 9.75543785]
Loss = [ 3.03861761]
Step #100 A = [ 9.76567459]
Loss = [ 2.32474542]
Step #5 A = [[ 0.43515953]]
Loss = 91.6266
Step #10 A = [[ 2.21869946]]
Loss = 60.0579
Step #15 A = [[ 3.64795804]]
Loss = 39.7023
Step #20 A = [[ 4.80799341]]
Loss = 28.0705
Step #25 A = [[ 5.75893736]]
Loss = 18.8791
Step #30 A = [[ 6.53450441]]
Loss = 12.4817
Step #35 A = [[ 7.16327238]]
Loss = 7.52569
Step #40 A = [[ 7.69103813]]
Loss = 5.7873
Step #45 A = [[ 8.10608864]]
Loss = 3.52929
Step #50 A = [[ 8.45858669]]
Loss = 3.31691
Step #55 A = [[ 8.74719143]]
Loss = 1.71037
Step #60 A = [[ 8.96679878]]
Loss = 1.68568
Step #65 A = [[ 9.12759972]]
Loss = 1.42947
Step #70 A = [[ 9.28309345]]
Loss = 1.41552
Step #75 A = [[ 9.40476608]]
Loss = 0.775301
Step #80 A = [[ 9.53863621]]
Loss = 1.01685
Step #85 A = [[ 9.59559345]]
Loss = 1.00151
Step #90 A = [[ 9.6467247]]
Loss = 0.616646
Step #95 A = [[ 9.71541309]]
Loss = 0.69294
Step #100 A = [[ 9.7681818]]
Loss = 1.0665