TensorFlow的移动平均(Moving Averages)

一些训练优化算法，比如GradientDescent 和Momentum 在优化过程中便可以使用到移动平均方法。使用移动平均常常可以较明显地改善结果。

操作	描述
class tf.train.ExponentialMovingAverage	将指数衰退加入到移动平均中
tf.train.ExponentialMovingAverage.apply(var_list=None)	对var_list变量保持移动平均
tf.train.ExponentialMovingAverage.average_name(var)	返回var均值的变量名称
tf.train.ExponentialMovingAverage.average(var)	返回var均值变量
tf.train.ExponentialMovingAverage.variables_to_restore(moving_avg_variables=None)	返回用于保存的变量名称的映射

▷ tf.train.ExponentialMovingAverage

# Example usage when creating a training model:
# Create variables.
var0 = tf.Variable(...)
var1 = tf.Variable(...)
# ... use the variables to build a training model...
...
# Create an op that applies the optimizer.  This is what we usually
# would use as a training op.
opt_op = opt.minimize(my_loss, [var0, var1])

# Create an ExponentialMovingAverage object
ema = tf.train.ExponentialMovingAverage(decay=0.9999)

# Create the shadow variables, and add ops to maintain moving averages
# of var0 and var1.
maintain_averages_op = ema.apply([var0, var1])

# Create an op that will update the moving averages after each training
# step.  This is what we will use in place of the usual training op.
with tf.control_dependencies([opt_op]):
    training_op = tf.group(maintain_averages_op)

...train the model by running training_op...

#Example of restoring the shadow variable values:
# Create a Saver that loads variables from their saved shadow values.
shadow_var0_name = ema.average_name(var0)
shadow_var1_name = ema.average_name(var1)
saver = tf.train.Saver({shadow_var0_name: var0, shadow_var1_name: var1})
saver.restore(...checkpoint filename...)
# var0 and var1 now hold the moving average values