TensoFlow slim fine tune "BatchNorm/gamma/RMSProp_1 not found in checkpoint"

在用slim的model zoo fine tune的时候，经常想先tune fc layer，在tune conv layer，

但是如果调完fc直接free 所有conv layer的时候，会出现"BatchNorm/gamma/RMSProp_1 not found in checkpoint"，意思就是优化器RMSprop的参数没有找到，一个解决办法是吧优化器换成没有额外参数的SGD，但这个只是曲线救国

在https://github.com/tensorflow/models/issues/1836找到了原因，因为之前调fc是freeze了前面的conv layer，自然也对这些layer的优化器参数做了某些处理，等到free conv的时候，因为指定的是同一个train_dir，slim会自动load这里面的ckpt，也就是说load进来的model的conv layer是没有RMSprop参数的，但是free conv的时候又要train所有layer，自然就会报找不到参数

如果换一个train_dir就OK了，应该是因为换了一种load parameter参数的方法吧

Make sure that the path --train_dir is initialized when you did your training.
I had the same problem before, it was because in my --train_dir there was an checkpoint file of a different model, so when you run with the current model, this wrong checkpoint file will be searched and used instead .

def _get_init_fn():
  """Returns a function run by the chief worker to warm-start the training.

  Note that the init_fn is only run when initializing the model during the very
  first global step.

  Returns:
    An init function run by the supervisor.
  """
  if FLAGS.checkpoint_path is None:
    return None

  # Warn the user if a checkpoint exists in the train_dir. Then we'll be
  # ignoring the checkpoint anyway.
  if tf.train.latest_checkpoint(FLAGS.train_dir):
    tf.logging.info(
        'Ignoring --checkpoint_path because a checkpoint already exists in %s'
        % FLAGS.train_dir)
    return None

  exclusions = []
  if FLAGS.checkpoint_exclude_scopes:
    exclusions = [scope.strip()
                  for scope in FLAGS.checkpoint_exclude_scopes.split(',')]

  # TODO(sguada) variables.filter_variables()
  variables_to_restore = []
  for var in slim.get_model_variables():
    excluded = False
    for exclusion in exclusions:
      if var.op.name.startswith(exclusion):
        excluded = True
        break
    if not excluded:
      variables_to_restore.append(var)

  if tf.gfile.IsDirectory(FLAGS.checkpoint_path):
    checkpoint_path = tf.train.latest_checkpoint(FLAGS.checkpoint_path)
  else:
    checkpoint_path = FLAGS.checkpoint_path

  tf.logging.info('Fine-tuning from %s' % checkpoint_path)

  return slim.assign_from_checkpoint_fn(
      checkpoint_path,
      variables_to_restore,
      ignore_missing_vars=FLAGS.ignore_missing_vars)