Bug: tf.contrib.checkpoint.NoDependency object

环境

Python: 3.6
Tensorflow-gpu: 2.1.0

错误信息

something error in training. ValueError(“Unable to save the object ListWrapper([‘Algorithm’, ‘ACC’, ‘MAE’, ‘MAPE’, ‘MSE’, ‘RMSE’, ‘R2’, ‘ME’, ‘Spearman’]) (a list wrapper constructed to track trackable TensorFlow objects). The wrapped list was modified outside the wrapper (its final value was [‘Algorithm’, ‘ACC’, ‘MAE’, ‘MAPE’, ‘MSE’, ‘RMSE’, ‘R2’, ‘ME’, ‘Spearman’], its value when a checkpoint dependency was added was None), which breaks restoration on object creation.\n\nIf you don’t need this list checkpointed, wrap it in a tf.contrib.checkpoint.NoDependency object; it will be automatically un-wrapped and subsequently ignored.”,)
Traceback (most recent call last):

tf.contrib.checkpoint.NoDependency object
NoDependency also works with tf.keras.Model, but only for checkpoint dependencies: wrapping a Layer in NoDependency will assign the (unwrapped) Layer to the attribute without a checkpoint dependency, but the Model will still track the Layer (so it will appear in Model.layers, and its variables will appear in Model.variables).

分析

简单来说，就是在变量(记为param)的值在model training 过程中发生了变化，从而导致在执行保存模型(即，checkpoint.save)时出现无法保存的情况。

代码的逻辑是这样的:

argparser: argparse

class build_model():
		''' 
		     model builder class
		 '''
		def __init__(parser,....):
			  if build:
                 super().__init__(*self.init_model())
        def init_model():
              pass

def  train_model(parser, data)
		params = parse_argument(parser)
		eval_metric_name = hparams['eval_metric_name']
		pp = [x for x in eval_metric_name]
		**pp.insert(0, 'Algorithm')**
		for i in np.arange(num_iterations):
			train_batch(data)
			checkpoint.save(model)  # the place error reported
		...
		
parser = argparse.ArgumentParser()
parser.add_argument('--eval_metric_name', type=list, default=['ACC', 'MAE', 'MAPE', 'MSE', 'RMSE', 'R2', 'ME', 'Spearman'], help='please set eval_metric_name')

// define model
model = build_model(parser)

// load data
data = dataloader.load_data()

// model train
model.train_model(parser, data)

如上， tractable variable 在training过程中发生变化：pp.insert(0, ‘Algorithm’)， 从而导致save model时出现问题，

解决问题

insert操作会修改变量内存本身的值
pp = [‘Algorithm’] + pp

总结

这个问题也是偶然碰到的， 之前一直没有问题， 起因在更换了model的定义方式。
之前model 的定义基于Functional 的方式： 模型类的call方法中最终返回的是tf.keras.Model 对象，
因为一些其他原因， 将模型的定义方式改为了基于Model subclass 的方式，然后就出现了这个问题， 莫名其妙。