bert代码学习

1 bert训练的时候打出日志

基于官方的https://github.com/google-research/bert/issues/70

 if mode == tf.estimator.ModeKeys.TRAIN:

      train_op = optimization.create_optimizer(
          total_loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu)

      logging_hook = tf.train.LoggingTensorHook({"loss": total_loss}, every_n_iter=10)

      output_spec = tf.contrib.tpu.TPUEstimatorSpec(
          mode=mode,
          loss=total_loss,
          train_op=train_op,
          training_hooks=[logging_hook],
          scaffold_fn=scaffold_fn)

tensorflow版本1.11

2 Early Stopping

Step1：建一个hook

early_stopping_hook = tf.contrib.estimator.stop_if_no_decrease_hook(
            estimator=estimator,
            metric_name='eval_loss',
            max_steps_without_decrease=FLAGS.max_steps_without_decrease,
            eval_dir=None,
            min_steps=0,
            run_every_secs=None,
            run_every_steps=FLAGS.save_checkpoints_steps)

Step2：加到estimator.train里

estimator.train(input_fn=train_input_fn, max_steps=num_train_steps, hooks=[early_stopping_hook])

Train and Evaluate
Step1：创建train和eval的spec，这里需要把early stopping的hook加到trainSpec

train_spec = tf.estimator.TrainSpec(input_fn=train_input_fn, max_steps=num_train_steps,
                                                hooks=[early_stopping_hook])
eval_spec = tf.estimator.EvalSpec(input_fn=eval_input_fn, throttle_secs=60)
tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)

https://zhuanlan.zhihu.com/p/51762599