BERT 模型预训练流程简单认识

BERT流程图

代码模块

代码参考：https://github.com/cmd23333/BERT-Tensorflow2.x
https://github.com/MorvanZhou/NLP-Tutorials

1、 损失函数模块

with tf.GradientTape() as t:
    nsp_predict, mlm_predict, sequence_output = model((batch_x, batch_padding_mask, batch_segment),
                                                      training=True)
    nsp_loss, mlm_loss = loss_fn((mlm