简单记录Bert

读到一篇论文中对Bert的一点简单介绍，感觉比较便于加深理解，在此mark一下。

BERT [6] is a transformer [49] model that learns textual representations by conditioning on both left and right context for all layers. BERT was pre-trained for two different tasks, MLM and NSP. For MLM, 15% of the tokens are replaced with a [MASK] token, and the model is trained to predict the masked tokens). For NSP, the model is trained to distinguish (binary classification) between pairs of sentences A and B, where 50% of the time B is the next and 50% it is not the next
sentence (a random sentence is selected). The special token [CLS] is added to every sentence during pre-training; it is used for classification tasks. [SEP] is another special token that is used to separate sentence pairs that are packed together into a single sequence. Additionally, there is a special learned embedding which indicates whether each token comes from sentence A or B. BERT was pre-trained using both English Wikipedia (2.5m words) and the BookCorpus [63],