来自于李宏毅老师的bert课程,http://speech.ee.ntu.edu.tw/~tlkagk/courses_ML19.html
0. 前置基础
word can have multiple sense.
e.g. bank is word type, can be multiple word tokens
contextalized embedding: 同一个word type,在不同的上下文中,有不同的word embedding
1. ELMO
embedding from Language Model 语言模型
RNN-based language models(trained from lots of sentences)
e.g.对于某个句子,如何计算某个token的embedding?
A:将bi-RNN的某个token的前后两个hidden states,拼接起来作为当前token的embedding
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-N5lOkV8M-1583759173930)(./DL_picture/bert1.png)]
2. BERT
bi-directional encoder representation for transformer
bert = encoder of transformer
备注:训练bert是不需要label的;半监督的==其实还是有label的; learned from a large amount of text without annotaion
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Q1GsRRD2-1583759173938)(./DL_picture/bert2.png)]<