本文汇集了多个关于BERT及其在NLP中应用的研究，探讨了BERT如何在预训练中捕获语法和语义信息，以及如何通过结构探针对词汇表示进行分析。还介绍了BERT在多任务学习、知识蒸馏、语义理解、句向量表示、适应器模块等方面的改进和应用。此外，文章提到了RoBERTa、SenseBERT、Sentence-BERT等BERT的变体，并指出XLNET通过自注意力机制实现了双向上下文的理解。
2019.9.5 note
A Structural Probe for Finding Syntax in Word Representations
- The probe identifies a linear transformation under which squared L2 distance encodes the distance between words in the parse tree, and one in which squared L2 norm encodes depth in the parse tree. Using this probe, we show that such transformations exist, providing evidence that entire syntax trees are embedded implicitly in deep models’ vector geometry.
This defines d ( x , y ) = f ( x ) T f ( y ) d(x, y)=f(x)^Tf(y) d(x,y)=f(x)Tf(y) and f ( x ) = A v x f(x)=Av_x f(x)=Avx for BERT embedding v v v. This finds that this d d d can learn the distances on parsing tr
最低0.47元/天 解锁文章
扫一扫
1567
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
