Bert系列

自从BERT发布以后, NLP领域基于BERT的改进也层出不穷。nlp_paper github的一个总结。
下面是BERT系列论文阅读计划, 有些之前做过笔记，就会直接贴过来连接。

1. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding - （NAACL 2019）
2. ERNIE 2.0: A Continual Pre-training Framework for Language Understanding - （arXiv 2019)
3. StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding - （arXiv 2019)
4. RoBERTa: A Robustly Optimized BERT Pretraining Approach - （arXiv 2019)
5. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations - arXiv 2019)
6. Multi-Task Deep Neural Networks for Natural Language Understanding - arXiv 2019)
7. What does BERT learn about the structure of language? (ACL2019)
8. Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (ACL2019) [github]
9. Open Sesame: Getting Inside BERT’s Linguistic Knowledge (ACL2019 WS)
10. Analyzing the Structure of Attention in a Transformer Language Model (ACL2019 WS)
11. What Does BERT Look At? An Analysis of BERT’s Attention (ACL2019 WS)
12. Do Attention Heads in BERT Track Syntactic Dependencies?
13. Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains (ACL2019 WS)
14. Inducing Syntactic Trees from BERT Representations (ACL2019 WS)
15. A Multiscale Visualization of Attention in the Transformer Model (ACL2019 Demo)
16. Visualizing and Measuring the Geometry of BERT
17. How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (EMNLP2019)
18. Are Sixteen Heads Really Better than One? (NeurIPS2019)
19. On the Validity of Self-Attention as Explanation in Transformer Models
20. Visualizing and Understanding the Effectiveness of BERT (EMNLP2019)
21. Attention Interpretability Across NLP Tasks
22. Revealing the Dark Secrets of BERT (EMNLP2019)
23. Investigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs (EMNLP2019)
24. The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (EMNLP2019)
25. A Primer in BERTology: What we know about how BERT works
26. Do NLP Models Know Numbers? Probing Numeracy in Embeddings (EMNLP2019)
27. How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations (CIKM2019)
28. Whatcha lookin’ at? DeepLIFTing BERT’s Attention in Question Answering
29. What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?
30. Calibration of Pre-trained Transformers
31. exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models [github]
32. MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices [github]