模型蒸馏探索(Bert)

1. 蒸馏是什么?

​  所谓的蒸馏,指的是从大模型(通常称为teacher model)中学习小模型(通常称为student model)。何以用这个名字呢?在化学中,蒸馏是一个有效的分离沸点不同的组分的方法,大致步骤是先升温使低沸点的组分汽化,然后降温冷凝,达到分离出目标物质的目的。那么,从大模型中,通过一定的技术手段,将原模型中的知识提取出来,这个过程很类似于物质分离,所以将其称为是蒸馏。

2. 蒸馏方法

2.1 Logit Distillation

 深度学习巨头Hinton提出,是一篇开创性的工作。

 其改进是针对softmax进行的改进:

q i = e x p ( z i / T ) ∑ j e x p ( z j / T ) q_{i}=\frac {exp(z_{i}/T)}{\sum_{j}exp(z_{j}/T)} qi=jexp(zj/T)exp(zi/T)

 其中的T是temperature,为设定的超参数。

计算流程

 最终的loss为:

L = ( 1 − α ) c r o s s _ e n t r o p h y ( y , p ) + α ∗ c r o s s _ e n t r o p h y ( q , p ) T 2 y : 真 实 l a b e l p : s t u d e n t   m o d e l 预 测 结 果 q : t e a c h e r   m o d e l 预 测 结 果 α : 蒸 馏 l o s s 权 重 因 为 求 梯 度 的 时 候 会 新 的 目 标 函 数 会 导 致 梯 度 是 以 前 的 1 T 2 , 所 以 要 再 乘 上 T 2 L=(1-\alpha)cross\_entrophy(y,p)+\alpha *cross\_entrophy(q,p)T^{2} \\ y:真实label\\p:student\ model预测结果\\q:teacher\ model预测结果 \\ \alpha:蒸馏loss权重 \\因为求梯度的时候会新的目标函数会导致梯度是以前的\frac {1}{T^2},所以要再乘上T^{2} L=(1α)cross_entrophy(y,p)+αcross_entrophy(q,p)T2ylabelp:student modelq:teacher modelα:lossT21T2

 这个改进的motivation有一下几点:

  • softmax函数自身对数据分布敏感

 对于相同的logits,当采用不同的temperature的时候,softmax之后的分布变化较大,温度越大,分布越平缓,结果的区分度越低,相当于增大了学习的难度,以后做inference的时候,temperature=1,分类结果会得到较好的提升。

  • soft prediction本身带有额外的信息

soft prediction代表teacher model对不同类别的识别概率,这个概率分布本身就带有一定的信息的,比如预测轿车的时候,识别为垃圾车和胡萝卜的概率可能都比较低,但是识别为垃圾车的概率显然要比识别为胡萝卜更高,这个信息说明垃圾车本身相比于胡萝卜与轿车的相关性更高。

 这里有人可能会好奇,为何需要先训练teacher model,然后再蒸馏到student model上面?为何不能直接训练student model?

 要注意的是,蒸馏的核心思想是好的模型不是为了拟合训练数据,而是学习如何泛化到新的数据,所以蒸馏到目的是为了让学生模型学习到教师模型的泛化能力。单纯训练学生模型的话,因为模型比较简单,所以训练难度也更大,其训练出的模型的泛化能力大概率也不如教师模型强大。

 另外注意,模型蒸馏是一种思想,理解了这篇文章的思想,可以泛化到后续的许多模型中去,因为蒸馏的使用其实本质就是各种loss function的设计。

2.2 Distilled BiLSTM

 这篇文章在性能方面完全不存在竞争力,在transformer满天飞的年代,其蒸馏的结果仅仅是获得了ELMo级别的性能,不过,这篇文章最大的亮点是,在ELMo性能级别下,其使用的参数少了大约100倍,推理时间少了15倍,这对于资源敏感类任务来说可谓是一个巨大的诱惑。

Distilled BiLSTM两类任务,其teacher model使用的是BERT-large,student model为BiLSTM+Relu

 注意FIgure 2中的d操作,对于两个句子向量,其操作为: f ( h s 1 , h s 2 ) = [ h s 1 , h s 2 , h s 1 ⊙ h s 2 , ∥ h s 1 − h s 2 ∥ ] , ⊙ f(h_{s1},h_{s2})=[h_{s1},h_{s2},h_{s1}\odot h_{s2},\|h_{s1}-h_{s2}\|],\odot f(hs1,hs2)=[hs1,hs2,hs1hs2,hs1hs2],代表 elementwise multiplication.

 上损失函数:

KaTeX parse error: No such environment: equation at position 8: \begin{̲e̲q̲u̲a̲t̲i̲o̲n̲}̲\begin{aligned}…

 其中, z B , z S z^{B},z^{S} zB,zS分别为teacherstudentlogits,即预测值, t i t_{i} ti为真实one-hot类别向量 t t t为第i个元素,对于无标签元素, t i = 1   i f   i = a r g m a x y B   e l s e   0 t_{i}=1\ if\ i=argmaxy^{B}\ else\ 0 ti=1 if i=argmaxyB else 0

 论文中作者还提出了一些nlp领域的数据增强技术,可以看原文了解一下。

2.3 DistilBERT

这篇文章没啥难理解的地方,记录一下就行了。

 效果:模型尺寸降低40%,保留97%的泛化能力,提升了60%的速度。

 模型:teacher model为标准的Bertstudent modellayers=teacher model layers/2Bert,从teacher modellayers中每隔2层取一层初始化student modellayer

 损失函数:公式(1)的cross entropymasked language modeling loss,外加两模型的首层的隐状态的cos loss

2.4 BERT-PKD

PKD(patient knowledge distill),其teacher model为标准BERT,而student model也是BERT,不过其堆叠的层数要少于teacher。先上图为敬:

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-LrC5KmQ9-1632812843154)(data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAkGBxETEhUTEhMWFRUXGBYTGBgWFxgaGhohGBkWGhoaGh0aHSoiGBolHhkYIzEhJSkrLi4xFx8zODMsNygtLisBCgoKDg0OGxAQGy0mICUvLS0vLTAtLS01LS0tKy0tLS0vLS0yLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLf/AABEIAO0A1QMBIgACEQEDEQH/xAAbAAACAwEBAQAAAAAAAAAAAAAABQMEBgIBB//EAEYQAAIBAgQDBQMICAUDBAMAAAECEQADBBIhMQUiQRMyUWFxBoGRIzNCUlOTodMUFRZicoKx0nOSsrPBNEPiJGOi8aPR4f/EABkBAAMBAQEAAAAAAAAAAAAAAAACAwEEBf/EACgRAAMAAgICAgEEAgMAAAAAAAABAgMRIUESMQRRYRMUIvBCsXGBof/aAAwDAQACEQMRAD8A+40uv8Uyi4QhItsLbGYMwp0HUcwHjPSNaY1BcwdppLW0JaM0qDMRE6a7D4CgCNcZz21yxnV366ZSg2IBk5xvHWof15hoJ7QaAHZtjkIO3g6f5h41HihGIs2xh1ZCt0l+Tk1U6AmYJiYHVfA1c/V9mI7JI2jIsem1a1oxMs0UUVhoUUUUAQYnFBCggkuSqgR0BY7kACBVX9brlutkf5IMzDknlUNAAbcg6TA0q5fsK8ZhMHMDsQYIkEagwSNOhI61VxuFVbd0pbzkq/JMBpG2pgTAFAEt7iFpIDuFJAMHfUMR7+Vv8p8KlsX1cEqZAJU+o0I9x0qlw/CI9tLj2srsilgWLEaHTNJkDMwGugY7SauYfDJbBCKFBMmPQD+gA9ABQ1oCaiiigCHF4lbdt7jmFRWdjvAUEk/AVWbiMXEttbYFwzCSp0U2gdidZuD/ACt5TeYA6HUVXGAtCCEAyggR0ByyPTlXTblHhQBwOIJ8pmlcj9n4ycivyganRvwNdYfiNpzlRwx12/dgH1AJGtUsAqXXvh7GXLciW1z8qHP5HQe4L6BhbwdsHMEAMzMdYifWNPTSta0YnsnooorDQooooArXMYoc2wCzAKzARoHLKpMkblW2+qahw/Fbb20uKCVcoq936YBBMHTcedWHwiF88c0BSQSJAJIBg6wSYnbMY3NLeK2ksWUVLT3ALllQqu8qAVUGc06AaCYJjxJrUtvSMb0MWxtoEg3FBG+o031PgJBE+R8KnVgRI2ql+qLH1OubvNvJM76GWYz4sT1NWsPZVFCIIVRAA6AbCsNJKKKKACluL4wqOUFt3KxmK5IBInLzODMEH+YUyNYD9FxBzsHAuF2aSTyt2ZVkiCCvay3hBEbCluvFbFp6NE/GJdW7C7ADDe19LL/7nlUw4+uma1dUSAWPZwJMSYuEwOpisw1vF5ozgZizToVUA2gFMpvHae8+VdpaxG9xky/SBIIy5rkjuCOQpr1ykfvVP9V/gTyZu6Sr7QpCkowzJau9MoW6LhUs2y62yJMCSNda54ZbxhCE3LYt9noGRjcmRlznMJbLvEak6HerCcLIGUCwAABAsmIAgDv7AaR4V0OfyUTLK49ZuZuUW3W3O8llQiAP41HrXOH4rYcqEcMW7sTrChvD6pB94qJ+HsSSRZJYgkm0dSIAJ59xlXXyFAwLggqLII2ItkRoRpzabn4ms1+Q2WOI40WlBjMSQqrMSTrv5AEnyU0vfjDkEdkv3n/hSridnFf+nN9kLagi1KqTqWWSZzEBW6D5JxoGqgbWLUd4EDM2kE/QyrqstHPJ3OnunkpzWkTdM0OG4rcVFXsl0UD5w9BH1KZcNxwuqTGVlOVlmY0BEHqCCD/9VjlTFHLzCOQnuhozyQeXfJA+PrTf2ZW72hLxITLcI2JzTbjwIXMSOnaDfesi3T5Nmns0tFFFUKHhMUpXjykSLV0g6g/JiR0MFwR7xNSe0mb9HfLoDAc+Ck8x9I38p61llXFAiNQRaVgxBIYH5R5J7pWAB4jYTSXfiJVND/D8XytcJs3OZww1tfURftPFTV7BcUW4+TK6NBYZsuoBAMZWO0jfxrH2ruL+qDEKZCgz2ckg5oPOQDEbGJ6WsGb4a3mjOGtZDpq0EXJA+jlzmfCdJGuLI29PQqpmqx2N7OORmkMYXL9ESe8RrVZOOWytxlDHswGI0DEEtBAJmCFkEwCDodCBHbw2IuKf0hLLEFwAC+XKdIgjWRoZ86mbBsQwNu1DaNzNrqTHd2kkx5nxqzkpssXuI2kJVnAI0jXc5dPXnTTfmFTYe+rrmQyNdfQwaoHAkzNm0cwIMu2oO8ymswJneBXOIGKRQLFqzJcE5rjRB7xPJM+evoaFOw2T8Q4mtpguVnYgtC5dAIEnMRE9PGD4UvxfFi6gCzc71ttTb+i6sfp+ApTxm3fN69BhynLB0UkDsmP1kBDyPEPoZEVb93FAzGmYqICnc2wGjN/iHcedRrI5rgm6Zpjx3/2Lnxtf30zw19XRXXusAw9/9DWIZsXryroDAGWCc5jUmQMmXXxJ0O1ab2cnI/1DcbJ+Gf3Z8/49CKIt17GmnsbUUUVQcKWYvgyu5cXLiFokLkgkCM3MhMwAN/oimdJMR7QrJFpDc/eJyofQwSfULB6GtUuuEha12VcVwy4t60iveZHz53mwMsDSAUk+dXf1Auma7dYSCVPZQYMwYtgweomlq8ee4Q6i0csiFcsNdCCQPLwq5a9oWHzlqB422zx5kFVPuEnyp3hfGpQi8R9RUdi8rqGUypEgipKmVCiiigCDGYVbi5WmJB0MEEGQQaoXuDIFJU3GIBIXPEmNBJ2mrPEeJ27MZpZjMKoljG51gAeZIFKbvtK4IAt2xOwa7DH+UIfwJpljddCU57JOEcM7Sxbe4XV2UFgHkSRuPI7++m+Cwi2lyrOpLEkyST1J69B6ACkeF41cRVTsVyqAoIumYAjY24J94ptw7ilu7IEqwElWABjxEEhh5gmJ1rbxtNvQS5L1FFFIOeOoIIIkHQg9aTrwEAAC9cgaCQh/ErJ9TTTFYlLal3MKNz6mAABqSTAAGpJpJifaWAStoBR1u3Anv0DaesGtUOuha8eyLh2Ae415Wa6gtvkUkWuYZVM93Tc6eBHnTTBcJW2+cuzkAqM2WBMSRlA10Gvr4mlNjjV0FmFpGDkP860d1V0PZmRCg++mOD46jMFdTbYkAEwVJOwDDY9BmAkmBNNWLXKQs+I2ooopCgUUUUAUsfw1bpVszIygiVy6gwYOYEHUD8fE0o4xw25bQNae67F7akE2RozAE6pvrA8yD0pnjuM27ZKwzuN1QAkddSSFUxBgmdaVYnjzPyi2uhR4N3m5WVtQEMbDr1qk422m1snXiXzwIfbXf/xfl0zw9lUVUUQqgKB4ACBSa37Q/XssPNGDAeubKfgDTfCYpLq5kaRt1BHkQdQfI60rhz0NPj0TUUUUox4RWYu8DvW9LYFxAIGuVwBsCDyt6yPStRXF68qDMzBR4sQB8TTzbn0LUp+zA4rg5JCPauBn7QqJsyZIZsvMdqaYPg2IgKEFsCdbjKSJJOi2yQY8JX1phisdg2vWrhuWSUDw2dDlkDrOk603w2Lt3Nbbq4/dYN/Q1SsjS9CKUznh+EFq2EBJiTJ3JYlmPvJJqxRRUG9lQooooAUcY4U1xu0tkB8oUhpysASRqJKkFjrB322IQY/hN4jmtOAA4LI1qOZYJ5mnQeQrbVHftKylWAZWBUg7EHQg+Iqk5WuBHCfJiOG8IufOW7dwq4LAlrIWHykbGYAGmhjMa0fCOEsj9pcIzZSgVZIAJUmSQCx5R0Eee9XOD2lWxaVQAAi6DbYGrlbeRvaQTC9hRRRUhynxTA9smXNlIIZTEwR4jqIkEeehB1rNY7hF8o1trZIZWUtaZDAYESM8Gf5T762NV8TjbVv5y4ifxMF/qapFtcIWpT5MMnCS7syJdZlcZgOzEMCbmUlo6OOuwXwp3heC3XI7QC2kgkSC5gzGnKoPjJMTsdRY4bjcHbe+Vu2lz3A051GbkSTqdeYt+NPLbhhKkEHYgyKe8jXoWZTOqKocXxVxFTswpZmKANseRyBuI1UD31VfiV0LiWPZ/IsFXlaDNu08nm11cjpt7qgUHNFK7/G0UtClgusjUaZNRGpHNAiZKMNxVzBYoXFzAEakQfIx6H1EjwJoAScQ4RdDs9sB1YlssgOCd4nRhPiRG2tI8fgHks4u282S3opOvygUShIJLXPHdRW/pLx+7hnVVuPblblpgC4BBzqCd+gJq8ZHwmTqF7QkwPC74kLbuNmYtLkKFmNOZswUR0BrScG4ebStmILOcxjYaAADx0G538tALljEo4lHVh+6Qf6VLSVkbWhlKXIUUUVMY8NYfUv8t8/9LP3h45Z2SdsulbmosRh0cZXVXHgwBHwNUx34i1OzAm5iFBhS5l4nKNm5NiN1191TkiFNwBX1Cx35k9zKc0kawpnWnuL4Jb7ezkS0tuLmdMne0EbaCN9qb4bA2rfzdtEnfKoWfgKq8qRNQ2R8JNzsU7XvwZneJOXNH0ssT5zVyiq1zH2VJDXbYK6MC6iNAdddNCD7xXOyxZoqO3iEYwrKTAeAQTBmG9DB18qkrACvDXteGgCvwz5m3/An+kVZqtwz5m3/AAJ/pFWa1+zF6CiiisNF3HmuCyezzTIzZO9l+lljWfTWJjWKyN35t2sQWhiCsHMwBiT1JO81v6qYnhti4cz2kZvrFRm+O9VjJ4rQlTsxP6ReDR2crmAnrGZgW2jYBv5vKreGLLdHYfOkqSF0BE6m4B9CJ5iNOmsU24XwZM9/tUR1FyEHMQFyIcpDGDuD6k07w+HRBlRVQeCgAfAVSsqXAqh+ylg7uLIPaWrQOYxF1ojp/wBsz+HoKn7S/wDZ2/vW/Kqe/fRBmdgo0EsQBroNTUP6xsQx7W3CAs5zrCgaktroB4moNr6KaPO0v/Z2/vW/KqDG4jFqAbdm2xzAEdqRp1MlBH4+hpkKKE19Br8mP4s7PdZL3ictsnkK6QQNrnmTMEkaUtvXLiM2RMyhVhQMon5QkSAfqoP5632Iw6OMrqrr4MAR8DSLjnBlCKbCqrdpaB57irBdQQApgTMehNWjIuJJ1HYhs82ZrlsKVYgMd4EcwJAK/wD8rUezd641s5iWUNFtm3ZYHU6sJmGO/nubNrg+HUgi0pI1BYZiPQtJFXqXJkVLQ0xphRRRURzwmszd47dua28ttDqCRmcg7HXRfSG/4rT0kxHs8uptObf7pGZB6CQR6BoHQCqY3Kf8hK30Z88ZuTmbEXFK5wMy2ehytsh0kdaYYbjN8AEOtweDrlJ8syAZfepqrjODst23ZKq/a9oc4tPlXZjm1IGY+dNcP7NkaPd03i2uTfUySzH3iD51ZuNciLy6LeH9ocOxVC2V2VmykajKcrSRpo2m+tV72Gw7G8xupnuzDGJQFbSso12PZKT6DwFOLOHRAqqoAUQPKpa5256RVbE+KNpmuMMSqdpbFowRmGXtCGUzoec7g7VAliyCD+lLo4eM6xoxbKJYwuuUDYCOoDU/orOA5EN32mtsB+jFbpIkmSAmpHMNy0g8umx1GkrMVxq+CM13LMmURAogSZz5iNPOtBjuD27gETbKiFKQIG8EEQR7tJMRNJeIcCdUZne26qrE8rrpGo0LTpV5ePr/ANJtUV8Fxa4Blt3w+UZYZUIGWBByhTpp16094Txc3G7N1AeCwK91gCAd9VOo0133OsJeE8Fa6i3kNtM4z7O55oJBkrB0E+laDhvCUtEtJdyIzNGg3hQNFEx5mBJMCsyOPXYT5f8AQwoooqBUq8SxotJmIJMhVA6k/wBBuSfAGs5jeOXwrOWVFUFiLahmAAk8z6N/lFafGYVLqlHEgx1IIjUEEagg9aRYn2bcgqLqspBBFxNSDvLKQP8A41XG41yJXl0K7PF3Vj/6k5mYSpFrvaJGib8sQD0NNcNx51I7YKVmC6yuXzIJMjxIIjwpXgOFds90L2YNq5lJOc6jnzAQJGZm67zT3B8BAIa6/aEEEADKkjUEiSSR5mNtJFUvwXDEny6PbnEcLiEhb6ZQ6mcwE9mwaBJGkiJqO/bwzW71sX7ardTs4VkGQZMnLrG1OlQDYASZMePj611UH49FeRDi7di4SWxSyQByuoGmfbmmObUEkEqp6QZrXEMLYU5sShDOTJdTBbzGw0Jk6CeggBxXF2yrRmUNBDCRMEbET1oXj2HIlx/HGzFbKqY0LtJE+CqO9GxMj30nxXGbhJW5eyZcj6Kir3iVgvPVDpP0TT3H8DzMXtvkZtWBGZSfGJBUnyMdYnWs9xvh5sjPe7GGKWpN1xM5lGmTQAO5Ou0+FXx+D0l7J15dl7Dccvbrct3QDBkayOmZDCn+U0+4XxAXlJgqynKynWDAOh6qQdD/AEIIpJhfZ26J1t25OYwXuSTvvlg/Gn3DsAllcqySTLM27GAJMQNgNAAKTJ4a49mz5b5LVFFFRKBSjE8ftgkW1a4RIJEBQR+8Tr/KDTYisrc4NetAKq9qgAVSpUNA2zKxAkDqCZ8BtVMalv8AkJTa9Hf6/uMystq2QuYaXid48LWm1XbXtCP+5aZfNDnA+EN8FNZm/wALY8pF5Se0Ii208zZjECDH/NMcHwvEEBRbIGvNcIUb+AJbToIG24qzmO/9ibo1tq4rKGUhlIBBBkEHYg9RXdVuG4TsrapMxJJ2ksSzGOgknTpVmuZlgooorACvDXteGgCvwz5m3/An+kVZqtwz5m3/AAJ/pFWa1+zF6CiiisNPGYASTAGsmldzj9j6Ja5/ApIPoxhT7jUvG8G121lWJlWg7NBmD/X1ArKcRsuUdGR7bFSAWUwCQYOZZXTfQ1XHE0uRKpod4bjihrhNq4AzBh3DpkRejzuppng+K2bpyo3NvlYFW03IDAEjzGlYZc+blvKQGHKsHTMTlgLPdKj3edN8NgLt0rlRkAYNncFcsGeVW5i3hpHiehpWOf8AgVWzQcXN7KosmGLQTEgcjwToYGbLVV3xAXEnM5KsBa5F1HZ2jKjLzc5cddvKrGCweIUEPiM5LEg9mo0Owjy2qfsbv2o/yD/91BpfZRMq3+IXQWyWWYDUSCCe5sD1MuIMdyTowNXMFeZll1ymSI9DuJ1j1APkNq57K99ov3f/AJVXxuGxTAC3fRDmBJ7KdBuILaz7vWhJPsG/wWcZj7VqO0cAnYbsY3hRqfcKVY3jltlARbjc9tpy5dFdWPfIOwPSqXEsHcS47srMGM51BbTorASVy6jwjWZJFIsTc52KXkWVUANc2I7SeUyBJNvpMKdutoxrSfsnVv0bO3x+x9Isn8aMAPVgMo95pmjggEEEHUEag+YrBYC6xzDMLpzHL2YzkDSAci6HfWtV7PYN7aNnGXM2YJIOXQTMaAkySBprO5NLkxqVwbNNsa0UUVEoFFeGsYcZcvAM7tr9BSVVfFSBBYg6HNOoOg2DxDoWq0aPGYJGxFi4QcyC5lIZgBIAOgMGfOmNfNHvWlBL2VJBfuqn0Gy/SI1O9M8JdIAa2z299A2ggkGV1Q7bxVXi32Ir10bild7jaqzr2dwlCFMBI1yRqW0767wTrAMGIOH8YvPkBwzkMjN2ilAjEMAMoZtAw5hJ22zb1ZYAhgcPch2DtzW9SMsH5z90fCouWiipMnXHrndGBUpbS6xbLADm4ANCdR2bT021NepxG0TAcE5skeepy+REGR0jWqt62rsWbDXCWUI3NbggZoBHawQMzb+NeDDpM/orzKtqbZ1U5gfnNeYlvUzvWaDY1rw1l8Txy9cCAI+HzLnIcLnOsELMwBpJInnXu9U+PuZYLA3BDE52Z25ROmc1VYX2I8i6Nxwz5m3/AAJ/pFWa+e4K+hYqivbIzDlOTu5QY7NtuYfjWj4Fj7huG0zZxkLgnvLBUAEjcGTE68p1PQvFrlBN9D+iiiolAoqhxvGNatZliSVWTsuYxJ/oPMisnxG84R3ZnuFVZoZjBgExlWFBO2gqkY3S2JV64NTwrCKlzEFc3NdzGWJ1Nu2dJ23+AA6Uyr5wMVbDZezIOYLmUKNSzJMhpiVPuinGGx920Vh2dSyrkcli2YgcrMZDeGseXUPWJvnYqtLo02Pxi2kzsCRKrpH0iFG5A3IqseMpku3ClwLaQ3GJABjLn0BMyV1ggecGo7OLa8vPhby5XkAlAZQgqe+OoB6j1ru7bVhcDYe4RdGV+ZNRlyx85ppppUnLRTaLl7GW0MOwU6aE66zA9TBgdYNSWrqtJUyASNPEb0ru4W205sPcaRBzMDPe3m54Mw9GI20ou4wWFlMNdOdxIUBjLCM3e1gAe4UKW+A2hvS3j+GNy2oDukXbJlIn5xY3B0kg+4dNKVcQ4ndd3VWNtUbJCxmMD6RPdBkEBYMQZ1gIsXjgrstwuQoV5N52OvaHZjpHZMd/CKrGNrVCVafB9EorC4HHM2Y27t1SrFDLloIidGLLGvhWn4Fj3uq2eMyNlzLoDoG26NrqPfpMBKxuVs1XvgZ0UUVMcKo4vhFm4czLDHdlJUnwkqRm981eorU2vQNbEx9n0n526PKbf/KTU1ngVgGWUuf32LD/AC92fOKs3I7ZNpyXPqzGa3P70bbabTrFWqZ3X2KkvoKKKKQYKKKKAIMXhLdwZbihhuJ6HxB3B8xS257PW/o3LqDwDK343FY/jTmo7/db0Ph4eenxplTXpmNJiq37PW+t2648CVH4oqn8aZYPB27Qi2oUHU+JPiSdWPma9wUdmkbZV2iNhtl0+GlTUVTftmJIKKKKUY8dQQQQCDoQdjSq77P2foF7f8Dae5WBVR6AUwxeKS2pdzCjyJ30AAGpJOkCkeJ9pSoLC2qqN2u3AsecKCI/mFPCr/EWnPZBwzhDO94XGvIEuZE1snMMqmTlSQdZ6aEdZp3g+E2bZzKst9ZiWYTvBYnKD4CBSXD8YvAswS2wch+8w+iq6GD9WZjrTDB8eViFuKbZJABnMhJ2GbQgnzAEkCZNUtWxZcjiiqfE8OblvKsTntnXaFdWI2PQHpVO3wbL2gWINhLKFiS0r2ksxjrmWTrOXyFQKDiilLYXE8oV0QBQsAk7K4O67ElD0I7PQ82lvh9u6obtWDEmRE6CBp8Z90bmSQDnG8LtXTmZebbMpKt5AkbjyMiknGOEtaQNba9cJe2hWbIgMwBOqDXoPMjpNNcfxpLbFFDXHESFiFkSMxJidtBJ1GmtKcRx535AloEFHjtSTysrajIImI99Wx+a19E68Rjb9nrf0rl1x4Fgv421U/jTSxYVFCooVRsAIFJLftC479nT9y5mPwdVH403wWMS6uZDImCCCCD4EHUHUH0IOxpbV/5DS56LFFFFTGCiiigCu5PaqNYyP4xunSI+JnwB1ixVV8VZzAm4gYAr3x1gnSfIVZVgdQZHlWsxHtFFIcHxPEMLBYWwLzACATp2DXCe9pzKV18j5Vho+opS3GAouZoYq7qAu8KsiZOhMZQdASVHWpsJxRXuG3lYGGMnyIHT1Hl4TBgAYVxe7p9D/Tyruq+JxdpdLlxFn6zBf6mgDvCnkX+FfHwHiAfiAalqthcXZaFt3EaBACuGOnvqzWsxBRRRWGlTieCF5Ms5SCGUjWCPLqNwR4E7b1nMbwXEFGQpnDApNplBgiCYuQFPvatdRTzkc+hXKZg/2eul8/YXQcwfvYfcMzfWndo9APCnGG4Fcc/KlVTQlVJLN1gmAFHjE+op7hrjEvm6PA5Y0hT4nNqTrp6aSZ6Z5aMUIXYPgtm0uVAwEs0do+7Ek/S8TU/6Ani/3lz+6rVFI6b9sbSKv6Anjc+9u/3VWxnA7VwKGN3lYOIvXdwCN80jfpBpnRQrpemHimZ/H8FuBma0QwYlirEggnU5TBzSdYMRJ1iAEPEuFvq1y3dQNkt96zEy6pHMdSbvXqF2g1vqX8bwyXLah1DAXLRgif8AuKP6Ej31SMj4TEqO0ZzAcFvrmC2mGZi/yj2wBMaDs5IGm0VpOEcP7FTLZmY5mIEDYAADwAHXxPoL9FJWR0tDKUgooopBjw1h2c3fnpZx3kbZD9UJssdDuRBk71uarYrAWrkdpbViNiQJHodx7qpjtSxanZgVv3FU/JZtX2UrorOF0ymZCg/zCrdkKqi5HYtoSVOQg+BOkiehEHwpzi+Ar29nJItkPnU3b0toIg5+WD8abYbhVhCGW2uYbMeZh/M0n8as8qSJqGUeH4nGtkzWreQoSWZmVycwCllyckrrEb/ViCw7S/8AZ2/vW/Kq3RXO6T6Kpfkqdpf+zt/et+VR2l/7O39635dW6Kzf4DRkuIYvElbf6SvYgrqEc5S0nRmG2kQkkHMe9GizEqbcG0gjmJVVAmFJAkDSTA99b9lBEESDpBpVjOCWcjdnbVWg5YLIsxoTkI0mrTlXrWibhmVts1xityyMvNBPMDlIA3UbyT7qeezV5+0ZVJa0FMySQrArCqTtpmlRoIG081jhHBbRs2zdtq7lFLZizgmBrDk706RAAAAABoANAKMmRa8QmOzqiiioFQooooAp8PibsR84ZjJvkTfLrO3e126RVyoMNbYF83VpHMTpCjqBl1B019danrWYgooorDQooooAKq8S7g/xLP8AupVqqvEu4P8AEs/7qVs+0Y/RaooorDQooooA8JrM3eO3bmtvLbQ6gkZnIOx10X0hv+K09JMR7PLvac2/3SuZB6CQR6ZoHQCqY3Kf8ha30Z88ZuTmbEXFK5wMy2ehytshESOtMMNxm+ACHW4PB1yk+WZAMvvU1VxnB2W7bslVfte0OcW3yrsxzakDMfOmuH9myNHu6bxbXJM6kElmPjqIPnVm41yTSrocYDFi7bDqCAZEHcEEqwMeBBGnhVio7FlUUKohQIAFSVzMsFFFFYAVxfPKfQ/0ruosWeRv4W/oaEBzgR8mn8K/0FT1FhhyL/CP6Cpa1+wQUUUVgFTiWNFpM0FiSFVRpJPn0G5J8Ad9qzmO43fCs5fIFBaLSqSABJ1uTmj0HpWoxmFS4hRxIMdSDpqCCNQQetI8R7OOQVFxXUggrcTUg7gspAj+Wq43GuRKVdCteNXJg4l5kCCtreSI0txMgjQ9D4UzwvHLikC6FZSQMyjKyzpJEkMPGIjwNLMBwo3nugLbU2rmUkhzJHPmAKiRmZuvietPcHwEKQ11+0IIIAGVJGoMSSSPMkbGJFUvwXDEny6HNFFFcxYKKKKACqvEu4P8Sz/upVqqvEu4P8Sz/upWz7Rj9FqiiisNCiiigCpxPHCymcqW1CwsTr6kClv7SD7G58bf99Te0/zS/wAaf81nLpOUxvBjr0/GuXPmrHSSOv4+CcktscXOPgurdjchQ3W31j97yqX9pB9jc+Nv++scOJXRlBtEk5VE5hJyXGP0IB5NtubcVJbx90xNlh3QRDTqELbgCAWI88pInap/ucn0i37bF+TccL4sLzOoRkKhW5suuYsBGUn6p+IplWb9l/nb38Fn/VerSV146dSmziyyptygooopyYiHtKvSzc+Nv+6o8R7QhkZRZuSQRqbfUR9aktru0qt8TvBVzWSWIUa5lluyzkRk0MgrHkfCuF/JyeTSS4PQXxsfim9myT2jAAHY3NBG9v8AvrpfaRZUdjcGZkSZTTOwUEw20kVkf0+7J+RaBmGzSYNvYxH0m9culMfpW/8AGsf7tutn5Nukmlywr42NS2t8G5ooortPPFfEeNLafJ2bscoeVyxqWA7zDXlNVv2kH2Nz42/76p+0H/UH/Ct/671JeI33QKUUtq0gAmQLdxgNAYlgonzrjy/IubcrX9R3YfjxWNU9/wBZoLHHgrXCbNzmYMNbf1EXXm8VNTftIPsbnxt/31kDxK5LAWScpIMZp7wAMZNiCW9B4aixhMS7GGtldGMwQNHIA16lYPxpH8nJ9IdfFxfk3fD8WLtsOAVnMIMSMrFTsY3FWaWezn/Tr/Fd/wBx6Z13L0ee1phUWKvhEZyCQqsxA35QTp8Klqlxr/p73+Hc/wBDUy9mMX/tIPsbnxt/31Di+PhlAFm53kbU2/ourH6XgKSY1mCMVnMBpAk/DrVC5xO6DAsEmHIEsCQvZ7Sni/8A8epMV58/KyPlJHo18XEvezYftIPsbnxt/wB9TYLjq3Lip2brmkAkpGgJ6MTsDWQsYy4Wg2yBmKgww0BIDGRsY9RI9accJ+fterf7b0+P5F1al6EyfHiYdLZrqKKK7DhKPF8CbyZVYKQwaSuYaeUj+tKf2du/bJ9035lWuLYHEu7G1cyqbYUDOy65MQCYG2r2jI15PIT0uCv9vnLnssoUr2jzM3iTvEcyCNzA1AWGSsc17RSMtwtSymfZu79tb+6b8yj9nbv2yfdN+ZV61gL3ZWlZyWXs85Fy5J27SWmXk7aCvBbxsiWtxl1gwZzDVeQwMs7zuB50v6GP6G/cZPs94NwlrLOzXFfMEWAhWMpc9WMzm/Cm1JrOHxgAXOsDIJmWjkz6ldT3oJ8BMzo5qiSS0iVU6e2FFFVuI2na1cW2YdkdVMkQSpAMjUaxqK0wSD2buja8kedo/mUH2bu/bW/um/Mqa9w/FFboFwyyXFQ9o4yljdKk+EZk1E93yFWMPgr4ZCz6hboJLuVJZ1Ns5JEgLmEEkiQJbvVN4Yb3oss+RLWyj+zt37ZPum/Mr1PZy5mQm8kK9tyBaInI6tE9ppMRPnVh8LjBmyXF1zkZmJ72TLMqdF54gDSJBMmu1sYzMTnXKXLd6SBCKAoKQBAJjx8Z0FhhPaRjz5GtNjiiqvDluhPliC+5jbYbaDSZq1VCQm4rwZ7tztFuKvIqQULd0uZkOPrfhVT9nbv2yfdN+ZU9nAYkOCXJXMTHaPtN7T0hrXpkPgJ6wnD8QI7RyflM2l1+6UylfPUA79TtU6xRT20VnNcrSZWHs3d+2t/dN+ZR+zt37ZPum/Mq/fwuIzOyOJK2wJZwoIZ85CGQOUrG8lem9RXrGNYMM6DVCpVoIh1ZgeTUQCB49Rrpn6GP6N/cZPsv8LwhtWghbMQWJIEd5mbaTG/jVuqGCTEZibrLl5oC7bjLus7T138av1UiFQ42x2lt0mMyskxMZgRMdd6mqHGIzW3VTDFWCmSIJBjUajXrQAi/Z279sn3TfmV5+zd3ftrf3TfmVK/D8VlcC4ZKXFX5RxBYcpJ6QTuJ6VYw2BvhlLOTAvAnO5BLXFa2csiQBmEEyBAlt6l+hj+i37jJ9lL9nbv2yfdN+ZVjAcDdLiu11WCyYFsrMqV3Lnx8K7OFxYLFbggliMzE7uhjVSAAucCIiQDMZiLYxsk501zTDE+S5QUgCNT5+NbOKJe0jKzXS02OaKq8OS6FPakFuXbbuIG6D6WY+/3AqhItUUUUAFQ4jEBMs7swUfAk/AAn3VNXF20GiRMEEeooAXXONJ2dy4oLBLb3RuMwQsGA00Mr+IqH9fHM6G0VZEa7DGJVTcEry6yVBjcK4JAOhYrgLQBUW1ykZSuUZSNdI2jU/Gvb2CtMZa2jHQ8yg7AqNx4Mw9GPjQBUHFxFwFflE+gGBLcqvy9Tow0iZ6GRPGG47bYE9MwUFCXDSqtpAmZYCI6r4imAwtvLlyLlmYyiJ8Y8aju8PtNui7g6ADZgwnyzAH3UATYe8rqrr3WAYaRoRI3qSvAK9oAKKKKACiiigAooooAKo43ia22ggmArNAJIDlgCAAS3cbQeFXqiuYZGMsoOkGRMwZE+MGY8JPjQAvxfGcqgpba4S7W8qnUlbb3BHQ5goiY7wkjWucPxrPOVBIudiQWIIILyYyzlgAg9Z8ACWQwyacokMXBIkgmZInYwSPQxUZ4fZ37K34dxfFj4eLMf5j40AVsPxdHFsrHP3hmEoApYlonTQDwOYHauv1xakAZiSQAMpG5QTrEgZ1nwnXXSrb4W2YlFOUQJUaDTQeA5R8B4VzbwdpYy20EbQoEemmlAE9FFFABRRRQAUUUUAf/Z)]

两种架构.(左)PKD-skip:每2层学习一下teacher的输出.(右)PKD-last:学习teacher网络的最后六层输出。两种结果都有注意一个细节,就是最后的一层输出不参与PT loss,而是参与DS loss

 从架构图可以看出,相比于直接学习最终的输出,PKD方法还教导student model学习中间层的输出,last方法的先验假设是认为teacher modeltop layers包含最丰富的信息以便指导student model,而skip的先验假设则是认为teacher modellower layers也包含了需要被蒸馏的重要信息,从作者的结果来看,PKD-Skip 效果slightly better,作者认为PKD-Skip抓住了老师网络不同层的多样性信息。而PKD-Last抓住的更多相对来说同质化信息,因为集中在了最后几层。

 对于BERT类模型来说,由于其输入序列长度比较大,如果学习所有的tokens,不仅computationally expensive, 也可能introduce noise,又考虑到BERT的预测是只针对*“[CLS]" token的最后一层输出,所以如果student model可以获得teacher model[CLS]的表达能力,那么它就有了teacher model的泛化能力,所以直接学习[CLS]*。

 损失函数设计:

L P K D = ( 1 − α ) L C E s + α L D S + β L P T 上 标 s 代 表 s t u d e n t   m o d e l L_{PKD}=(1-\alpha)L_{CE}^{s}+\alpha L_{DS}+\beta L_{PT}\quad 上标s代表student\ model L

  • 3
    点赞
  • 38
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值