Bert模型冻结指定参数

bert模型做微调有的时候就需要只训练部分参数，那么就需要把其他的参数冻结掉，固定住，又能微调bert模型，还能提高模型训练的效率。这个就需要用到parameter的requires_grad的属性，来冻结和放开参数。

比如说现在我们要放开第11和12层以及bert.pooler和out层参数，冻结其他的参数，怎么实现呢？

pytorch中有 model.named_parameters() 和 requires_grad，直接写一个遍历然后设置就好。具体实现代码：

import torch.nn as nn
from transformers import BertModel
import torch
 
class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.bert = BertModel.from_pretrained('pretrained_models/Chinese-BERT-wwm')
        self.out = nn.Linear(768,2)
 
    def forward(self):
        out = self.bert()
        return out
 
if __name__ == '__main__':
    model = Model()
    unfreeze_layers = ['layer.10','layer.11','bert.pooler','out.']
 
    for name, param in model.named_parameters():
        print(name,param.size())
 
    print("*"*30)
    print('\n')
 
    for name ,param in model.named_parameters():
        param.requires_grad = False
        for ele in unfreeze_layers:
            if ele in name:
                param.requires_grad = True
                break
    #验证一下
    for name, param in model.named_parameters():
        if param.requires_grad:
            print(name,param.size())
 
    #过滤掉requires_grad = False的参数
    optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=0.00001)

参考：https://blog.csdn.net/HUSTHY/article/details/104006106
https://www.zhihu.com/question/311095447/answer/589307812