bert网络结构

最新推荐文章于 2024-05-06 16:39:30 发布

ox180x

最新推荐文章于 2024-05-06 16:39:30 发布

阅读量855

点赞数

文章标签： bert transformer 自然语言处理深度学习人工智能

本文链接：https://blog.csdn.net/ox180x/article/details/124095361

版权

这里以BertForMaskedLM为例，记录下BertModel的网络结构和一些思考，cls那部分网络结构不涉及，即BertOnlyMLMHead那部分。

BertForMaskedLM(
  (bert): BertModel(
    (embeddings): BertEmbeddings(
      (word_embeddings): Embedding(21128, 768, padding_idx=0)
      (position_embeddings): Embedding(512, 768)
      (token_type_embeddings): Embedding(2, 768)
      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (encoder): BertEncoder(
      (layer): ModuleList(
        (0): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )

        # 这里重复11次，一共12层结构。
      )
    )
  )
  (cls): BertOnlyMLMHead(
    (predictions): BertLMPredictionHead(
      (transform): BertPredictionHeadTransform(
        (dense): Linear(in_features=768, out_features=768, bias=True)
        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
      )
      (decoder): Linear(in_features=768, out_features=21128, bias=True)
    )
  )
)

1. position_embedding和transformer里面的位置编码不一样

transformer里面的positional embedding，是正余弦绝对位置编码，对于sequence分配不同的position id。而bert是学习出来的，分配一个(512,768)embedding。

随后word_embedding，positional embedding, token_type_embedding进行相加，然后经过LN、dropout获得最终embedding。

2. BertIntermediate是用来干嘛的

在transformer结构里，是没有这个中间层的，细看下，是将768扩大4倍，随后在BertOutput那里又降维到768。那bert加这一个linear的作用是干嘛的呢？

搜了下网上的看法，很少关注或者没有相关思考。但是这这篇论文：Undivided Attention: Are Intermediate Layers Necessary for BERT? 给出了自己的解释。作者观点如下：

In recent times, BERT-based models have been extremely successful in solving a variety of natural language processing (NLP) tasks such as reading comprehension, natural language inference, sentiment analysis, etc. All BERT-based architectures have a self-attention block followed by a block of intermediate layers as the basic building component. However, a strong justification for the inclusion of these intermediate layers remains missing in the literature. In this work we investigate the importance of intermediate layers on the overall network performance of downstream tasks. We show that reducing the number of intermediate layers and modifying the architecture for BERT-Base results in minimal loss in fine-tuning accuracy for downstream tasks while decreasing the number of parameters and training time of the model. Additionally, we use the central kernel alignment (CKA) similarity metric and probing classifiers to demonstrate that removing intermediate layers has little impact on the learned self-attention representations.

结论就是实验后发现降低网络复杂度后，同时也能保持微调任务的准确性。

ox180x

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
bert网络结构

这里以BertForMaskedLM为例，记录下BertModel的网络结构和一些思考，cls那部分网络结构不涉及，即BertOnlyMLMHead那部分。12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849BertForMaskedLM(...
复制链接

扫一扫