文章目录
bert理解
一句话概括, bert就是一个抽取器。输入一句话(词序列),输出抽取后的embedding序列。
再简单理解就是,它就是一个 encoder。
简单机理
我们可以用transformer在语言模型上做预训练。因为transformer是encoder-decoder结构,语言模型就只需要encoder部分就够了。BERT,利用transformer的encoder来进行预训练。
那么什么是transformer?
这是一个新的训练结构,发展历程而言就是CNN,RNN,transformer; transformer是基于attention机理发展而来。
transformer由编码器和解码器组成。编码器和解码器都是基于attention机制。如下图
什么是注意力机制,一图简单领会,后面我们单独开一篇动手实践一下
注意力机制就是,当前词的含义,必须结合结合上下文才能更好的理解。
encoder输入输出
-
输入会加入特殊的[CLS]代表整句话的含义,可以用于分类。
-
input的词help,prince,mayuko等,一共512,这是截取的最大长度。
-
然后经过12层的encoder
-
最后输出的是每个token对应的embedding序列,每个token对应一个768维的向量。这个应该很好理解。
输出的结果
out = bert(xx)
Return:
:obj:`tuple(torch.FloatTensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs:
**last_hidden_state** (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):
Sequence of hidden-states at the output of the last layer of the model.
**pooler_output** (:obj:`torch.FloatTensor`: of shape :obj:`(batch_size, hidden_size)`):
Last layer hidden-state of the first token of the sequence (classification token)
further processed by a Linear layer and a Tanh activation function. The Linear
layer weights are trained from the next sentence prediction (classification)
objective during pre-training.
This output is usually *not* a good summary
of the semantic content of the input, you're often better with averaging or pooling
the sequence of hidden-states for the whole input sequence.
**hidden_states** (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_hidden_states=True``):
Tuple of :obj:`torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer)
of shape :obj:`(batch_size, sequence_length, hidden_size)`.
Hidden-states of the model at the output of each layer plus the initial embedding outputs.
**attentions** (:obj:`tuple(torch.FloatTensor)`, `optional`, returned when ``config.output_attentions=True``):
Tuple of :obj:`torch.FloatTensor` (one for each layer) of shape
:obj:`(batch_size, num_heads, sequence_length, sequence_length)`.
Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
heads.
作用
有了词序列对应的embedding向量,就可以对词分类、句子向量构建,句子分类、句子相似度比较等。
code(notebook)
#%% md
# bert
#%%
!pip install transformers
#%%
import torch
from transformers import BertModel, BertTokenizer
#%%
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
#%%
input_ids =