【论文】命名实体识别

✨✨ 欢迎大家来访Srlua的博文（づ￣3￣）づ╭❤～✨✨

🌟🌟 欢迎各位亲爱的读者，感谢你们抽出宝贵的时间来阅读我的文章。

我是Srlua小谢，在这里我会分享我的知识和经验。🎥

希望在这里，我们能一起探索IT世界的奥妙，提升我们的技能。🔮

记得先点赞👍后阅读哦~ 👏👏

📘📚 所属专栏：​​​​​​传知代码论文复现

欢迎访问我的主页：Srlua小谢 获取更多信息和资源。✨✨🌙🌙

​​

​​

目录

主要目标

应用

论文介绍

技术方法

BERT编码

BiLSTM特征提取

注意力机制

CRF标注

LoRA

数据处理

数据来源

数据标注

BIO标签定义

核心逻辑

BaLC模型

LoRA结构

实现方式

演示效果

---

 本文所有资源均可在该地址处获取：地址

概述

实体识别任务（Named Entity Recognition，简称NER）是自然语言处理（NLP）中的一个基本任务，旨在从文本中识别和分类命名实体。命名实体通常包括专有名词，如人名、地名、组织名等。

下图展示了一个简单的实体抽取任务，在句子中抽取出来阿里巴巴（组织名）、马云（人名）和杭州（地名）三个实体。

主要目标

NER 的主要目标是找到文本中有意义的实体，并将其归类到预定义的类别中。以下是一些常见的类别：

• 人名（Person）：例如“乔布斯”、“马云”。

• 地名（Location）：例如“纽约”、“长城”。

• 组织名（Organization）：例如“微软”、“联合国”。

应用

命名实体识别是自然语言处理领域的一个重要的任务，它在很多具体任务上有着自己的应用：

• 信息抽取：从大量文档中自动提取有价值的信息。

• 问答系统：帮助系统更准确地理解问题并返回相关答案。

• 文本摘要：在生成文本摘要时识别出关键实体以保留重要信息。

• 推荐系统：通过识别用户偏好的实体来提供个性化推荐。

论文介绍

本文的工作启发于论文BERT-BiLSTM-CRF Chinese Resume Named Entity Recognition

Combining Attention Mechanisms

BERT-BiLSTM-CRF Chinese Resume Named Entity Recognition Combining Attention Mechanisms | Proceedings of the 4th International Conference on Artificial Intelligence and Computer Engineering

技术方法

BERT编码

首先，将输入的中文文本通过预训练的 BERT 模型进行编码，生成每个字的上下文表示。BERT模型通过其双向Transformer架构，能够捕捉文本中每个字与其前后文之间的复杂关系，从而生成高质量的字级别表示，有助于后续的特征提取和实体识别。

BiLSTM特征提取

接下来，将 BERT 输出的特征向量输入到双向长短时记忆网络（BiLSTM）中，以捕捉序列中的前后依赖关系。BiLSTM网络能够从两个方向处理序列数据，即从前向后和从后向前。使得模型可以充分利用上下文信息，对每个字在整个序列中的位置和角色进行更准确的建模，从而提取出更丰富的特征表示。

注意力机制

在 BiLSTM 层之后，引入注意力机制，以便模型能够聚焦于更相关的特征。注意力机制通过计算序列中各个字之间的相关性权重，使模型能够动态地调整对不同位置的字的关注程度。

CRF标注

最后，将经过注意力机制处理的特征向量输入CRF层，进行全局序列标注，输出最终的实体识别结果。CRF是一种用于序列标注的概率图模体型，它考虑了标注序列的全局依赖关系，从而在预测每个字的标签时，不仅依赖于当前字的特征，还综合考虑其邻近字的标注情况。

论文提出的BERT-BiLSTM-Att-CRF模型在中文数据集上取得了较好的识别效果。

结合论文提出的框架，本文新增了一个LoRA层，用来优化模型

LoRA

神经网络包含许多密集层执行矩阵乘法。这些层中的权重矩阵通常是满秩的。当适应特定下游任务时，研究表明：预训练语言模型拥有较低的内在维度，也就是说，存在一个极低维度的参数，对它进行微调，和在全参数空间中进行微调，训练效果是相近的。受此启发，在参数更新过程中，应当也存在一个相对较低的“本征秩”。对于预训练的权重矩阵，通过低秩分解来约束其更新。在涉及到矩阵相乘的模块，增加一个新的通路，通过前后两个矩阵A,B相乘，第一个矩阵A负责降维，第二个矩阵B负责升维，中间层维度为r，从而来模拟本征秩。

数据处理

数据来源

本文所用的训练数据是MSRA-NER数据集。

MSRA-NER是由微软亚洲研究院标注的新闻领域的实体识别数据集。该数据集包含5万多条中文实体识别标注数据，实体类别分为人物、地点、机构三类。

数据集包含训练集46364个句子，验证集4365个句子。

格式举例如下：

<span style="background-color:#f8f8f8"><span style="color:#333333"><span style="color:#000000">中</span> <span style="color:#000000">共</span> <span style="color:#000000">中</span> <span style="color:#000000">央</span> <span style="color:#000000">致</span> <span style="color:#000000">中</span> <span style="color:#000000">国</span> <span style="color:#000000">致</span> <span style="color:#000000">公</span> <span style="color:#000000">党</span> <span style="color:#000000">十</span> <span style="color:#000000">一</span> <span style="color:#000000">大</span> <span style="color:#000000">的</span> <span style="color:#000000">贺</span> <span style="color:#000000">词</span> <span style="color:#000000">各</span> <span style="color:#000000">位</span> <span style="color:#000000">代</span> <span style="color:#000000">表</span> <span style="color:#000000">、</span> <span style="color:#000000">各</span> <span style="color:#000000">位</span> <span style="color:#000000">同</span> <span style="color:#000000">志</span> <span style="color:#000000">：</span> <span style="color:#000000">在</span> <span style="color:#000000">中</span> <span style="color:#000000">国</span> <span style="color:#000000">致</span> <span style="color:#000000">公</span> <span style="color:#000000">党</span> <span style="color:#000000">第</span> <span style="color:#000000">十</span> <span style="color:#000000">一</span> <span style="color:#000000">次</span> <span style="color:#000000">全</span> <span style="color:#000000">国</span> <span style="color:#000000">代</span> <span style="color:#000000">表</span> <span style="color:#000000">大</span> <span style="color:#000000">会</span> <span style="color:#000000">隆</span> <span style="color:#000000">重</span> <span style="color:#000000">召</span> <span style="color:#000000">开</span> <span style="color:#000000">之</span> <span style="color:#000000">际</span> <span style="color:#000000">，</span> <span style="color:#000000">中</span> <span style="color:#000000">国</span> <span style="color:#000000">共</span> <span style="color:#000000">产</span> <span style="color:#000000">党</span> <span style="color:#000000">中</span> <span style="color:#000000">央</span> <span style="color:#000000">委</span> <span style="color:#000000">员</span> <span style="color:#000000">会</span> <span style="color:#000000">谨</span> <span style="color:#000000">向</span> <span style="color:#000000">大</span> <span style="color:#000000">会</span> <span style="color:#000000">表</span> <span style="color:#000000">示</span> <span style="color:#000000">热</span> <span style="color:#000000">烈</span> <span style="color:#000000">的</span> <span style="color:#000000">祝</span> <span style="color:#000000">贺</span> <span style="color:#000000">，</span> <span style="color:#000000">向</span> <span style="color:#000000">致</span> <span style="color:#000000">公</span> <span style="color:#000000">党</span> <span style="color:#000000">的</span> <span style="color:#000000">同</span> <span style="color:#000000">志</span> <span style="color:#000000">们</span> <span style="color:#000000">致</span> <span style="color:#000000">以</span> <span style="color:#000000">亲</span> <span style="color:#000000">切</span> <span style="color:#000000">的</span> <span style="color:#000000">问</span> <span style="color:#000000">候</span> <span style="color:#000000">！</span>
<span style="color:#000000">B</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">O</span> <span style="color:#000000">B</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">B</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">B</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">B</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span> <span style="color:#000000">O</span></span></span>

数据标注

采用BIO标注方式对获得的文本句子进行标注

BIO数据标注方式是命名实体识别（NER）任务中常用的一种标注方法。BIO代表三种标签：B（Begin），I（Inside）和O（Outside），用于标记文本中每个词属于某个命名实体的开头、内部或外部。以下是对BIO标注方式的详细介绍：

BIO标签定义

• B（Begin）：表示命名实体的开始。一个实体的第一个词标注为B-<实体类型>。

• I（Inside）：表示命名实体的内部。属于同一个实体的后续词标注为I-<实体类型>。

• O（Outside）：表示不属于任何命名实体的词。

例如，B-ORG表示组织实体的开头，I-ORG表示组织实体的内部。下图展示了一个标注好的例子，其中未标注的字段都是无实体（O）。

核心逻辑

BaLC模型

自定义attention层，bert模型、LSTM模型、CRF模型调用pytorch库中的相关模型，使用BERT预训练的语言模型对输入文本进行字符级编码，获得动态词向量，然后使用双向长短期记忆（BiLSTM）网络提取全局语义特征，然后使用注意力机制分配权重，更好地捕捉关键特征，最后使用条件随机场（CRFs）输出全局最优标记序列。具体定义如下：

<span style="background-color:#f8f8f8"><span style="color:#333333"><span style="color:#000000">import</span> <span style="color:#000000">torch</span>.<span style="color:#000000">nn</span> <span style="color:#000000">as</span> <span style="color:#000000">nn</span>
<span style="color:#000000">from</span> <span style="color:#000000">transformers</span> <span style="color:#000000">import</span> <span style="color:#000000">BertPreTrainedModel</span>, <span style="color:#000000">BertModel</span>, <span style="color:#000000">BertConfig</span>
<span style="color:#000000">from</span> <span style="color:#000000">torchcrf</span> <span style="color:#000000">import</span> <span style="color:#000000">CRF</span>
<span style="color:#000000">import</span> <span style="color:#000000">math</span>
<span style="color:#000000">import</span> <span style="color:#000000">torch</span>
​
​
<span style="color:#770088">class</span> <span style="color:#0000ff">Self_Attention</span>(<span style="color:#000000">nn</span>.<span style="color:#000000">Module</span>):
    <span style="color:#000000">def</span> <span style="color:#0000ff">__init__</span>(<span style="color:#000000">self</span>, <span style="color:#000000">input_dim</span>, <span style="color:#000000">dim_k</span>, <span style="color:#000000">dim_v</span>):
        <span style="color:#000000">super</span>(<span style="color:#000000">Self_Attention</span>, <span style="color:#000000">self</span>).<span style="color:#000000">__init__</span>()
        <span style="color:#000000">self</span>.<span style="color:#000000">q</span> <span style="color:#981a1a">=</span> <span style="color:#000000">nn</span>.<span style="color:#000000">Linear</span>(<span style="color:#000000">input_dim</span>, <span style="color:#000000">dim_k</span>)
        <span style="color:#000000">self</span>.<span style="color:#000000">k</span> <span style="color:#981a1a">=</span> <span style="color:#000000">nn</span>.<span style="color:#000000">Linear</span>(<span style="color:#000000">input_dim</span>, <span style="color:#000000">dim_k</span>)
        <span style="color:#000000">self</span>.<span style="color:#000000">v</span> <span style="color:#981a1a">=</span> <span style="color:#000000">nn</span>.<span style="color:#000000">Linear</span>(<span style="color:#000000">input_dim</span>, <span style="color:#000000">dim_v</span>)
        <span style="color:#000000">self</span>.<span style="color:#000000">_norm_fact</span> <span style="color:#981a1a">=</span> <span style="color:#116644">1</span> <span style="color:#981a1a">/</span> <span style="color:#000000">math</span>.<span style="color:#000000">sqrt</span>(<span style="color:#000000">dim_k</span>)
​
    <span style="color:#000000">def</span> <span style="color:#0000ff">forward</span>(<span style="color:#000000">self</span>, <span style="color:#000000">x</span>):
        <span style="color:#000000">Q</span> <span style="color:#981a1a">=</span> <span style="color:#000000">self</span>.<span style="color:#000000">q</span>(<span style="color:#000000">x</span>)  
        <span style="color:#000000">K</span> <span style="color:#981a1a">=</span> <span style="color:#000000">self</span>.<span style="color:#000000">k</span>(<span style="color:#000000">x</span>) 
        <span style="color:#000000">V</span> <span style="color:#981a1a">=</span> <span style="color:#000000">self</span>.<span style="color:#000000">v</span>(<span style="color:#000000">x</span>) 
        <span style="color:#000000">atten</span> <span style="color:#981a1a">=</span> <span style="color:#000000">torch</span>.<span style="color:#000000">bmm</span>(<span style="color:#000000">Q</span>, <span style="color:#000000">K</span>.<span style="color:#000000">permute</span>(<span style="color:#116644">0</span>, <span style="color:#116644">2</span>, <span style="color:#116644">1</span>)) <span style="color:#981a1a">*</span> <span style="color:#000000">self</span>.<span style="color:#000000">_norm_fact</span>  <span style="color:#000000">#</span> <span style="color:#000000">Q</span> <span style="color:#981a1a">*</span> <span style="color:#000000">K</span>.<span style="color:#000000">T</span>()
        <span style="color:#000000">atten</span> <span style="color:#981a1a">=</span> <span style="color:#000000">nn</span>.<span style="color:#000000">Softmax</span>(<span style="color:#000000">dim</span><span style="color:#981a1a">=-</span><span style="color:#116644">1</span>)(<span style="color:#000000">atten</span>)
        <span style="color:#000000">output</span> <span style="color:#981a1a">=</span> <span style="color:#000000">torch</span>.<span style="color:#000000">bmm</span>(<span style="color:#000000">atten</span>, <span style="color:#000000">V</span>)  
        <span style="color:#770088">return</span> <span style="color:#000000">output</span>
​
​
<span style="color:#770088">class</span> <span style="color:#0000ff">BERT_BiLSTM_ATT_CRF</span>(<span style="color:#000000">nn</span>.<span style="color:#000000">Module</span>):
    <span style="color:#000000">def</span> <span style="color:#0000ff">__init__</span>(<span style="color:#000000">self</span>, <span style="color:#000000">bert_model</span>, <span style="color:#000000">hidden_dropout_prob</span>, <span style="color:#000000">num_labels</span>, <span style="color:#000000">hidden_dim</span><span style="color:#981a1a">=</span><span style="color:#116644">128</span>):
        <span style="color:#000000">super</span>(<span style="color:#000000">BERT_BiLSTM_ATT_CRF</span>, <span style="color:#000000">self</span>).<span style="color:#000000">__init__</span>()
        <span style="color:#000000">self</span>.<span style="color:#000000">bert</span> <span style="color:#981a1a">=</span> <span style="color:#000000">BertModel</span>.<span style="color:#000000">from_pretrained</span>(<span style="color:#000000">bert_model</span>)
        <span style="color:#000000">bert_config</span> <span style="color:#981a1a">=</span> <span style="color:#000000">BertConfig</span>.<span style="color:#000000">from_pretrained</span>(<span style="color:#000000">bert_model</span>)
        <span style="color:#000000">self</span>.<span style="color:#000000">dropout</span> <span style="color:#981a1a">=</span> <span style="color:#000000">nn</span>.<span style="color:#000000">Dropout</span>(<span style="color:#000000">hidden_dropout_prob</span>)
​
        <span style="color:#000000">self</span>.<span style="color:#000000">bilstm</span> <span style="color:#981a1a">=</span> <span style="color:#000000">nn</span>.<span style="color:#000000">LSTM</span>(<span style="color:#000000">input_size</span><span style="color:#981a1a">=</span><span style="color:#000000">bert_config</span>.<span style="color:#000000">hidden_size</span>, <span style="color:#000000">hidden_size</span><span style="color:#981a1a">=</span><span style="color:#000000">hidden_dim</span>, <span style="color:#000000">num_layers</span><span style="color:#981a1a">=</span><span style="color:#116644">1</span>, <span style="color:#000000">bidirectional</span><span style="color:#981a1a">=</span><span style="color:#000000">True</span>, <span style="color:#000000">batch_first</span><span style="color:#981a1a">=</span><span style="color:#000000">True</span>)
        <span style="color:#000000">out_dim</span> <span style="color:#981a1a">=</span> <span style="color:#000000">hidden_dim</span><span style="color:#981a1a">*</span> <span style="color:#116644">2</span>
​
        <span style="color:#000000">self</span>.<span style="color:#000000">hidden2tag</span> <span style="color:#981a1a">=</span> <span style="color:#000000">nn</span>.<span style="color:#000000">Linear</span>(<span style="color:#000000">in_features</span><span style="color:#981a1a">=</span><span style="color:#000000">out_dim</span>, <span style="color:#000000">out_features</span><span style="color:#981a1a">=</span><span style="color:#000000">num_labels</span>)
        <span style="color:#000000">self</span>.<span style="color:#000000">attention</span> <span style="color:#981a1a">=</span> <span style="color:#000000">Self_Attention</span>(<span style="color:#116644">128</span>, <span style="color:#116644">128</span>, <span style="color:#116644">128</span>)
        <span style="color:#000000">self</span>.<span style="color:#000000">crf</span> <span style="color:#981a1a">=</span> <span style="color:#000000">CRF</span>(<span style="color:#000000">num_tags</span><span style="color:#981a1a">=</span><span style="color:#000000">num_labels</span>, <span style="color:#000000">batch_first</span><span style="color:#981a1a">=</span><span style="color:#000000">True</span>)
​
    <span style="color:#000000">def</span> <span style="color:#0000ff">forward</span>(<span style="color:#000000">self</span>, <span style="color:#000000">input_ids</span>, <span style="color:#000000">tags</span>, <span style="color:#000000">token_type_ids</span><span style="color:#981a1a">=</span><span style="color:#000000">None</span>, <span style="color:#000000">attention_mask</span><span style="color:#981a1a">=</span><span style="color:#000000">None</span>):
        <span style="color:#000000">outputs</span> <span style="color:#981a1a">=</span> <span style="color:#000000">self</span>.<span style="color:#000000">bert</span>(<span style="color:#000000">input_ids</span>, <span style="color:#000000">token_type_ids</span><span style="color:#981a1a">=</span><span style="color:#000000">token_type_ids</span>, <span style="color:#000000">attention_mask</span><span style="color:#981a1a">=</span><span style="color:#000000">attention_mask</span>)
        <span style="color:#000000">sequence_output</span> <span style="color:#981a1a">=</span> <span style="color:#000000">outputs</span>[<span style="color:#116644">0</span>]  
        <span style="color:#000000">sequence_output</span>, <span style="color:#000000">_</span> <span style="color:#981a1a">=</span> <span style="color:#000000">self</span>.<span style="color:#000000">bilstm</span>(<span style="color:#000000">sequence_output</span>)  
        <span style="color:#000000">sequence_output</span> <span style="color:#981a1a">=</span> <span style="color:#000000">self</span>.<span style="color:#000000">dropout</span>(<span style="color:#000000">sequence_output</span>)
        <span style="color:#000000">sequence_output</span> <span style="color:#981a1a">=</span> <span style="color:#000000">self</span>.<span style="color:#000000">attention</span>(<span style="color:#000000">sequence_output</span>)
        <span style="color:#000000">sequence_output</span> <span style="color:#981a1a">=</span> <span style="color:#000000">self</span>.<span style="color:#000000">hidden2tag</span>(<span style="color:#000000">sequence_output</span>) 
        <span style="color:#000000">outputs</span> <span style="color:#981a1a">=</span> <span style="color:#000000">self</span>.<span style="color:#000000">crf</span>(<span style="color:#000000">sequence_output</span> , <span style="color:#000000">tags</span>, <span style="color:#000000">mask</span><span style="color:#981a1a">=</span><span style="color:#000000">attention_mask</span>.<span style="color:#000000">byte</span>())
        
        <span style="color:#770088">return</span> <span style="color:#000000">outputs</span>
​</span></span>

LoRA结构

其中，为了优化模型的算法，在Bert的encoder模块加入了LoRA方法，使得模型的训练速度得到提高，训练效果得到提高。 原则上，可以将LoRA应用到神经网络中，以减少可训练参数的数量。在Transformer结构中，自注意力模块有四个权重矩阵(Wq,Wk,Wv,Wo)(W q​ ,W k​ ,W v​ ,W o​ )，MLP模块有两个权重矩阵。通常来说，将LoRA添加到WqW q​ 和WkW k 两个模块中，效果是比较好的。因此，在Q(query)和K(key)模块，插入LoRA层。

<span style="background-color:#f8f8f8"><span style="color:#333333">    <span style="color:#770088">for</span> <span style="color:#000000">layer</span> <span style="color:#770088">in</span> <span style="color:#000000">model</span>.<span style="color:#000000">bert</span>.<span style="color:#000000">encoder</span>.<span style="color:#000000">bert_layer_groups</span>:
        <span style="color:#000000">layer</span>.<span style="color:#000000">bert_layers</span>[<span style="color:#116644">0</span>].<span style="color:#000000">attention</span>.<span style="color:#000000">query</span> <span style="color:#981a1a">=</span> <span style="color:#000000">LinearLora</span>(<span style="color:#000000">layer</span>.<span style="color:#000000">bert_layers</span>[<span style="color:#116644">0</span>].<span style="color:#000000">attention</span>.<span style="color:#000000">query</span>,<span style="color:#000000">rank</span><span style="color:#981a1a">=</span><span style="color:#116644">8</span>,<span style="color:#000000">alpha</span><span style="color:#981a1a">=</span><span style="color:#116644">16</span>)
        <span style="color:#000000">layer</span>.<span style="color:#000000">bert_layers</span>[<span style="color:#116644">0</span>].<span style="color:#000000">attention</span>.<span style="color:#000000">key</span> <span style="color:#981a1a">=</span> <span style="color:#000000">LinearLora</span>(<span style="color:#000000">layer</span>.<span style="color:#000000">bert_layers</span>[<span style="color:#116644">0</span>].<span style="color:#000000">attention</span>.<span style="color:#000000">key</span>,<span style="color:#000000">rank</span><span style="color:#981a1a">=</span><span style="color:#116644">8</span>,<span style="color:#000000">alpha</span><span style="color:#981a1a">=</span><span style="color:#116644">16</span>)
       
    <span style="color:#000000">model</span>.<span style="color:#000000">to</span>(<span style="color:#000000">device</span>)
​</span></span>

实现方式

首先下载Bert预训练模型，然后收集自己要训练的数据集，放入文件中，修改源码中的路径名称。

运行train.py函数，开始训练，可以自行调整训练中的epoch等参数，其中训练的时候会调用测试函数进行输出。

演示效果

运行train.py函数，可以看到模型开始训练，在模型训练结束后，会根据测试集的结果生成测试的结果

<span style="background-color:#f8f8f8"><span style="color:#333333">           <span style="color:#000000">precision</span>    <span style="color:#000000">recall</span>  <span style="color:#000000">f1</span><span style="color:#981a1a">-</span><span style="color:#000000">score</span>   <span style="color:#000000">support</span>
    <span style="color:#000000">B</span><span style="color:#981a1a">-</span><span style="color:#000000">LOC</span>     <span style="color:#116644">0.9257</span>    <span style="color:#116644">0.8245</span>    <span style="color:#116644">0.8721</span>      <span style="color:#116644">2871</span>
    <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">LOC</span>     <span style="color:#116644">0.8894</span>    <span style="color:#116644">0.8796</span>    <span style="color:#116644">0.8845</span>      <span style="color:#116644">4370</span>
    <span style="color:#000000">B</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span>     <span style="color:#116644">0.8625</span>    <span style="color:#116644">0.7800</span>    <span style="color:#116644">0.8192</span>      <span style="color:#116644">1327</span>
    <span style="color:#000000">B</span><span style="color:#981a1a">-</span><span style="color:#000000">PER</span>     <span style="color:#116644">0.9513</span>    <span style="color:#116644">0.9021</span>    <span style="color:#116644">0.9261</span>      <span style="color:#116644">1972</span>
    <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">PER</span>     <span style="color:#116644">0.9254</span>    <span style="color:#116644">0.9521</span>    <span style="color:#116644">0.9386</span>      <span style="color:#116644">3845</span>
        <span style="color:#000000">O</span>     <span style="color:#116644">0.9913</span>    <span style="color:#116644">0.9917</span>    <span style="color:#116644">0.9915</span>    <span style="color:#116644">150935</span>
    <span style="color:#000000">I</span><span style="color:#981a1a">-</span><span style="color:#000000">ORG</span>     <span style="color:#116644">0.8631</span>    <span style="color:#116644">0.9255</span>    <span style="color:#116644">0.8932</span>      <span style="color:#116644">5640</span>
<span style="color:#000000">avg</span><span style="color:#981a1a">/</span><span style="color:#000000">total</span>     <span style="color:#116644">0.9804</span>    <span style="color:#116644">0.9803</span>    <span style="color:#116644">0.9802</span>    <span style="color:#116644">170960</span></span></span>

运行demo.py可以根据输入的句子，进行实体识别，例如：

<span style="background-color:#f8f8f8"><span style="color:#333333"><span style="color:#000000">sentence</span> <span style="color:#981a1a">=</span> <span style="color:#aa1111">"在 唐 胜 利 康 复 回 乡 前 一 天 ， 北 京 博 爱 医 院 院 长 吴 弦 光 代 表 医 院 向 唐 胜 利 及 其 父 亲 赠 送 编 织 机 。"</span>
<span style="color:#000000">output</span> <span style="color:#981a1a">=</span>[[<span style="color:#aa1111">'<START>'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'B-PER'</span>, <span style="color:#aa1111">'I-PER'</span>, <span style="color:#aa1111">'I-PER'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'B-LOC'</span>, <span style="color:#aa1111">'B-ORG'</span>, <span style="color:#aa1111">'B-ORG'</span>, <span style="color:#aa1111">'B-ORG'</span>, <span style="color:#aa1111">'B-ORG'</span>, <span style="color:#aa1111">'B-ORG'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'B-PER'</span>, <span style="color:#aa1111">'I-ORG'</span>, <span style="color:#aa1111">'I-ORG'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'B-LOC'</span>, <span style="color:#aa1111">'B-ORG'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'B-PER'</span>, <span style="color:#aa1111">'I-PER'</span>, <span style="color:#aa1111">'I-PER'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'O'</span>, <span style="color:#aa1111">'<END>'</span>]]
​</span></span>

可以看出，模型能识别句子中的实体，并按照BIO标注返回结果

• 本文所有资源均可在该地址处获取：地址

 ​​

希望对你有帮助！加油！

若您认为本文内容有益，请不吝赐予赞同并订阅，以便持续接收有价值的信息。衷心感谢您的关注和支持！