CRFEntityExtractor文档

CRFEntityExtractor文档

简介

条件随机场(CRF)实体提取

输出

entities

需要

tokens和dense_features(可选)

输出示例

{
    "entities": [{
        "value": "New York City",
        "start": 20,
        "end": 33,
        "entity": "city",
        "confidence": 0.874,
        "extractor": "CRFEntityExtractor"
    }]
}

描述

该组件实现了一个条件随机字段(CRF)来进行命名实体识别。crf可以被认为是一个无向马尔可夫链，其中时间步是单词，状态是实体类。单词的特征(大写、POS标记等)为某些实体类提供概率，相邻实体标记之间的转换也是如此:然后计算并返回最可能的标记集。

如果你想通过自定义特性,比如pre-trained字嵌入添加到CRFEntityExtractor,您可以在CRFEntityExtractor之前将任何稠密特性添加到管道中，然后配置CRFEntityExtractor，通过在其特性配置中添加“text_dense_feature”来利用密集特性。CRFEntityExtractor自动找到额外的密集特征，并检查密集特征是否为len(tokens)的可迭代对象，其中每个条目都是一个向量。如果检查失败，将显示一个警告。然而，CRFEntityExtractor将继续训练，而不需要额外的自定义特性。在密集特征存在的情况下，CRFEntityExtractor将通过密集特征sklearn crfsuite并使用它们进行训练。

配置

CRFEntityExtractor有一个要使用的默认特性列表。但是，您可以覆盖缺省配置。以下特性是可用的:

===================  ==========================================================================================
Feature Name         Description
===================  ==========================================================================================
low                  Checks if the token is lower case.
upper                Checks if the token is upper case.
title                Checks if the token starts with an uppercase character and all remaining characters are
                     lowercased.
digit                Checks if the token contains just digits.
prefix5              Take the first five characters of the token.
prefix2              Take the first two characters of the token.
suffix5              Take the last five characters of the token.
suffix3              Take the last three characters of the token.
suffix2              Take the last two characters of the token.
suffix1              Take the last character of the token.
pos                  Take the Part-of-Speech tag of the token (``SpacyTokenizer`` required).
pos2                 Take the first two characters of the Part-of-Speech tag of the token
                     (``SpacyTokenizer`` required).
pattern              Take the patterns defined by ``RegexFeaturizer``.
bias                 Add an additional "bias" feature to the list of features.
text_dense_features  Adds additional features from a dense featurizer.
===================  ==========================================================================================

当特性器使用滑动窗口在用户消息中的令牌上移动时，您可以在滑动窗口中为以前的令牌、当前的令牌和下一个令牌定义特性。将这些特性定义为[before, token, after]数组。

另外，您可以设置一个标志来确定是否使用BILOU标记模式。

pipeline:
- name: "CRFEntityExtractor"
  # BILOU_flag determines whether to use BILOU tagging or not.
  "BILOU_flag": True
  # features to extract in the sliding window
  "features": [
    ["low", "title", "upper"],
    [
      "bias",
      "low",
      "prefix5",
      "prefix2",
      "suffix5",
      "suffix3",
      "suffix2",
      "upper",
      "title",
      "digit",
      "pattern",
    ],
    ["low", "title", "upper"],
    ["text_dense_features"]
  ]
  # The maximum number of iterations for optimization algorithms.
  "max_iterations": 50
  # weight of the L1 regularization
  "L1_c": 0.1
  # weight of the L2 regularization
  "L2_c": 0.1
  # Name of dense featurizers to use.
  # If list is empty all available dense features are used.
  "featurizers": []
  # Indicated whether a list of extracted entities should be split into individual entities for a given entity type
  "split_entities_by_comma":
      address: False
      email: True

注意1
如果使用了POS功能(POS或pos2)，那么您的管道中需要有SpacyTokenizer。

注意2
如果使用pattern特性，则需要在管道中包含regexfeatuizer。

注意3
如果使用text_dense_features，你需要在你的管道中有一个密集特性器(例如languagemodelfeataturizer)。

参考

https://rasa.com/docs/rasa/components#crfentityextractor