CRFEntityExtractor文档
简介
条件随机场(CRF)实体提取
输出
entities
需要
tokens
和dense_features
(可选)
输出示例
{
"entities": [{
"value": "New York City",
"start": 20,
"end": 33,
"entity": "city",
"confidence": 0.874,
"extractor": "CRFEntityExtractor"
}]
}
描述
该组件实现了一个条件随机字段(CRF)来进行命名实体识别。crf可以被认为是一个无向马尔可夫链,其中时间步是单词,状态是实体类。单词的特征(大写、POS标记等)为某些实体类提供概率,相邻实体标记之间的转换也是如此:然后计算并返回最可能的标记集。
如果你想通过自定义特性,比如pre-trained字嵌入添加到CRFEntityExtractor,您可以在CRFEntityExtractor之前将任何稠密特性添加到管道中,然后配置CRFEntityExtractor,通过在其特性配置中添加“text_dense_feature”来利用密集特性。CRFEntityExtractor自动找到额外的密集特征,并检查密集特征是否为len(tokens)
的可迭代对象,其中每个条目都是一个向量。如果检查失败,将显示一个警告。然而,CRFEntityExtractor将继续训练,而不需要额外的自定义特性。在密集特征存在的情况下,CRFEntityExtractor将通过密集特征sklearn crfsuite并使用它们进行训练。
配置
CRFEntityExtractor有一个要使用的默认特性列表。但是,您可以覆盖缺省配置。以下特性是可用的:
=================== ==========================================================================================
Feature Name Description
=================== ==========================================================================================
low Checks if the token is lower case.
upper Checks if the token is upper case.
title Checks if the token starts with an uppercase character and all remaining characters are
lowercased.
digit Checks if the token contains just digits.
prefix5 Take the first five characters of the token.
prefix2 Take the first two characters of the token.
suffix5 Take the last five characters of the token.
suffix3 Take the last three characters of the token.
suffix2 Take the last two characters of the token.
suffix1 Take the last character of the token.
pos Take the Part-of-Speech tag of the token (``SpacyTokenizer`` required).
pos2 Take the first two characters of the Part-of-Speech tag of the token
(``SpacyTokenizer`` required).
pattern Take the patterns defined by ``RegexFeaturizer``.
bias Add an additional "bias" feature to the list of features.
text_dense_features Adds additional features from a dense featurizer.
=================== ==========================================================================================
当特性器使用滑动窗口在用户消息中的令牌上移动时,您可以在滑动窗口中为以前的令牌、当前的令牌和下一个令牌定义特性。将这些特性定义为[before, token, after]
数组。
另外,您可以设置一个标志来确定是否使用BILOU
标记模式。
pipeline:
- name: "CRFEntityExtractor"
# BILOU_flag determines whether to use BILOU tagging or not.
"BILOU_flag": True
# features to extract in the sliding window
"features": [
["low", "title", "upper"],
[
"bias",
"low",
"prefix5",
"prefix2",
"suffix5",
"suffix3",
"suffix2",
"upper",
"title",
"digit",
"pattern",
],
["low", "title", "upper"],
["text_dense_features"]
]
# The maximum number of iterations for optimization algorithms.
"max_iterations": 50
# weight of the L1 regularization
"L1_c": 0.1
# weight of the L2 regularization
"L2_c": 0.1
# Name of dense featurizers to use.
# If list is empty all available dense features are used.
"featurizers": []
# Indicated whether a list of extracted entities should be split into individual entities for a given entity type
"split_entities_by_comma":
address: False
email: True
注意1
如果使用了POS
功能(POS或pos2),那么您的管道中需要有SpacyTokenizer
。
注意2
如果使用pattern
特性,则需要在管道中包含regexfeatuizer。
注意3
如果使用text_dense_features
,你需要在你的管道中有一个密集特性器(例如languagemodelfeataturizer
)。
参考
https://rasa.com/docs/rasa/components#crfentityextractor