doccano是一个开源的语料标注工具,其可以用来标注实体识别训练语料。但是标注之后的数据是不能直接作为训练数据的,还需要将其转录一下,下面以转录为BIO为例。
1 doccano标注之后的数据格式(json)
{"id": 4, "text": "?生益科技主要从事覆铜板制造与销售业务,销售收入占到公司总收入的81.52%。", "meta": {}, "annotation_approver": null, "labels": [[1, 5, "ORG"], [32, 38, "NUM"]]}
{"id": 6, "text": "目前,益科技,已是覆铜板的龙头企业。早在公司2018 年年报,生益科技硬质覆铜板销售总额全球排名第二。", "meta": {}, "annotation_approver": null, "labels": [[31, 35, "ORG"], [22, 27, "NUM"], [3, 6, "ORG"], [20, 22, "ORG"]]}
{"id": 9, "text": "专注高频覆铜板的南通项目一期产能为100w平米/年,现逐步投产。", "meta": {}, "annotation_approver": null, "labels": [[17, 21, "NUM"]]}
{"id": 10, "text": "?获得NOKIA、华为、中兴、浪潮、格力、国星光电等客户的认可。", "meta": {}, "annotation_approver": null, "labels": [[3, 8, "ORG"], [9, 11, "ORG"], [12, 14, "ORG"], [15, 17, "ORG"], [18, 20, "ORG"], [21, 25, "ORG"]]}
{"id": 11, "text": "③业绩符合预期,产品未来看点足", "meta": {}, "annotation_approver": null, "labels": [[1, 3, "PRO"], [5, 7, "TAP"]]}
需要注意的是以上并不是标准的json格式,所以直接从doccano系统里下载下来的json是不能用python直接加载的。
标准的json文件格式如下:
[
{''''},
{'''''},
{'''''}
]
转换代码如下:
def generate_json():
'''将标注系统下载下来的文件转换为标准json格式'''
f1 = open('out.json', 'w', encoding='utf-8')
f1.write("[")
with open('in.json', 'r', encoding='utf-8')as f2:
lines = f2.readlines()
k = len(lines)
i = 0
while i < k-2:
f1.write(lines[i].strip() + ',\n')
i += 1
f1.write(lines[i].strip() + '\n')
f1.write(']')
f1.close()
转换为标准格式后可以直接加载,然后将其中的数据转录为BIO形式。
2.转录为BIO
转录代码如下:
import json
def tranfer2bio():
'''
将json文件中的数据转录为BIO形式,保存规则可以在43行修改
:return:
'''
f1 = open('./train.txt', 'w', encoding='utf-8')
with open("./1.json", 'r', encoding='utf-8') as inf:
load = json.load(inf)
for i in range(len(load)):
labels = load[i]['labels']
text = load[i]['text']
tags = ['O'] * len(text)
for j in range(len(labels)):
label = labels[j]
#print(label)
tags[label[0]] = 'B-' + str(label[2])
k = label[0]+1
while k < label[1]:
tags[k] = 'I-' + str(label[2])
k += 1
print(tags)
for word, tag in zip(text, tags):
f1.write(word + '\t' + tag + '\n')
f1.write("\n")
#tranfer2bio()
3.根据BIO序列提取实体
模型预测出对应文本的BIO序列后,如何根据序列提取实体呢?这个问题有很多解决办法。博主这里给出一个开源通用的解决方法[1]。
思路1:遇到B则前面存在的实体,进行一次存储。多个i粘连一块儿也可能被认为是一个实体。错误的情况是B识别成i了。对于类别判断失误,粘连的实体取众数。
#标签转录BIO格式
string="我是李明,我爱中国,我来自呼和浩特"
predict=["o","o","i-per","i-per","o","o","o","b-loc","i-loc","o","o","o","o","b-per","i-loc","i-loc","i-loc"]
item = {"string": string, "entities": []}
entity_name = ""
flag=[]
visit=False
for char, tag in zip(string, predict):
if tag[0] == "b":
if entity_name!="":
x=dict((a,flag.count(a)) for a in flag)
y=[k for k,v in x.items() if max(x.values())==v]
item["entities"].append({"word": entity_name,"type": y[0]})
flag.clear()
entity_name=""
entity_name += char
flag.append(tag[2:])
elif tag[0]=="i":
entity_name += char
flag.append(tag[2:])
else:
if entity_name!="":
x=dict((a,flag.count(a)) for a in flag)
y=[k for k,v in x.items() if max(x.values())==v]
item["entities"].append({"word": entity_name,"type": y[0]})
flag.clear()
flag.clear()
entity_name=""
if entity_name!="":
x=dict((a,flag.count(a)) for a in flag)
y=[k for k,v in x.items() if max(x.values())==v]
item["entities"].append({"word": entity_name,"type": y[0]})
print(item)
{'string': '我是李明,我爱中国,我来自呼和浩特', 'entities': [{'word': '李明', 'type': 'per'}, {'word': '中国', 'type': 'loc'}, {'word': '呼和浩特', 'type': 'loc'}]}
思路2:只取B开头的实体,其它的不要。同样类别也是取众数。
#标签转录BIO格式
string="我是李明,我爱中国,我来自呼和浩特"
predict=["o","o","i-per","i-per","o","o","o","b-loc","i-loc","o","o","o","o","b-per","i-loc","i-loc","i-loc"]
item = {"string": string, "entities": []}
entity_name = ""
flag=[]
visit=False
for char, tag in zip(string, tags):
if tag[0] == "b":
if entity_name!="":
x=dict((a,flag.count(a)) for a in flag)
y=[k for k,v in x.items() if max(x.values())==v]
item["entities"].append({"word": entity_name,"type": y[0]})
flag.clear()
entity_name=""
visit=True
entity_name += char
flag.append(tag[2:])
elif tag[0]=="i" and visit:
entity_name += char
flag.append(tag[2:])
else:
if entity_name!="":
x=dict((a,flag.count(a)) for a in flag)
y=[k for k,v in x.items() if max(x.values())==v]
item["entities"].append({"word": entity_name,"type": y[0]})
flag.clear()
flag.clear()
visit=False
entity_name=""
if entity_name!="":
x=dict((a,flag.count(a)) for a in flag)
y=[k for k,v in x.items() if max(x.values())==v]
item["entities"].append({"word": entity_name,"type": y[0]})
print(item)
{'string': '我是李明,我爱中国,我来自呼和浩特', 'entities': [{'word': '中国', 'type': 'loc'}, {'word': '呼和浩特', 'type': 'loc'}]}
2020年12月31日 于上海
参考文献
[1]BIO序列提取实体(NER命名实体识别).https://blog.csdn.net/hqh131360239/article/details/107764716
原博客https://blog.csdn.net/broccoli2/article/details/112008692
复制一下,怕以后不见了