问题描述
目前找到的python库主要就是langid和langdetect,对两个库的识别准确率做了一下对比,测试集来自于机器翻译的测试集newstest2018
首先将sgm文件转成txt格式,去掉sgm格式的标记符,只保留文本内容
针对每个文件分别计算langid和langdetect的识别准确率,再将识别正确的数量加和除以总数量计算整体准确率
langid_total = []
langdectect_total = []
data_total = []
path_set = os.listdir(dir_out)
for path in path_set:
lang = path[-6:-4]
langid_acc = 0
langdectect_acc = 0
with open( os.path.join(dir_out, path), encoding='utf-8') as f:
data = f.readlines()
for i in data:
try:
a = identifier.classify(i)
if a[0] == lang:
langid_acc +=1
b = detect(i)
if if (b == lang)