python 语种识别对比

问题描述

目前找到的python库主要就是langid和langdetect，对两个库的识别准确率做了一下对比，测试集来自于机器翻译的测试集newstest2018
首先将sgm文件转成txt格式，去掉sgm格式的标记符，只保留文本内容
针对每个文件分别计算langid和langdetect的识别准确率，再将识别正确的数量加和除以总数量计算整体准确率

langid_total = []
langdectect_total = []
data_total = []
path_set = os.listdir(dir_out)
for path in path_set:
    lang = path[-6:-4]
    langid_acc = 0
    langdectect_acc = 0
    with open( os.path.join(dir_out, path), encoding='utf-8') as f:
        data = f.readlines()
    for i in data:
        try:
            a = identifier.classify(i)
            if a[0] == lang:
                langid_acc +=1
            b = detect(i)
            if if (b == lang)