jieba 多线程分词

原因

• jieba分词太慢了，动不动就几十分钟
• 句子越短、总的句子数量越多，花费时间越长

解决方案

• 构造线程池，多线程分词，然后把分词结果放到一个字典中
• 参考资料：Python线程池及其原理和使用（超级详细）

代码

import json
import random
import jieba
from tqdm import tqdm
from concurrent.futures import ThreadPoolExecutor
import threading

if __name__ == '__main__':
	# 加载原始数据，是个列表，列表中每个item 是个字典。
    with open("../csc_sample/train.json", 'r') as f:
        data = json.load(f)

	# 遍历列表中的每个字典，把字段是 correct_text 拿出来，每个分配一个编号，存在字典中
    lst = [item['correct_text'] for item in data]
    dic = {}
    
    for sen_idx, sen in enumerate(lst):
        dic[sen_idx] = sen

	# 定义向线程池提交的函数，定义返回的内容
    def func(sent, k):
        words = jieba.lcut(sent)
        return words, k

	# 创建线程池，因为我想让多线程分词，但是分词完的结果也要与之前分配的编号对应，也就是说句子的编号在分词前后不能变，所以用 mapping 来构建分词后的字典
    with ThreadPoolExecutor(max_workers=1000) as pool:
        mapping = {}
        for k in tqdm(dic.keys()):
        	# 向线程池提交处理请求 func, 把 func 的参数放在 submit 函数后面
            feature = pool.submit(func, dic[k], k)

			# 定义回调函数，将线程处理的结果放到 mapping 字典中
            def get_result(feature):
                words, k = feature.result()
                mapping[k] = words
			# 调用回调函数
            feature.add_done_callback(get_result)
		# 构建排序字典
		mapping = OrderedDict(sorted(mapping.items(), key=lambda obj: obj[0]))

		# 把数据存起来
        with open("train_words.json", 'w') as f:
            json.dump(mapping, f, ensure_ascii=False)

• 未经过处理的原始数据

[
    {
        "id":"-",
        "original_text":"目前区次事件的细节还不清楚，伤亡人数也未确定。",
        "wrong_ids":[
            2
        ],
        "correct_text":"目前这次事件的细节还不清楚，伤亡人数也未确定。"
    },
    {
        "id":"-",
        "original_text":"报导中并未说明出口量，但据引述药厂主管的话指出，每一种药物最大的庄度出口量都达到十二吨之谱。",
        "wrong_ids":[
            32
        ],
        "correct_text":"报导中并未说明出口量，但据引述药厂主管的话指出，每一种药物最大的年度出口量都达到十二吨之谱。"
    },
    {
        "id":"-",
        "original_text":"丈夫拒不承认家暴庭审一结束就打期子。",
        "wrong_ids":[
            15
        ],
        "correct_text":"丈夫拒不承认家暴庭审一结束就打妻子。"
    },
    {
        "id":"-",
        "original_text":"报导并末说明事故发生的原因。",
        "wrong_ids":[
            3
        ],
        "correct_text":"报导并未说明事故发生的原因。"
    }
]

• 处理之后的数据：

{"0": ["目前", "这次", "事件", "的", "细节", "还", "不", "清楚", "，", "伤亡人数", "也", "未确定", "。"], "1": ["报导", "中", "并未", "说明", "出口量", "，", "但据", "引述", "药厂", "主管", "的话", "指出", "，", "每", "一种", "药物", "最大", "的", "年度", "出口量", "都", "达到", "十二", "吨", "之", "谱", "。"], "2": ["丈夫", "拒不承认", "家暴", "庭审", "一", "结束", "就", "打", "妻子", "。"], "3": ["报导", "并未", "说明", "事故", "发生", "的", "原因", "。"]
}