word2vec中文语料小试牛刀
最近学习了词向量工具word2vec的原理之后,决定实战一下。
1. 环境配置
- 环境:win10 + python3.6
- 所需工具包:gensim(word2vec)、jieba(中文分词)
- 以上工具包均用pip install安装
- 文本编辑器:EmEditor
2.语料库说明
从网上下载的TXT文档《诛仙》(一部玄幻小说),原始编码为GB2312。
3.实验步骤
3.1.文件编码转换
因为文件“诛仙.txt”原始编码为GBK,整个项目编码为UTF-8,需统一格式。否则直接读取文本将得到乱码。
于是对文件进行编码转换,将编码转为UTF-8,存至一个新的文件。
# -*- coding: utf-8 -*-
file_out = open('诛仙_utf8.txt', 'w', encoding="utf-8")
with open('诛仙.txt', 'r', encoding="GB18030") as file_object:
for line in file_object:
line = line.strip()
file_out.write(line+"\n")
file_out.close()
print("end")此处输入代码
所做工作就是解码原始文件,逐句读取,去除空白,再用utf-8编码,写入新的文件“诛仙_utf8.txt”。处理后如下,一下子舒服了很多。
3.2.中文分词
第二步进行中文分词,去掉一些标点符号。要去掉的停用词(标点符号)放在一个独立的文件stop_words.txt中。
分词代码如下,调用jieba分词库,考虑停用词,将分词结果写入到新的文件“诛仙_cut_word.txt”。
import jieba
stop_words_file = "stop_words.txt"
stop_words = list()
with open(stop_words_file, 'r', encoding="GBK") as stop_words_file_object:
contents = stop_words_file_object.readlines()
for line in contents:
line = line.strip()
stop_words.append(line)
origin_file = "诛仙_utf8.txt"
target_file = open("诛仙_cut_word.txt", 'w', encoding="utf-8")
with open(origin_file, 'r', encoding="utf-8") as origin_file_object:
contents = origin_file_object.readlines()
for line in contents:
line = line.strip()
out_str = ''
word_list = jieba.cut(line, cut_all=False)
for word in word_list:
if word not in stop_words:
if word != "\t":
out_str += word
out_str += ' '
target_file.write(out_str.rstrip()+"\n")
target_file.close()
print("end")
分词后的结果如下,因为没有自制词典,所以效果不是非常理想,不过也算看得过去了。
3.3.模型训练
调用gensim库进行word2vec模型的训练。
import logging
import os
import sys
import multiprocessing
from gensim.models import Word2Vec
from gensim.models.word2vec import LineSentence
if __name__ == '__main__':
program = os.path.basename(sys.argv[0])
logger = logging.getLogger(program)
logging.basicConfig(format='%(asctime)s: %(levelname)s: %(message)s')
logging.root.setLevel(level=logging.INFO)
logger.info("running %s" % ' '.join(sys.argv))
# check and process input arguments
if len(sys.argv) < 4:
print(globals()['__doc__'] % locals())
sys.exit(1)
inp, outp1, outp2 = sys.argv[1:4]
model = Word2Vec(LineSentence(inp), size=400, window=5, min_count=5,
workers=multiprocessing.cpu_count())
# trim unneeded model memory = use(much) less RAM
# model.init_sims(replace=True)
model.save(outp1)
model.wv.save_word2vec_format(outp2, binary=False)
print("end")
3.4.模型测试
输入一个词,得到与它相似度最高的几个词。即word2vec的CBOW模型的效果。
代码如下:
import warnings
import gensim
if __name__ == '__main__':
warnings.filterwarnings(action='ignore', category=UserWarning,module='gensim')
model = gensim.models.Word2Vec.load("zhuxian.model")
word = '张小凡'
result = model.similar_by_word(word)
print("跟 "+word+" 最相近的词:")
for i in result:
print(i)
部分测试结果: