包括四个步骤:1)下载中文维基百科语料;2)利用opencc进行繁简转换;3)对语料分词;4)利用gensim训练词向量
1)中文维基百科
下载地址是:https://dumps.wikimedia.org/zhwiki/latest/zhwiki-latest-pages-articles.xml.bz2,下载下来的是xml文件,需要转换成text格式。通过下面这个脚本(process_wiki.py)实现:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import logging
import os.path
import sys
from gensim.corpora import WikiCorpus
if __name__ == '__main__':
program = os.path.basename(sys.argv[0])
logger = logging.getLogger(program)
logging.basicConfig(format='%(asctime)s: %(levelname)s: %(message)s')
logging.root.setLevel(level=logging.INFO)
logger.info("running %s" % ' '.join(sys.argv))
# check and process input arguments
if len(sys.argv) < 3:
print globals()['__doc__'] % locals()
sys.exit(1)
inp, outp = sys.argv[1:3]
space = " "
i = 0
output = open(outp, 'w')
wiki = WikiCorpus(inp, lemmatize=False, dictionary={})
for text in wiki.get_texts():
output.write(space.join(text) + "\n")
i = i + 1
if (i % 10000 == 0):
logger.info("Saved " + str(i) + " articles")
output.close()
logger.info("Finished Saved " + str(i) + " articles")
cd到process_wiki.py目录,执行:
python process_wiki.py zhwiki-latest-pages-articles.xml.bz2 wiki.zh.text
2)繁简转换
下载opencc并解压,将wiki.zh.text拷贝到opencc-0.4.2目录下,打开cmd进入该目录,执行:
opencc -i wiki.zh.text -o wiki.zh.text.jian -c zht2zhs.ini
3)jieba分词
该步骤包含分词,剔除标点符号和去文章结构标识。(建议word2vec训练数据不要去除标点符号,比如在情感分析应用中标点符号很有用)最终将得到分词好的纯文本文件,每行对应一篇文章,词语间以空格作为分隔符。script_seg.py如下:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import sys, codecs
import jieba.posseg as pseg
reload(sys)
sys.setdefaultencoding('utf-8')
if __name__ == '__main__':
if len(sys.argv) < 3:
print "Usage: python script.py infile outfile"
sys.exit()
i = 0
infile, outfile = sys.argv[1:3]
output = codecs.open(outfile, 'w', 'utf-8')
with codecs.open(infile, 'r', 'utf-8') as myfile:
for line in myfile:
line = line.strip()
if len(line) < 1:
continue
if line.startswith('<doc'):
i = i + 1
if(i % 1000 == 0):
print('Finished ' + str(i) + ' articles')
continue
if line.startswith('</doc'):
output.write('\n')
continue
words = pseg.cut(line)
for word, flag in words:
if flag.startswith('x'):
continue
output.write(word + ' ')
output.close()
print('Finished ' + str(i) + ' articles')
将wiki.zh.text.jian和script_seg.py拷贝到同一目录,在cmd中执行:
python script_seg.py wiki.zh.text.jian zh.wiki.text
4)训练词向量
这里选择的是Python版的word2vec。script_train.py如下:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import os, sys, codecs
import gensim, logging, multiprocessing
reload(sys)
sys.setdefaultencoding('utf-8')
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
if __name__ == '__main__':
if len(sys.argv) < 3:
print "Usage: python script.py infile outfile"
sys.exit()
infile, outfile = sys.argv[1:3]
model = gensim.models.Word2Vec(gensim.models.word2vec.LineSentence(infile), size=200, window=5, min_count=5, sg=0, workers=multiprocessing.cpu_count())
#model.save(outfile)
model.wv.save_word2vec_format(outfile + '.bin', binary=True)
执行:
python script_train.py zh.wiki.text zh.wiki.model
加载bin文件,查看词向量训练效果:
#encoding=utf-8
from gensim.models import word2vec
model= word2vec.KeyedVectors.load_word2vec_format("./zh.wiki.model.bin", binary=True)
result = model.most_similar(u"足球")
for i in result:
print i[0],i[1]
文章转自: