windows下训练中文维基百科的word2vec

包括四个步骤：1）下载中文维基百科语料；2）利用opencc进行繁简转换；3）对语料分词；4）利用gensim训练词向量

1）中文维基百科

下载地址是：https://dumps.wikimedia.org/zhwiki/latest/zhwiki-latest-pages-articles.xml.bz2，下载下来的是xml文件,需要转换成text格式。通过下面这个脚本(process_wiki.py)实现：

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import logging
import os.path
import sys

from gensim.corpora import WikiCorpus

if __name__ == '__main__':
    program = os.path.basename(sys.argv[0])
    logger = logging.getLogger(program)

    logging.basicConfig(format='%(asctime)s: %(levelname)s: %(message)s')
    logging.root.setLevel(level=logging.INFO)
    logger.info("running %s" % ' '.join(sys.argv))

    # check and process input arguments
    if len(sys.argv) < 3:
        print globals()['__doc__'] % locals()
        sys.exit(1)
    inp, outp = sys.argv[1:3]
    space = " "
    i = 0

    output = open(outp, 'w')
    wiki = WikiCorpus(inp, lemmatize=False, dictionary={})
    for text in wiki.get_texts():
        output.write(space.join(text) + "\n")
        i = i + 1
        if (i % 10000 == 0):
            logger.info("Saved " + str(i) + " articles")

    output.close()
    logger.info("Finished Saved " + str(i) + " articles")

cd到process_wiki.py目录，执行：

python process_wiki.py zhwiki-latest-pages-articles.xml.bz2 wiki.zh.text

 

2）繁简转换

下载opencc并解压，将wiki.zh.text拷贝到opencc-0.4.2目录下，打开cmd进入该目录，执行：

opencc -i wiki.zh.text -o wiki.zh.text.jian -c zht2zhs.ini

 

3）jieba分词

该步骤包含分词，剔除标点符号和去文章结构标识。（建议word2vec训练数据不要去除标点符号，比如在情感分析应用中标点符号很有用）最终将得到分词好的纯文本文件，每行对应一篇文章，词语间以空格作为分隔符。script_seg.py如下：

#!/usr/bin/python
# -*- coding: utf-8 -*-
import sys, codecs
import jieba.posseg as pseg
reload(sys)
sys.setdefaultencoding('utf-8')
if __name__ == '__main__':
    if len(sys.argv) < 3:
        print "Usage: python script.py infile outfile"
        sys.exit()
    i = 0
    infile, outfile = sys.argv[1:3]
    output = codecs.open(outfile, 'w', 'utf-8')
    with codecs.open(infile, 'r', 'utf-8') as myfile:
        for line in myfile:
            line = line.strip()
            if len(line) < 1:
                continue
            if line.startswith('<doc'):
                i = i + 1
                if(i % 1000 == 0):
                    print('Finished ' + str(i) + ' articles')
                continue
            if line.startswith('</doc'):
                output.write('\n')
                continue
            words = pseg.cut(line)
            for word, flag in words:
                if flag.startswith('x'):
                    continue
                output.write(word + ' ')
    output.close()
    print('Finished ' + str(i) + ' articles')

将wiki.zh.text.jian和script_seg.py拷贝到同一目录，在cmd中执行：

python script_seg.py  wiki.zh.text.jian zh.wiki.text

 

4)训练词向量

这里选择的是Python版的word2vec。script_train.py如下：

#!/usr/bin/python
# -*- coding: utf-8 -*-
import os, sys, codecs
import gensim, logging, multiprocessing
reload(sys)
sys.setdefaultencoding('utf-8')
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
if __name__ == '__main__':
    if len(sys.argv) < 3:
        print "Usage: python script.py infile outfile"
        sys.exit()
    infile, outfile = sys.argv[1:3]
    model = gensim.models.Word2Vec(gensim.models.word2vec.LineSentence(infile), size=200, window=5, min_count=5, sg=0, workers=multiprocessing.cpu_count())
    #model.save(outfile)
    model.wv.save_word2vec_format(outfile + '.bin', binary=True)

执行：

python script_train.py zh.wiki.text zh.wiki.model

加载bin文件，查看词向量训练效果：

#encoding=utf-8
from gensim.models import word2vec
model= word2vec.KeyedVectors.load_word2vec_format("./zh.wiki.model.bin", binary=True)
result = model.most_similar(u"足球")
for i in result:
	print i[0],i[1]

文章转自：

http://www.52nlp.cn/%E4%B8%AD%E8%8B%B1%E6%96%87%E7%BB%B4%E5%9F%BA%E7%99%BE%E7%A7%91%E8%AF%AD%E6%96%99%E4%B8%8A%E7%9A%84word2vec%E5%AE%9E%E9%AA%8C/comment-page-1

https://flystarhe.github.io/2016/09/04/word2vec-test/