1. 环境及语料
1.1 环境
- Homebrew
- Python
- jieba分词库
- gensim库
1.2 下载维基百科语料
从这个链接下载http://download.wikipedia.com/zhwiki/latest/zhwiki-latest-pages-articles.xml.bz2
本文只使用中文WiKi进行Word2Vec实验,英文的Wiki比较好处理,不用使用分词库就可以完成处理。
1.3 安装opencc
opencc用来进行繁体和简体转换
命令:brew install opencc
2. 实验
2.1 转换格式
将下载下来的Wiki XML文件转换成TXT格式文件,运行如下脚本。在Terminal中执行:python parse_wiki_corpora.py zhwiki-latest-pages-articles.xml.bz2 corpus.zhwiki.txt
import logging
import os.path
import sys
from gensim.corpora import WikiCorpus
if __name__ == '__main__':
program = os.path.basename(__file__)
logger = logging.getLogger(program)
logging.basicConfig(format='%(asctime)s: %(levelname)s: %(message)s')
logging.root.setLevel(level=logging.INFO)
logger.info("running %s" % ' '.join(sys.argv))
if len(sys.argv) < 3:
print (globals()['__doc__'] % locals())
sys.exit(1)
inp, outp = sys.argv[1:3]
space = " "
i = 0
output = open(outp, 'w')
wiki = WikiCorpus(inp, lemmatize=False, dictionary={})
for text in wiki.get_texts():
output.write(space.join(text) + "\n")
i += 1
if i % 10000 == 0:
logger.info("Saved " + str(i) + " articles")
output.close()
logger.info("Finished Saved " + str(i) + " articles")
2.2 简繁体转换
命令: opencc -i corpus.zhwiki.txt -o corpus.zhwiki.simplified.txt -c t2s.json
转换其他字体,可参考https://github.com/BYVoid/OpenCC#configurations-%E9%85%8D%E7%BD%AE%E6%96%87%E4%BB%B6 修改上面命令的-c部分。
2.3 移除空白行
执行如下脚本。命令:python remove_en_blank.py corpus.zhwiki.simplified.txt corpus.zhwiki.simplified.done.txt
import re
import logging
import os.path
import sys
if __name__ == '__main__':
# program = os.path.basename(sys.argv[0])
program = os.path.basename(__file__)
logger = logging.getLogger(program)
logging.basicConfig(format='%(asctime)s: %(levelname)s: %(message)s')
logging.root.setLevel(level=logging.INFO)
logger.info("running %s" % ' '.join(sys.argv))
# check and process input arguments
if len(sys.argv) < 3:
print(globals()['__doc__'] % locals())
sys.exit(1)
inp, outp = sys.argv[1:3]
i = 0
output = open(outp, 'w')
f = open(inp)
line = f.readline()
while line:
line = f.readline()
rule = re.compile(r'[ a-zA-z]') # delete english char and blank
result = rule.sub('', line)
output.write(result + "\n")
i += 1
logger.info("Saved " + str(i) + " lines")
f.close()
output.close()
logger.info("Finished Saved " + str(i) + " lines")
2.4 分词
使用jieba进行分词。执行如下脚本,命令:python -m jieba -d ’ ’ corpus.zhwiki.simplified.done.txt > corpus.zhwiki.segwithb.txt
注意英文引号和导出符>
2.5 Training
运行如下脚本,命令:python train.py corpus.zhwiki.segwithb.txt zhiwiki.model
# -*- coding: utf-8 -*-
import logging
import os.path
import sys
import multiprocessing
from gensim.corpora import WikiCorpus
from gensim.models import Word2Vec
from gensim.models.word2vec import LineSentence
if __name__ == '__main__':
# program = os.path.basename(sys.argv[0])
program = os.path.basename(__file__)
logger = logging.getLogger(program)
logging.basicConfig(format='%(asctime)s: %(levelname)s: %(message)s')
logging.root.setLevel(level=logging.INFO)
logger.info("running %s" % ' '.join(sys.argv))
# check and process input arguments
if len(sys.argv) < 3:
print(globals()['__doc__'] % locals())
sys.exit(1)
inp, outp = sys.argv[1:3]
model = Word2Vec(LineSentence(inp), size=400, window=5, min_count=5,
workers=multiprocessing.cpu_count())
model.save(outp)
# model.save_word2vec_format(outp2, binary=False)
2.6 Testing
使用如下脚本进行测试,命令:python test.py
from gensim.models import Word2Vec
model = Word2Vec.load('zhwiki.model')
print(model.most_similar(u'中国', topn=20))
print(model.most_similar(u'北京', topn=20)
3. Word2Vec 存在的问题
- 对每个local Context window单独训练,没有利用包含在global co-currence矩阵中的统计信息
- 对多义词无法很好的表示和处理,因为使用了唯一的词向量
- 一种更好的方法是使用GloVe进行词向量表示
4. Word2Vec 与 GloVe
4.1 Word2Vec中的 CBOW 和 SK-gram 如何选择?
样本量不是特别大的时候,选择CBOW;样本量特别大的时候应该选择SK-gram
4.2 Word2Vec 和 GloVe 对比
GloVe随着迭代次数增加,精度提升。Word2Vec 未使用类似迭代次数的Epoch,用Negative Samples 模拟