中文维基百科word2vec训练及其代码

最新推荐文章于 2024-09-08 21:00:00 发布

poson

最新推荐文章于 2024-09-08 21:00:00 发布

阅读量862

点赞数

分类专栏：数据挖掘

本文链接：https://blog.csdn.net/poson/article/details/78980167

版权

数据挖掘专栏收录该内容

17 篇文章 0 订阅

订阅专栏

参考文章：中英文维基百科语料上的Word2Vec实验

数据来自：https://dumps.wikimedia.org/zhwiki/latest/zhwiki-latest-pages-articles.xml.bz2

繁体转为简体：

opencc -i zhwiki.txt -o zhwiki.txt.simle -c zht2zhs.ini

先把文件拆分为多个文件：

split  -l  30000  ../zhwiki.txt.simple seg

$ 多进程分词

cat multi_cut.py

import jieba
from multiprocessing import Pool,cpu_count



def cut(name):
    print(name)
    out = open('out/'+name,'w')
    with open('seg/'+name,'r') as f:
        while True:
            line = f.readline()
            if not line:
                break
            line = line.strip()
            wordss = []
            for ss in line.split(" "):
                sent = jieba.lcut(ss,cut_all=False)
                words = [i for i in sent]
                wordss.append(" ".join(words))
            s = ' '.join(wordss)
            out.write(s+"\n")
    out.close()
    # f = open(path,'r')
    # 读取数据
    # data = f.readlines()
    # f.close()
    # return
    #

if __name__ == '__main__':
    path = "zhwiki.txt.simple"
    files = "segaa  segab  segac  segad  segae  segaf  segag  segah  segai  segaj  segak".replace("  "," ").split(" ")
    print(files)
    pool = Pool(cpu_count()-1)
    data = pool.map(cut, files)

训练的代码来自互联网：

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Author: Pan Yang (panyangnlp@gmail.com)
# Copyrigh 2017

from __future__ import print_function

import logging
import os.path
import six
import sys

from gensim.corpora import WikiCorpus

if __name__ == '__main__':
    program = os.path.basename(sys.argv[0])
    logger = logging.getLogger(program)

    logging.basicConfig(format='%(asctime)s: %(levelname)s: %(message)s')
    logging.root.setLevel(level=logging.INFO)
    logger.info("running %s" % ' '.join(sys.argv))

    # check and process input arguments
    if len(sys.argv) != 3:
        print("Using: python process_wiki.py enwiki.xxx.xml.bz2 wiki.en.text")
        sys.exit(1)
    inp, outp = sys.argv[1:3]
    space = " "
    i = 0

    output = open(outp, 'w')
    wiki = WikiCorpus(inp, lemmatize=False, dictionary={})
    for text in wiki.get_texts():
        if six.PY3:
            output.write(' '.join(text) + '\n')
        #   ###another method###
        #    output.write(
        #            space.join(map(lambda x:x.decode("utf-8"), text)) + '\n')
        else:
            output.write(space.join(text) + "\n")
        i = i + 1
        if (i % 10000 == 0):
            logger.info("Saved " + str(i) + " articles")

    output.close()
    logger.info("Finished Saved " + str(i) + " articles")