Word2vec训练中文词向量
发布时间:2018-08-16 10:55,
浏览次数:445
, 标签:
Word
vec
Word2vec训练中文词向量
将所有的语料读入到列表中,把每一句已经分好词的句子作为列表中的一个元素,类似:
[[‘i’,’am’,’a’,’girl’],[‘he’,’is’,’a’,’boy’]]
from gensim.models import Word2Vec
import os
import sys
##下面的代码是训练字向量的代码
sentence = []
word = []
tag = 0
with open('data_all.txt',encoding='utf-8') as fp: ##
print(type(fp))
for line in fp.readlines():
# print(line)
line = line.strip()
if line == '':
continue
tag += 1
print(tag)
if tag >10000:
break
if tag == 1:
line1 = ['']+line.split()
sentence.append(line1)
word.append('')
else: