【tensorflow2.0】34.word2vec作业代码

最新推荐文章于 2021-06-22 10:37:53 发布

一只很菜很菜的tfer

最新推荐文章于 2021-06-22 10:37:53 发布

阅读量346

点赞数

分类专栏： tensorflow2.0学习笔记文章标签： tensorflow

本文链接：https://blog.csdn.net/weixin_39122088/article/details/107092757

版权

tensorflow2.0学习笔记专栏收录该内容

50 篇文章

订阅专栏

本文介绍了一种利用Word2Vec模型进行文本数据预处理的方法，包括数据集的读取、jieba分词、模型训练及保存，展示了如何获取单词向量和计算单词间的相似性。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

#没有包的可以先下载
pip install gensim
pip install jieba

import pandas as pd
import numpy as np
from gensim.models import Word2Vec

#读取数据集:这里我们有训练集 验证集 测试集
train = pd.read_csv('./cnews/train.tsv',sep='\t',header=None,names=['label','content'])
val = pd.read_csv('./cnews/dev.tsv',sep='\t',header=None,names=['label','content'])
test = pd.read_csv('./cnews/test.tsv',sep='\t',header=None,names=['label','content'])

#.head（）函数可以让我们查看从头开始的前n行
train.head(10)

在这里插入图片描述

import jieba
#jieba.lcut 直接生成一个list  jion函数将这个list连成一个字符串
def content_cut(x):
    x = jieba.lcut(x)
    x = " ".join(x)
    return x

#调用我们刚定义的函数
train['content'] = train['content'].map(lambda x: content_cut(x))
val['content'] = val['content'].map(lambda x: content_cut(x))
test['content'] = test['content'].map(lambda x: content_cut(x))
'''
out:
Building prefix dict from the default dictionary ...
Dumping model to file cache C:\Users\贾昊\AppData\Local\Temp\jieba.cache
Loading model cost 2.109 seconds.
Prefix dict has been built successfully.
'''
#将训练集验证集测试机合并
df = pd.concat([train,val,test],axis=0)

#训练Word2vec
sentences = [document.split(' ') for document in df['content'].values]
model = Word2Vec(sentences=sentences,
     size=200,#维度
     alpha=0.025, #默认
     window=5, #默认
     min_count=2,#2，3
     sample=0.001,#
     seed=2018, #
     workers=11, #线程
     min_alpha=0.0001, 
     sg=0, #cbow
     hs=0, #负采样
     negative=5,#负采样个数
     ns_exponent=0.75, 
     cbow_mean=1,#求和再取平均
     iter=10, #10到20
     compute_loss =True
     )

#保存和读取模型
model.save("./word2vec/word2vec_word_200")
model = Word2Vec.load("./word2vec/word2vec_word_200")

#查看单词的向量
model.wv['816903'].shape  #(200,)

#和这个单词最相似的20个单词
model.most_similar("",topn=20)

'''

[('12875', 0.8677932620048523),
 ('679169', 0.8625671863555908),
 ('90540', 0.841310977935791),
 ('425105', 0.8043540716171265),
 ('866203', 0.7445841431617737),
 ('122513', 0.7241939902305603),
 ('1234861', 0.7100560069084167),
 ('85838', 0.7024739980697632),
 ('1189755', 0.6224364638328552),
 ('426716', 0.5778474807739258),
 ('816903', 0.5615671873092651),
 ('797828', 0.557973325252533),
 ('1254728', 0.5530299544334412),
 ('11177', 0.546566367149353),
 ('850976', 0.5452205538749695),
 ('48896', 0.5422906875610352),
 ('903604', 0.5324429273605347),
 ('1146147', 0.5293028354644775),
 ('1200328', 0.527854859828949),
 ('1104318', 0.5183314085006714)]
'''
#计算两个单词之间相似性
model.wv.similarity("816903","1226448") #0.6617146