word2vec查询词向量时报错：'utf-8' codec cann't decode bytes in position 96-07:unexpected end of data

最新推荐文章于 2024-06-21 10:51:21 发布

yinglish_

最新推荐文章于 2024-06-21 10:51:21 发布

阅读量3k

点赞数 2

分类专栏： python 文章标签： python word2vec 编码报错

本文链接：https://blog.csdn.net/yinglish_/article/details/80383486

版权

在使用word2vec模型时遇到'utf-8' codec无法解码bytes的错误。确认分词文件及bin文件为utf-8编码无误后，通过修改word2vec源码跳过导致错误的特定词语，发现该词语编码异常。为避免此类问题，理想的做法是在分词阶段进行数据清洗，但考虑到大文件的重跑成本，选择了直接跳过错误词语。

摘要由CSDN通过智能技术生成

加载word2vec模型时报错：

    model_path = "model/Hanlp_cut_news.bin"
    w2v_dict = word2vec.load(model_path)
    print(w2v_dict["奥运"])

Traceback (most recent call last):
  File "/home/iiip/PycharmProjects/smp_yinglish/demo1/data_preprocess.py", line 10, in <module>
    w2v_dict = word2vec.load(model_path)
  File "/home/iiip/.local/lib/python3.5/site-packages/word2vec/io.py", line 18, in load
    return word2vec.WordVectors.from_binary(fname, *args, **kwargs)
  File "/home/iiip/.local/lib/python3.5/site-packages/word2vec/wordvectors.py", line 202, in from_binary
    vocab[i] = word.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 96-97: unexpected end of data<