word2vec加载异常解决：UnicodeDecodeError: ‘utf-8‘ codec can‘t decode bytes in position。。。

最新推荐文章于 2024-09-04 06:00:00 发布

SeaSky_Steven

最新推荐文章于 2024-09-04 06:00:00 发布

阅读量1.9k

点赞数 1

分类专栏： algorithm 文章标签： word2vec gensim 编码异常 bytes error

原文链接：https://blog.csdn.net/W_Honor/article/details/105037033?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-1.channel_param&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-1.channel_par

版权

algorithm 专栏收录该内容

28 篇文章 8 订阅

订阅专栏

本文详细介绍了在使用Gensim库的word2vec模型时遇到的UnicodeDecodeError问题及其解决方案，包括调整存储模型的方式和在加载模型时设置unicode_errors参数为'ignore'。

摘要由CSDN通过智能技术生成

NLP领域最出名的python库之一就是gensim，该库包含了常见的word2vec模型，而我们在使用这些模型进行embedding的时候可会出现如下的编码问题：

UnicodeDecodeError: 'utf-8' codec can't decode bytes in position

个错误在提示加载模型时出现了编码错误，即在某一个位置的字节是无法编码的，并会给出相应的的位置。
这个问题在gensim官方github中也有反馈，并且作者统一给出了解决方法：

Answer: The strings (words) stored in your model are not valid utf8. By default, gensim decodes the words using the strict encoding settings, which results in the above exception whenever an invalid utf8 sequence is encountered.
The fix is on your side and it is to either:
a) Store your model using a program that understands unicode and utf8 (such as gensim). Some C and Java word2vec tools are known to truncate the strings at byte boundaries, which can result in cutting a multi-byte utf8 character in half, making it non-valid utf8, leading to this error.
b) Set the unicode_errors flag when running load_word2vec_model, e.g. load_word2vec_model(…, unicode_errors=‘ignore’). Note that this silences the error, but the utf8 problem is still there – invalid utf8 characters will just be ignored in this case.

第一种方法告诉了使用者可以用C和JAVA的相关工具去处文本中的无效编码字符；
第二种方法建议在load_word2vec_model函数中设置参数unicode_errors=‘ignore’，可以暂时忽略这个错误。

def get_word_vectors(model_path, dim, words):
    wordVec = gensim.models.KeyedVectors.load_word2vec_format(model_path, binary=True, unicode_errors='ignore')
    wordEmbedding = []
    for word in words:
        try:
            if "<PAD>" == word:
                wordEmbedding.append(np.zeros(dim))
            else:
                vector = wordVec.wv[word]
                wordEmbedding.append(vector)
        except:
            print(word + "不存在于词向量中")
            wordEmbedding.append(np.random.randn(dim))

    return np.array(wordEmbedding)