requirements.txt依赖文件生成，UnicodeDecodeError: ‘utf-8‘ codec can‘t decode byte 0xce in position 845: inv

引言

基于Python的项目开发完成后，需要进行打包和部署。而Python项目开发过程中安装了很多依赖包，打包之前必须生成依赖包清单。别人通过依赖包清单安装包后，就能正常的运行项目了。生成Python依赖包清单的方式一般有两种，本文是对这两种方式的用法简介

【Python】一键生成项目依赖包清单（pipreqs）-CSDN博客

1、pip生成环境依赖文件requirements.txt

pip freeze > requirements.txt

2、[推荐]使用pipreqs 写入requirements.txt
使用pipreqs 写入requirements.txt，只是程序中所用到的包，并不是电脑中安装的所有的包。

pip install pipreqs

cd 到程序所在的文件夹位置
执行下面的语句，Windows下得加encoding=utf8，不然会出现编码问题。

pipreqs ./ --encoding=utf8 

报错

报错

------------------》pipreqs ./ --encoding=utf8
INFO: Not scanning for jupyter notebooks.
Traceback (most recent call last):
  File "D:\JY\anaconda3\envs\Langchain-Chatchat\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\JY\anaconda3\envs\Langchain-Chatchat\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "D:\JY\anaconda3\envs\Langchain-Chatchat\Scripts\pipreqs.exe\__main__.py", line 7, in <module>
  File "D:\JY\anaconda3\envs\Langchain-Chatchat\lib\site-packages\pipreqs\pipreqs.py", line 609, in main
    init(args)
  File "D:\JY\anaconda3\envs\Langchain-Chatchat\lib\site-packages\pipreqs\pipreqs.py", line 533, in init
    candidates = get_all_imports(
  File "D:\JY\anaconda3\envs\Langchain-Chatchat\lib\site-packages\pipreqs\pipreqs.py", line 136, in get_all_imports
    contents = read_file_content(file_name, encoding)
  File "D:\JY\anaconda3\envs\Langchain-Chatchat\lib\site-packages\pipreqs\pipreqs.py", line 181, in read_file_content
    contents = f.read()
  File "D:\JY\anaconda3\envs\Langchain-Chatchat\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xce in position 845: invalid continuation byte

解决报错

1.项目文件编码错误（直接使用enconding=utf8），使用iso-8859-1编码可以生成，或许可能少东西

2.将项目文件所有文件转换成utf8

import os
import chardet

def find_and_convert_files(directory):
    for root, _, files in os.walk(directory):
        for file in files:
            if file.endswith(".py"):
                file_path = os.path.join(root, file)
                with open(file_path, 'rb') as f:
                    raw_data = f.read()
                    encoding = chardet.detect(raw_data)['encoding']
                    if encoding != 'utf-8':
                        try:
                            with open(file_path, 'w', encoding='utf-8', errors='ignore') as f_utf8:
                                f_utf8.write(raw_data.decode(encoding, errors='ignore'))
                            print(f"Converted {file_path} to UTF-8")
                        except Exception as e:
                            print(f"Failed to convert {file_path}: {e}")

find_and_convert_files('D:/WH/Project/Embedding_fine')

重新执行命令