python 读取txt文件编码处理

python 读txt文件的时候，经常遇到编码报错的问题。处理文本读取，首先要确定文件的编码方式，然后通过指定encoding类别的方式读取文件，遇到无法解析的字符，可以通过指定未识别字符的处理方式处理。

1.识别文件编码

通过chardet 返回文件的编码类型，未识别的类型返回None

import chardet
 
# 获取文件编码类型
def get_encoding(file):
    # 二进制方式读取，获取字节数据，检测类型
    with open(file, 'rb') as f:
        return chardet.detect(f.read())['encoding']
 
 
file_name = 'my.ini'
encoding = get_encoding(file_name)

2. 读取文件指定encoding

with open('../corpus.txt', encoding='utf-8',  mode = 'r') as f:

3. 未识别字符处理方式

指定open的errors属性，设置未识别字符处理方式

with open('../corpus.txt', encoding='utf-8',  mode = 'r', errors='replace') as f:

errors 是open函数的可选参数，指定encoding和decoding过程的error处理方式，有几种标准error handler，也可以通过codecs.register_error()注册指定自定义handler

• 'strict'  遇到编码问题抛出异常
• 'ignore' 忽略异常，可能造成数据丢失
• 'replace' 遇到异常用？代替
• 'surrogateescape' will represent any incorrect bytes as code points in the Unicode Private Use Area ranging from U+DC80 to U+DCFF. These private code points will then be turned back into the same bytes when the surrogateescape error handler is used when writing data. This is useful for processing files in an unknown encoding.
• 'xmlcharrefreplace' is only supported when writing to a file. Characters not supported by the encoding are replaced with the appropriate XML character reference &#nnn;.
• 'backslashreplace' (also only supported when writing) replaces unsupported characters with Python’s backslashed escape sequences.

'strict'模式下遇到encoding error就报错终止，导致文件读取失败

‘ignore’和‘replace’模式忽略个别报错，能正常读取文件

参考：

https://blog.csdn.net/u013314786/article/details/77774579 python 获取文件字符编码类型

https://stackoverflow.com/questions/24616678/unicodedecodeerror-in-python-when-reading-a-file-how-to-ignore-the-error-and-ju

https://blog.csdn.net/joyfixing/article/details/79971667?utm_medium=distribute.pc_relevant_t0.none-task-blog-BlogCommendFromMachineLearnPai2-1.add_param_isCf&depth_1-utm_source=distribute.pc_relevant_t0.none-task-blog-BlogCommendFromMachineLearnPai2-1.add_param_isCf python 中文编码

https://www.crifan.com/character_encoding_charset_simpile_tutorial/ 字符编码