用chardet module自动识别文件编码

最新推荐文章于 2023-02-13 16:35:49 发布

win2cs

最新推荐文章于 2023-02-13 16:35:49 发布

阅读量452

点赞数

文章标签： module encoding import dictionary character function

本文链接：https://blog.csdn.net/win2cs/article/details/2131244

版权

http://chardet.feedparser.org/
返回encoding和confidence
试了下很有效

Example: Using the `detect` function

The detect function takes one argument, a non-Unicode string. It returns a dictionary containing the auto-detected character encoding and a confidence level from 0 to 1.

>>> import urllib
>>> rawdata = urllib.urlopen('http://yahoo.co.jp/').read()
>>> import chardet
>>> chardet.detect(rawdata)
{'encoding': 'EUC-JP', 'confidence': 0.99}

Example: Detecting encoding incrementally
import urllib
from chardet.universaldetector import UniversalDetector

usock = urllib.urlopen('http://yahoo.co.jp/')
detector = UniversalDetector()
for line in usock.readlines():
    detector.feed(line)
    if detector.done: break
detector.close()
usock.close()
print detector.result
{'encoding': 'EUC-JP', 'confidence': 0.99}

Example: Detecting encodings of multiple files
import glob
from charset.universaldetector import UniversalDetector

detector = UniversalDetector()
for filename in glob.glob('*.xml'):
    print filename.ljust(60),
    detector.reset()
    for line in file(filename, 'rb'):
        detector.feed(line)
        if detector.done: break
    detector.close()
    print detector.result