python 字节编码,Python检测字符串字节编码

I've got about 1000 filenames read by os.listdir()

some of them are encoded 'utf-8' and some are 'cp1252'.

I want to decode all of them to unicode for further processing in my script. Is there a way to get the source encoding to correctly decode into unicode?

Example:

for item in os.listdir(rootPath):

#Convert to Unicode

if isinstance(item, str):

item = item.decode('cp1252') # or item = item.decode('utf-8')

print item

解决方案

if your files either in cp1252 and utf-8, then there is an easy way.

import logging

def force_decode(string, codecs=['utf8', 'cp1252']):

for i in codecs:

try:

return string.decode(i)

except UnicodeDecodeError:

pass

logging.warn("cannot decode url %s" % ([string]))

for item in os.listdir(rootPath):

#Convert to Unicode

if isinstance(item, str):

item = force_decode(item)

print item

otherwise, there is a charset detect lib.