python open unicodedecodeerror_切换到Python 3导致UnicodeDecodeError

I've just added Python3 interpreter to Sublime, and the following code stopped working:

for directory in directoryList:

fileList = os.listdir(directory)

for filename in fileList:

filename = os.path.join(directory, filename)

currentFile = open(filename, 'rt')

for line in currentFile: ##Here comes the exception.

currentLine = line.split(' ')

for word in currentLine:

if word.lower() not in bigBagOfWords:

bigBagOfWords.append(word.lower())

currentFile.close()

I get a following exception:

File "/Users/Kuba/Desktop/DictionaryCreator.py", line 11, in

for line in currentFile:

File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/encodings/ascii.py", line 26, in decode

return codecs.ascii_decode(input, self.errors)[0]

UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 305: ordinal not in range(128)

I found this rather strange, because as far as I know Python3 is supposed to support utf-8 everywhere. What's more, the same exact code works with no problems on Python2.7. I've read about adding environmental variable PYTHONIOENCODING, but I tried it - to no avail (however, it appears it is not that easy to add an environmental variable in OS X Mavericks, so maybe I did something wrong with adding the variable? I modidified /etc/launchd.conf)

解决方案

Python 3 decodes text files when reading. The default encoding is taken from locale.getpreferredencoding(False), which evidently for your setup returns 'ASCII'. See the open() function documenation:

In text mode, if encoding is not specified the encoding used is platform dependent: locale.getpreferredencoding(False) is called to get the current locale encoding.

Instead of relying on a system setting, you should open your text files using an explicit codec:

currentFile = open(filename, 'rt', encoding='latin1')

where you set the encoding parameter to match the file you are reading.

Python 3 supports UTF-8 as the default for source code.

You may want to read up on Python 3 and Unicode in the Unicode HOWTO, which explains both about source code encoding and reading and writing Unicode data.