I have extracted a string from web crawl script as following:
u'\xe3\x80\x90\xe4\xb8\xad\xe5\xad\x97\xe3\x80\x91'
I want to decode u'\xe3\x80\x90\xe4\xb8\xad\xe5\xad\x97\xe3\x80\x91' with utf-8.
With http://ddecode.com/hexdecoder/, I can see the result is '【中字】'
I tried using the following syntax but failed.
msg = u'\xe3\x80\x90\xe4\xb8\xad\xe5\xad\x97\xe3\x80\x91'
result = msg.decode('utf8')
Error:
Traceback (most recent call last):
File "", line 1, in
File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-11: ordi
nal not in range(128)
May I ask how to decode the string correctly?
Thanks for help.
解决方案
The problem with
msg = u'\xe3\x80\x90\xe4\xb8\xad\xe5\xad\x97\xe3\x80\x91'
result = msg.decode('utf8')
is that you are trying to decode Unicode. That doesn't really make sense. You can encode from Unicode to some type of encoding, or you can decode a byte string to Unicode.
When you do
msg.decode('utf8')
Python 2 sees that msg is Unicode. It knows that it can't decode Unicode so it "helpfully" assumes that you want to encode msg with the default ASCII codec so the result of that transformation can be decoded to Unicode using the UTF-8 codec. Python 3 behaves much more sensibly: that code would simply fail with
AttributeError: 'str' object has no attribute 'decode'
The technique given in kennytm's answer:
msg.encode('latin1').decode('utf-8')
works because the Unicode codepoints less than 256 correspond directly to the characters in the Latin1 encoding (aka ISO 8859-1).
Here's some Python 2 code that illustrates this:
for i in xrange(256):
lat = chr(i)
uni = unichr(i)
assert lat == uni.encode('latin1')
assert lat.decode('latin1') == uni
And here is the equivalent Python 3 code:
for i in range(256):
lat = bytes([i])
uni = chr(i)
assert lat == uni.encode('latin1')
assert lat.decode('latin1') == uni
You may find this article helpful: Pragmatic Unicode, which was written by SO veteran Ned Batchelder.
Unless you are forced to use Python 2 I strongly advise you to switch to Python 3. It will make handling Unicode far less painful.