python中unicode编码怎么用,Python如何使用十六进制字符解码unicode

I have extracted a string from web crawl script as following:

u'\xe3\x80\x90\xe4\xb8\xad\xe5\xad\x97\xe3\x80\x91'

I want to decode u'\xe3\x80\x90\xe4\xb8\xad\xe5\xad\x97\xe3\x80\x91' with utf-8.

With http://ddecode.com/hexdecoder/, I can see the result is '【中字】'

I tried using the following syntax but failed.

msg = u'\xe3\x80\x90\xe4\xb8\xad\xe5\xad\x97\xe3\x80\x91'

result = msg.decode('utf8')

Error:

Traceback (most recent call last):

File "", line 1, in

File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode

return codecs.utf_8_decode(input, errors, True)

UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-11: ordi

nal not in range(128)

May I ask how to decode the string correctly?

Thanks for help.

解决方案

The problem with

msg = u'\xe3\x80\x90\xe4\xb8\xad\xe5\xad\x97\xe3\x80\x91'

result = msg.decode('utf8')

is that you are trying to decode Unicode. That doesn't really make sense. You can encode from Unicode to some type of encoding, or you can decode a byte string to Unicode.

When you do

msg.decode('utf8')

Python 2 sees that msg is Unicode. It knows that it can't decode Unicode so it "helpfully" assumes that you want to encode msg with the default ASCII codec so the result of that transformation can be decoded to Unicode using the UTF-8 codec. Python 3 behaves much more sensibly: that code would simply fail with

AttributeError: 'str' object has no attribute 'decode'

The technique given in kennytm's answer:

msg.encode('latin1').decode('utf-8')

works because the Unicode codepoints less than 256 correspond directly to the characters in the Latin1 encoding (aka ISO 8859-1).

Here's some Python 2 code that illustrates this:

for i in xrange(256):

lat = chr(i)

uni = unichr(i)

assert lat == uni.encode('latin1')

assert lat.decode('latin1') == uni

And here is the equivalent Python 3 code:

for i in range(256):

lat = bytes([i])

uni = chr(i)

assert lat == uni.encode('latin1')

assert lat.decode('latin1') == uni

You may find this article helpful: Pragmatic Unicode, which was written by SO veteran Ned Batchelder.

Unless you are forced to use Python 2 I strongly advise you to switch to Python 3. It will make handling Unicode far less painful.