linux将latin1转为utf8,Python：从ISO-8859-1 / latin1转换为UTF-8

犯罪嫌疑人X

这是一个普遍的问题，因此这里是一个相对详尽的说明。对于非unicode字符串(u例如u'\xc4pple'，没有前缀的字符串)，必须将本机编码(iso8859-1/ latin1，除非使用enigmaticsys.setdefaultencoding函数进行修改)解码为unicode，然后编码为可以显示所需字符的字符集，在这种情况下，会推荐UTF-8。首先，这是一个方便的实用程序函数，它将帮助阐明Python 2.7字符串和unicode的模式：>>> def tell_me_about(s): return (type(s), s)一个普通的字符串>>> v = "\xC4pple" # iso-8859-1 aka latin1 encoded string>>> tell_me_about(v)(, '\xc4pple')>>> v'\xc4pple'        # representation in memory>>> print v?pple             # map the iso-8859-1 in-memory to iso-8859-1 chars                  # note that '\xc4' has no representation in iso-8859-1,                   # so is printed as "?".解码iso8859-1字符串-将纯字符串转换为unicode>>> uv = v.decode("iso-8859-1")>>> uvu'\xc4pple'       # decoding iso-8859-1 becomes unicode, in memory>>> tell_me_about(uv)(, u'\xc4pple')>>> print v.decode("iso-8859-1")Äpple             # convert unicode to the default character set                  # (utf-8, based on sys.stdout.encoding)>>> v.decode('iso-8859-1') == u'\xc4pple'True              # one could have just used a unicode representation                   # from the start多一点插图-带“Ä”>>> u"Ä" == u"\xc4"True              # the native unicode char and escaped versions are the same>>> "Ä" == u"\xc4"  False             # the native unicode char is '\xc3\x84' in latin1>>> "Ä".decode('utf8') == u"\xc4"True              # one can decode the string to get unicode>>> "Ä" == "\xc4"False             # the native character and the escaped string are                  # of course not equal ('\xc3\x84' != '\xc4').编码为UTF>>> u8 = v.decode("iso-8859-1").encode("utf-8")>>> u8'\xc3\x84pple'    # convert iso-8859-1 to unicode to utf-8>>> tell_me_about(u8)(, '\xc3\x84pple')>>> u16 = v.decode('iso-8859-1').encode('utf-16')>>> tell_me_about(u16)(, '\xff\xfe\xc4\x00p\x00p\x00l\x00e\x00')>>> tell_me_about(u8.decode('utf8'))(, u'\xc4pple')>>> tell_me_about(u16.decode('utf16'))(, u'\xc4pple')unicode与UTF和latin1之间的关系>>> print u8Äpple             # printing utf-8 - because of the encoding we now know                  # how to print the characters>>> print u8.decode('utf-8') # printing unicodeÄpple>>> print u16     # printing 'bytes' of u16���pple>>> print u16.decode('utf16')Äpple             # printing unicode>>> v == u8False             # v is a iso8859-1 string; u8 is a utf-8 string>>> v.decode('iso8859-1') == u8False             # v.decode(...) returns unicode>>> u8.decode('utf-8') == v.decode('latin1') == u16.decode('utf-16')True              # all decode to the same unicode memory representation                  # (latin1 is iso-8859-1)Unicode例外 >>> u8.encode('iso8859-1')Traceback (most recent call last):  File "", line 1, in UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0:  ordinal not in range(128)>>> u16.encode('iso8859-1')Traceback (most recent call last):  File "", line 1, in UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0:  ordinal not in range(128)>>> v.encode('iso8859-1')Traceback (most recent call last):  File "", line 1, in UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 0:  ordinal not in range(128)可以通过将特定编码(latin-1，utf8，utf16)转换为unicode来解决这些问题u8.decode('utf8').encode('latin1')。因此，也许可以得出以下原理和概括：类型str是一组字节，可以具有多种编码中的一种，例如Latin-1，UTF-8和UTF-16类型unicode是一组字节，可以转换为任意数量的编码，最常见的是UTF-8和latin-1(iso8859-1)该print命令具有自己的编码逻辑，设置为sys.stdout.encoding并且默认为UTF-8str在转换为另一种编码之前，必须先将a解码为unicode。当然，所有这些变化在Python 3.x中都有。希望那是照亮的。