python 输入一个自然数 输出二进制,如何在Python中写入原始二进制数据？

I've got a Python program that stores and writes data to a file. The data is raw binary data, stored internally as str. I'm writing it out through a utf-8 codec. However, I get UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 25: character maps to in the cp1252.py file.

This looks to me like Python is trying to interpret the data using the default code page. But it doesn't have a default code page. That's why I'm using str, not unicode.

I guess my questions are:

How do I represent raw binary data in memory, in Python?

When I'm writing raw binary data out through a codec, how do I encode/unencode it?

解决方案

NOTE: this was written for Python 2.x. Not sure if applicable to 3.x.

Your use of str for raw binary data in memory is correct.

[If you're using Python 2.6+, it's even better to use bytes which in 2.6+ is just an alias to str but expresses your intention better, and will help if one day you port the code to Python 3.]

As others note, writing binary data through a codec is strange. A write codec takes unicode and outputs bytes into the file. You're trying to do it backwards, hence our confusion about your intentions...

[And your diagnosis of the error looks correct: since the codec expects unicode, Python is decoding your str into unicode with the system's default encoding, which chokes.]

What you want to see in the output file?

If the file should contain the binary data as-is:

Then you must not send it through a codec; you must write it

directly to the file. A codec encodes everything and can only

emit valid encodings of unicode (in your case, valid UTF-8).

There is no input you can give it to make it emit arbitrary

byte sequences!

If you require a mixture of UTF-8 and raw binary data, you

should open the file directly, and intermix writes of some_data

with some_text.encode('utf8')...

Note however that mixing UTF-8 with raw arbitrary data is very

bad design, because such files are very inconvenient to deal

with! Tools that understand unicode will choke on the binary

data, leaving you with not convenient way to even view (let alone

modify) the file.

If you want a friendly representation of arbitrary bytes in

unicode:

Pass data.encode('base64') to the codec. Base64 produces only

clean ascii (letters, numbers, and a little punctuation) so it

can be clearly embedded in anything, it clearly looks to people as

binary data, and it's reasonably compact (slightly over 33%

overhead).

P.S. you may note that data.encode('base64') is strange.

.encode() is supposed to take unicode but I'm giving it a

string?! Python has several pseudo-codecs that convert str->str

such as 'base64' and 'zlib'.

.encode() always returns an str but you'll feed it into a codec

expecting unicode?! In this case it will only contain clean

ascii, so it doesn't matter. You may write explicitly

data.encode('base64').encode('utf8') if it makes you feel

better.

If you need a 1:1 mapping from arbitrary bytes to unicode:

Pass data.decode('latin1') to the codec. latin1 maps

bytes 0-255 to unicode characters 0-255, which is kinda elegant.

The codec will, of course, encode your characters - 128-255 are

encoded as 2 or 3 bytes in UTF-8 (surprisingly, the average

overhead is 50%, more than base64!). This quite kills the

"elegance" of having a 1:1 mapping.

Note also that unicode characters 0-255 include nasty

invisible/control characters (newline, formfeed, soft hyphen, etc.)

making your binary data annoying to view in text editors.

Considering these drawbacks, I do not recommend latin1 unless

you understand exactly why you want it.

I'm just mentioning it as the other "natural" encoding that springs

to mind.