python偏移量口诀_将字符偏移量转换为字节偏移量（在Python中）

Suppose I have a bunch of files in UTF-8 that I send to an external API in unicode. The API operates on each unicode string and returns a list with (character_offset, substr) tuples.

The output I need is the begin and end byte offset for each found substring. If I'm lucky the input text contains only ASCII characters (making character offset and byte offset identical), but this is not always the case. How can I find the begin and end byte offsets for a known begin character offset and substring?

I've answered this question myself, but look forward to other solutions to this problem that are more robust, more efficient, and/or more readable.

解决方案

I'd solve this using a dictionary mapping character offsets to byte offsets and then looking up the offsets in that.

def get_char_to_byte_map(unicode_string):

"""

Generates a dictionary mapping character offsets to byte offsets for unicode_string.

"""

response = {}

byte_offset = 0

for char_offset, character in enumerate(unicode_string):

response[char_offset] = byte_offset

byte_offset += len(character.encode('utf-8'))

return response

char_to_byte_map = get_char_to_byte_map(text)

for begin_offset, substring in api_response:

begin_offset = char_to_byte_map[character_offset]

end_offset = char_to_byte_map[character_offset + len(substring)]

# do something

Performance of this solution as compared to yours depends a lot on the size of the input and the amount of substrings involved. Local micro-benchmarking suggests that encoding each individual character in a text takes about 1000 times as long as encoding the entire text at once.