《实现细节》字符索引向字词索引的转化代码

示例

在阅读理解任务中，答案往往是给字符的起始位置的索引，如SQuAD1.1数据集中的一个示例如下。

context：
Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi’s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the “golden anniversary” with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as “Super Bowl L”), so that the logo could prominently feature the Arabic numerals 50.

answer：

但是，这个answer_start往往不是我们需要的索引，所以需要将其转化为Denver Broncos在这个context中的单词索引位置。

代码

context:

she loves this puppy.

question:

what does this girl love?

answer:

puppy

answer_start:

15

---

tokens:

[‘she’,‘loves’,‘this’,‘puppy’,‘.’]

punctuations:

[’ ‘,’\n’]

idx = 0
flag = False
s_idx = answer_start
e_idx = s_idx + len(answer)

for i,t in enumerate(tokens):
	# 计算包含空格、换行在内的字符
	while idx < len(context):
	    if context[idx] in punctuations:
	        idx += 1
	    else:
	        break
	# 把该单词的长度计算在内
	idx += len(t)
	if idx > s_idx and flag == False:
	    # 词序列的开始位置
	    s_idx = i
	    flag = True
	if idx >= e_idx:
	    # 词序列的结束位置
	    e_idx = i
	    break