一步步读懂Pytorch Chatbot Tutorial代码(二) - 数据处理

最新推荐文章于 2021-12-03 11:20:43 发布

HHVic

最新推荐文章于 2021-12-03 11:20:43 发布

阅读量990

点赞数 2

分类专栏： Pytorch 文章标签： python pycharm 开发语言

本文链接：https://blog.csdn.net/landian0531/article/details/121354786

版权

Pytorch 专栏收录该内容

11 篇文章 2 订阅

订阅专栏

文章目录

自述
代码出处
目录
代码 Create formatted data file （为了方便理解，把代码的顺序略微改一下, 此章节略长。）

自述

我是编程小白，别看注册时间长，但从事的不是coding工作，为了学AI才开始自学Python。
平时就是照着书上敲敲代码，并没有深刻理解。现在想要研究chatbot了，才发现自己的coding水平急需加强，所以开这个系列记录自己一行行扣代码的过程。当然这不是从0开始的，只是把自己不理解的写出来，将来也可以作为资料备查。

最后还要重申一下，我没有系统学过编程，写这个系列就是想突破自己，各位大神请不吝赐教！

代码出处

Pytorch的CHATBOT TUTORIAL

https://pytorch.org/tutorials/beginner/chatbot_tutorial.html?highlight=gpu%20training

一步步读懂Pytorch Chatbot Tutorial代码(一) - 加载和预处理数据
 一步步读懂Pytorch Chatbot Tutorial代码(二) - 数据处理
 一步步读懂Pytorch Chatbot Tutorial代码(三) - 创建字典
 一步步读懂Pytorch Chatbot Tutorial代码(四) - 为模型准备数据
 一步步读懂Pytorch Chatbot Tutorial代码(五) - 定义模型

代码 Create formatted data file （为了方便理解，把代码的顺序略微改一下, 此章节略长。）

为了方便，通过下面代码将会创建一个标准格式的文件，包含每一行用TAB分隔的查询语句（quary sentence）和一个响应语句对（response sentence pair）

1. `loadLines` 将文件的每一行拆分为一个字段字典（lineID、characterID、movieID、character、text）

# Splits each line of the file into a dictionary of fields
lines = {}
MOVIE_LINES_FIELDS = ["lineID", "characterID", "movieID", "character", "text"]
def loadLines(fileName, fields):
    lines = {}
    with open(fileName, 'r', encoding='iso-8859-1') as f:
        for line in f:
            values = line.split(" +++$+++ ") 
            # Extract fields
            lineObj = {}
            for i, field in enumerate(fields):
                lineObj[field] = values[i]
            lines[lineObj['lineID']] = lineObj
    return lines               

lines = loadLines(os.path.join(corpus, "movie_lines.txt"), MOVIE_LINES_FIELDS)

查看字典`lines`内容

list(lines.items())[0]

('L1045',
 {'lineID': 'L1045',
  'characterID': 'u0',
  'movieID': 'm0',
  'character': 'BIANCA',
  'text': 'They do not!\n'})

encoding=‘iso-8859-1’

属于单字节编码,最多能表示的字符范围是0-255,应用于英文系列。无法表示中文字符。

更多编码内容可以参考https://www.cnblogs.com/huangchenggener/p/10983866.html

line.split（‘ + + + $ + + + ’）

split()方法：通过指定分隔符对字符串进行切片所以print(values) 得到结果：

['L1045 ', ' u0 ', ' m0 ', ' BIANCA ', ' They do not!\n']
['L1044 ', ' u2 ', ' m0 ', ' CAMERON ', ' They do to!\n']
['L985 ', ' u0 ', ' m0 ', ' BIANCA ', ' I hope so.\n']
['L984 ', ' u2 ', ' m0 ', ' CAMERON ', ' She okay?\n']
['L925 ', ' u0 ', ' m0 ', ' BIANCA ', " Let's go.\n"]
['L924 ', ' u2 ', ' m0 ', ' CAMERON ', ' Wow\n']
['L872 ', ' u0 ', ' m0 ', ' BIANCA ', " Okay -- you're gonna need to learn how to lie.\n"]
['L871 ', ' u2 ', ' m0 ', ' CAMERON ', ' No\n']
['L870 ', ' u0 ', ' m0 ', ' BIANCA ', ' I\'m kidding.  You know how sometimes you just become this "persona"?  And you don\'t know how to quit?\n']
...

enumerate

参考https://blog.csdn.net/landian0531/article/details/120081598

for i, field in enumerate(fields): 将fields按照字典{i : field}方式排列, 即

print(i,field) 得到如下结果：

0 lineID
1 characterID
2 movieID
3 character
4 text
0 lineID
1 characterID
2 movieID
3 character
4 text
0 lineID
1 characterID
2 movieID
3 character
4 text
0 lineID
1 characterID
2 movieID
3 character
4 text
0 lineID
1 characterID
2 movieID
3 character
4 text
0 lineID
1 characterID
2 movieID
3 character
4 text
......

字典添加键值对

举个栗子：从栗子中可以看出字典的添加非常的方便！

a = {'数学':95}
print(a)
#添加新键值对
a['语文'] = 89
print(a)
#再次添加新键值对
a['英语'] = 90
print(a)

{'数学': 95}
{'数学': 95, '语文': 89}
{'数学': 95, '语文': 89, '英语': 90}

回到代码中，由于初始字典lineObj={} 是空值
所以第一句lineObj[field]=values[i] 即往字典lineObj中加入新的键 / 值对, 循环多次后print(lineObj)得到的结果:

{'lineID': 'L1045 ', 'characterID': ' u0 ', 'movieID': ' m0 ', 'character': ' BIANCA ', 'text': ' They do not!\n'}
{'lineID': 'L1044 ', 'characterID': ' u2 ', 'movieID': ' m0 ', 'character': ' CAMERON ', 'text': ' They do to!\n'}
{'lineID': 'L985 ', 'characterID': ' u0 ', 'movieID': ' m0 ', 'character': ' BIANCA ', 'text': ' I hope so.\n'}
......

lines[lineObj[‘lineID’]] = lineObj

同上，这句代码把当前lineObj里面的lineID作为键，当前lineObj作为lineID的值。然后字典lines 结果为：

{'L1045': {'lineID': 'L1045',
  'characterID': 'u0',
  'movieID': 'm0',
  'character': 'BIANCA',
  'text': 'They do not!\n'},
 'L1044': {'lineID': 'L1044',
  'characterID': 'u2',
  'movieID': 'm0',
  'character': 'CAMERON',
  'text': 'They do to!\n'},
 'L985': {'lineID': 'L985',
  'characterID': 'u0',
  'movieID': 'm0',
  'character': 'BIANCA',
  'text': 'I hope so.\n'},
  ......

2. `loadConversationsloadLines`根据movie_conversations.txt将行的字段分组到对话中

由于这段代码和上面的功能非常相似，下面只列出各个阶段的变量结果，便于理解。

#Groups fields of lines from 'loadLines' into conversations based on *movie_conversation.txt*
MOVIE_CONVERSATIONS_FIELDS=['character1ID','character2ID','movieID','utteranceIDs']

def loadConversations(fileName, lines, fields):
    conversations = []
    with open(fileName, 'r', encoding='iso-8859-1') as f:
        for line in f:
            values = line.split(" +++$+++ ")
            # Extract fields
            convObj = {}
            for i, field in enumerate(fields):
                convObj[field] = values[i]
            # Convert string to list (convObj["utteranceIDs"] == "['L598485', 'L598486', ...]")
            utterance_id_pattern = re.compile('L[0-9]+')
            lineIds = utterance_id_pattern.findall(convObj["utteranceIDs"])
            # Reassemble lines
            convObj["lines"] = []
            for lineId in lineIds:
                convObj["lines"].append(lines[lineId])
            conversations.append(convObj)
    return conversations
conversations=loadConversations(os.path.join(corpus,'movie_conversations.txt'),lines,MOVIE_CONVERSATIONS_FIELDS)

查看movie_conversations的内容及格式

file=os.path.join(corpus,'movie_conversations.txt')

with open(file,'r') as datafile:
    lines=datafile.readlines()
    for line in lines[:10]:
        print(line)

u0 +++$+++ u2 +++$+++ m0 +++$+++ ['L194', 'L195', 'L196', 'L197']

u0 +++$+++ u2 +++$+++ m0 +++$+++ ['L198', 'L199']

u0 +++$+++ u2 +++$+++ m0 +++$+++ ['L200', 'L201', 'L202', 'L203']

u0 +++$+++ u2 +++$+++ m0 +++$+++ ['L204', 'L205', 'L206']

u0 +++$+++ u2 +++$+++ m0 +++$+++ ['L207', 'L208']

u0 +++$+++ u2 +++$+++ m0 +++$+++ ['L271', 'L272', 'L273', 'L274', 'L275']

u0 +++$+++ u2 +++$+++ m0 +++$+++ ['L276', 'L277']

u0 +++$+++ u2 +++$+++ m0 +++$+++ ['L280', 'L281']

u0 +++$+++ u2 +++$+++ m0 +++$+++ ['L363', 'L364']

u0 +++$+++ u2 +++$+++ m0 +++$+++ ['L365', 'L366']

line.split（‘ + + + $ + + + ’）

print(values) 得到结果：

['u0', 'u2', 'm0', "['L194', 'L195', 'L196', 'L197']\n"]
['u0', 'u2', 'm0', "['L198', 'L199']\n"]
['u0', 'u2', 'm0', "['L200', 'L201', 'L202', 'L203']\n"]
['u0', 'u2', 'm0', "['L204', 'L205', 'L206']\n"]
['u0', 'u2', 'm0', "['L207', 'L208']\n"]
['u0', 'u2', 'm0', "['L271', 'L272', 'L273', 'L274', 'L275']\n"]
['u0', 'u2', 'm0', "['L276', 'L277']\n"]
['u0', 'u2', 'm0', "['L280', 'L281']\n"]

enumerate

print(convObj) 的结果：

{'character1ID': 'u0'}
{'character1ID': 'u0', 'character2ID': 'u2'}
{'character1ID': 'u0', 'character2ID': 'u2', 'movieID': 'm0'}
{'character1ID': 'u0', 'character2ID': 'u2', 'movieID': 'm0', 'utteranceIDs': "['L194', 'L195', 'L196', 'L197']\n"}
{'character1ID': 'u0'}
{'character1ID': 'u0', 'character2ID': 'u2'}
{'character1ID': 'u0', 'character2ID': 'u2', 'movieID': 'm0'}
{'character1ID': 'u0', 'character2ID': 'u2', 'movieID': 'm0', 'utteranceIDs': "['L198', 'L199']\n"}
{'character1ID': 'u0'}
{'character1ID': 'u0', 'character2ID': 'u2'}
{'character1ID': 'u0', 'character2ID': 'u2', 'movieID': 'm0'}
{'character1ID': 'u0', 'character2ID': 'u2', 'movieID': 'm0', 'utteranceIDs': "['L200', 'L201', 'L202', 'L203']\n"}
{'character1ID': 'u0'}
{'character1ID': 'u0', 'character2ID': 'u2'}
{'character1ID': 'u0', 'character2ID': 'u2', 'movieID': 'm0'}
{'character1ID': 'u0', 'character2ID': 'u2', 'movieID': 'm0', 'utteranceIDs': "['L204', 'L205', 'L206']\n"}

re.compile

re 模块的一般使用步骤如下：

使用 compile 函数将正则表达式的字符串形式编译为一个 Pattern 对象
通过 Pattern 对象提供的一系列方法对文本进行匹配查找，获得匹配结果（一个 Match 对象）
最后使用 Match 对象提供的属性和方法获得信息，根据需要进行其他的操作

compile()与findall()一起使用，返回一个列表。

‘L[0-9]+’ 是正则表达式，代表L开头，[0-9]+匹配1个或者多个数字

print(lineIds) 的结果为

['L194', 'L195', 'L196', 'L197']
['L198', 'L199']
['L200', 'L201', 'L202', 'L203']
['L204', 'L205', 'L206']
['L207', 'L208']
['L271', 'L272', 'L273', 'L274', 'L275']
['L276', 'L277']
['L280', 'L281']
['L363', 'L364']
['L365', 'L366']

字典添加键值对

convObj["lines"] = [] ：添加新的键lines，值为空到字典convObj

print(convObj) 的结果：

{'character1ID': 'u0', 'character2ID': 'u2', 'movieID': 'm0', 'utteranceIDs': "['L194', 'L195', 'L196', 'L197']\n", 'lines': []}
{'character1ID': 'u0', 'character2ID': 'u2', 'movieID': 'm0', 'utteranceIDs': "['L198', 'L199']\n", 'lines': []}
{'character1ID': 'u0', 'character2ID': 'u2', 'movieID': 'm0', 'utteranceIDs': "['L200', 'L201', 'L202', 'L203']\n", 'lines': []}
{'character1ID': 'u0', 'character2ID': 'u2', 'movieID': 'm0', 'utteranceIDs': "['L204', 'L205', 'L206']\n", 'lines': []}
{'character1ID': 'u0', 'character2ID': 'u2', 'movieID': 'm0', 'utteranceIDs': "['L207', 'L208']\n", 'lines': []}
{'character1ID': 'u0', 'character2ID': 'u2', 'movieID': 'm0', 'utteranceIDs': "['L271', 'L272', 'L273', 'L274', 'L275']\n", 'lines': []}
......

append

append() 方法用于在列表末尾添加新的对象

for lineId in lineIds:
            convObj["lines"].append(lines[lineId])

这里通过上面re.compile 得到的lineIds，再经过for循环来提取其中的数值（第一个lineId是L194）。然后加上上一段代码返回的字典 lines，以及里面的键lineId 其对应的值L194，添加到字典convObj 中作为键 ‘lines’的值

所以print(convObj['lines'] 的结果：（其实和上一段return的lines内容一致，但是顺序变为以L194开头）

[{'lineID': 'L194', 'characterID': 'u0', 'movieID': 'm0', 'character': 'BIANCA', 'text': 'Can we make this quick?  Roxanne Korrine and Andrew Barrett are having an incredibly horrendous public break- up on the quad.  Again.\n'}]
[{'lineID': 'L194', 'characterID': 'u0', 'movieID': 'm0', 'character': 'BIANCA', 'text': 'Can we make this quick?  Roxanne Korrine and Andrew Barrett are having an incredibly horrendous public break- up on the quad.  Again.\n'}, {'lineID': 'L195', 'characterID': 'u2', 'movieID': 'm0', 'character': 'CAMERON', 'text': "Well, I thought we'd start with pronunciation, if that's okay with you.\n"}]
[{'lineID': 'L194', 'characterID': 'u0', 'movieID': 'm0', 'character': 'BIANCA', 'text': 'Can we make this quick?  Roxanne Korrine and Andrew Barrett are having an incredibly horrendous public break- up on the quad.  Again.\n'}, {'lineID': 'L195', 'characterID': 'u2', 'movieID': 'm0', 'character': 'CAMERON', 'text': "Well, I thought we'd start with pronunciation, if that's okay with you.\n"}, {'lineID': 'L196', 'characterID': 'u0', 'movieID': 'm0', 'character': 'BIANCA', 'text': 'Not the hacking and gagging and spitting part.  Please.\n'}]
[{'lineID': 'L194', 'characterID': 'u0', 'movieID': 'm0', 'character': 'BIANCA', 'text': 'Can we make this quick?  Roxanne Korrine and Andrew Barrett are having an incredibly horrendous public break- up on the quad.  Again.\n'}, {'lineID': 'L195', 'characterID': 'u2', 'movieID': 'm0', 'character': 'CAMERON', 'text': "Well, I thought we'd start with pronunciation, if that's okay with you.\n"}, {'lineID': 'L196', 'characterID': 'u0', 'movieID': 'm0', 'character': 'BIANCA', 'text': 'Not the hacking and gagging and spitting part.  Please.\n'}, {'lineID': 'L197', 'characterID': 'u2', 'movieID': 'm0', 'character': 'CAMERON', 'text': "Okay... then how 'bout we try out some French cuisine.  Saturday?  Night?\n"}]

conversations

最终print(conversations) 结果如下：

{'character1ID': 'u0', 'character2ID': 'u2', 'movieID': 'm0', 'utteranceIDs': "['L194', 'L195', 'L196', 'L197']\n", 'lines': [{'lineID': 'L194', 'characterID': 'u0', 'movieID': 'm0', 'character': 'BIANCA', 'text': 'Can we make this quick?  Roxanne Korrine and Andrew Barrett are having an incredibly horrendous public break- up on the quad.  Again.\n'}, {'lineID': 'L195', 'characterID': 'u2', 'movieID': 'm0', 'character': 'CAMERON', 'text': "Well, I thought we'd start with pronunciation, if that's okay with you.\n"}, {'lineID': 'L196', 'characterID': 'u0', 'movieID': 'm0', 'character': 'BIANCA', 'text': 'Not the hacking and gagging and spitting part.  Please.\n'}, {'lineID': 'L197', 'characterID': 'u2', 'movieID': 'm0', 'character': 'CAMERON', 'text': "Okay... then how 'bout we try out some French cuisine.  Saturday?  Night?\n"}]}
{'character1ID': 'u0', 'character2ID': 'u2', 'movieID': 'm0', 'utteranceIDs': "['L198', 'L199']\n", 'lines': [{'lineID': 'L198', 'characterID': 'u0', 'movieID': 'm0', 'character': 'BIANCA', 'text': "You're asking me out.  That's so cute. What's your name again?\n"}, {'lineID': 'L199', 'characterID': 'u2', 'movieID': 'm0', 'character': 'CAMERON', 'text': 'Forget it.\n'}]}
......

3.`extractSentencePairs` 从对话中提取句子对

# Extracts pairs of sentences from conversations
def extractSentencePairs(conversations):
    qa_pairs = []
    for conversation in conversations:
        # Iterate over all the lines of the conversation
        for i in range(len(conversation["lines"]) - 1):  # We ignore the last line (no answer for it)
            inputLine = conversation["lines"][i]["text"].strip()
            targetLine = conversation["lines"][i+1]["text"].strip()
            # Filter wrong samples (if one of the lists is empty)
            if inputLine and targetLine:
                qa_pairs.append([inputLine, targetLine])
    return qa_pairs

循环的妙用

根据第一句循环for conversation in conversations 将conversations 化整为零，所以 conversations[0] 即 conversation，结果如下：

{'character1ID': 'u0',
 'character2ID': 'u2',
 'movieID': 'm0',
 'utteranceIDs': "['L194', 'L195', 'L196', 'L197']\n",
 'lines': [{'lineID': 'L194',
   'characterID': 'u0',
   'movieID': 'm0',
   'character': 'BIANCA',
   'text': 'Can we make this quick?  Roxanne Korrine and Andrew Barrett are having an incredibly horrendous public break- up on the quad.  Again.\n'},
  {'lineID': 'L195',
   'characterID': 'u2',
   'movieID': 'm0',
   'character': 'CAMERON',
   'text': "Well, I thought we'd start with pronunciation, if that's okay with you.\n"},
  {'lineID': 'L196',
   'characterID': 'u0',
   'movieID': 'm0',
   'character': 'BIANCA',
   'text': 'Not the hacking and gagging and spitting part.  Please.\n'},
  {'lineID': 'L197',
   'characterID': 'u2',
   'movieID': 'm0',
   'character': 'CAMERON',
   'text': "Okay... then how 'bout we try out some French cuisine.  Saturday?  Night?\n"}]}

range() & len()

start: 计数从 start 开始。默认是从 0 开始。例如range（5）等价于range（0， 5）;
stop: 计数到 stop 结束，但不包括 stop。例如：range（0， 5）是[0, 1, 2, 3, 4]没有5
step：步长，默认为1。例如：range（0， 5）等价于 range(0, 5, 1)

conversations[0]['lines'] 结果如下：

[{'lineID': 'L194',
  'characterID': 'u0',
  'movieID': 'm0',
  'character': 'BIANCA',
  'text': 'Can we make this quick?  Roxanne Korrine and Andrew Barrett are having an incredibly horrendous public break- up on the quad.  Again.\n'},
 {'lineID': 'L195',
  'characterID': 'u2',
  'movieID': 'm0',
  'character': 'CAMERON',
  'text': "Well, I thought we'd start with pronunciation, if that's okay with you.\n"},
 {'lineID': 'L196',
  'characterID': 'u0',
  'movieID': 'm0',
  'character': 'BIANCA',
  'text': 'Not the hacking and gagging and spitting part.  Please.\n'},
 {'lineID': 'L197',
  'characterID': 'u2',
  'movieID': 'm0',
  'character': 'CAMERON',
  'text': "Okay... then how 'bout we try out some French cuisine.  Saturday?  Night?\n"}]

len(conversations[0]['lines'])结果：4 所以这句的用意是提取对话的次数。

conversation[“lines”][i][“text”].strip()

这句用来逐步提取对话内容，过程如下

print(conversations[0]["lines"][0]) 得到

{'lineID': 'L194',
 'characterID': 'u0',
 'movieID': 'm0',
 'character': 'BIANCA',
 'text': 'Can we make this quick?  Roxanne Korrine and Andrew Barrett are having an incredibly horrendous public break- up on the quad.  Again.\n'}

print(conversations[0]["lines"][0])['text'].strip() 得到:(提取了第一句，并通过strip()删除头尾的符号。)

'Can we make this quick?  Roxanne Korrine and Andrew Barrett are having an incredibly horrendous public break- up on the quad.  Again.'

AND

从左到右计算表达式，若所有值均为真，则返回最后一个值，若存在假，返回第一个假值。

所以但第一个inputLine 无法取值时，if inputLine and targetLine: 语句终止。

qa_pairs

print(qa_pairs[0] 结果：

['Can we make this quick?  Roxanne Korrine and Andrew Barrett are having an incredibly horrendous public break- up on the quad.  Again.', "Well, I thought we'd start with pronunciation, if that's okay with you."]

4. 调用这些函数并创建文件

这里注释掉的代码已经移到上面代码中。


# Define path to new file
datafile = os.path.join(corpus, "formatted_movie_lines.txt")

delimiter = '\t'
# Unescape the delimiter
delimiter = str(codecs.decode(delimiter, "unicode_escape"))

# Initialize lines dict, conversations list, and field ids
#lines = {}
#conversations = []
#MOVIE_LINES_FIELDS = ["lineID", "characterID", "movieID", "character", "text"]
#MOVIE_CONVERSATIONS_FIELDS = ["character1ID", "character2ID", "movieID", "utteranceIDs"]

# Load lines and process conversations
#print("\nProcessing corpus...")
#lines = loadLines(os.path.join(corpus, "movie_lines.txt"), #MOVIE_LINES_FIELDS)
#print("\nLoading conversations...")
#conversations = loadConversations(os.path.join(corpus, "movie_conversations.txt"),lines, MOVIE_CONVERSATIONS_FIELDS)

# Write new csv file
print("\nWriting newly formatted file...")
with open(datafile, 'w', encoding='utf-8') as outputfile:
    writer = csv.writer(outputfile, delimiter=delimiter, lineterminator='\n')
    for pair in extractSentencePairs(conversations):
        writer.writerow(pair)

# Print a sample of lines
print("\nSample lines from file:")
printLines(datafile)

codecs.decode

关于编码问题，可详细阅读下面文章。
https://www.jb51.net/article/92006.htm

其中这句会体现在这段代码的输出中：在Python3中，以字节形式表示的字符串则必须加上前缀b，也就是写成上文的b’xxxx’形式。

另外确实不太理解加 ‘\t’ 为什么要做编码转换，留到以后懂了再来填坑。

lineterminator

delimiter=’\t’就是使用制表符代替逗号来分隔单元格，lineterminator=’\n’就是设置一倍行距（’\n\n’是两倍行距）

write.writerow 和 write.writerows

writerow 单行写入
writerows 多行写入

printLines(datafile)

Writing newly formatted file...

Sample lines from file:
b"Can we make this quick?  Roxanne Korrine and Andrew Barrett are having an incredibly horrendous public break- up on the quad.  Again.\tWell, I thought we'd start with pronunciation, if that's okay with you.\n"
b"Well, I thought we'd start with pronunciation, if that's okay with you.\tNot the hacking and gagging and spitting part.  Please.\n"
b"Not the hacking and gagging and spitting part.  Please.\tOkay... then how 'bout we try out some French cuisine.  Saturday?  Night?\n"
b"You're asking me out.  That's so cute. What's your name again?\tForget it.\n"
b"No, no, it's my fault -- we didn't have a proper introduction ---\tCameron.\n"
b"Cameron.\tThe thing is, Cameron -- I'm at the mercy of a particularly hideous breed of loser.  My sister.  I can't date until she does.\n"
b"The thing is, Cameron -- I'm at the mercy of a particularly hideous breed of loser.  My sister.  I can't date until she does.\tSeems like she could get a date easy enough...\n"
b'Why?\tUnsolved mystery.  She used to be really popular when she started high school, then it was just like she got sick of it or something.\n'

HHVic

关注

2
点赞
踩
3

收藏

觉得还不错? 一键收藏
0
评论
一步步读懂Pytorch Chatbot Tutorial代码(二) - 数据处理

文章目录自述代码出处目录代码 Create formatted data file （为了方便理解，把代码的顺序略微改一下）1. `loadLines` 将文件的每一行拆分为一个字段字典（lineID、characterID、movieID、character、text）encoding='iso-8859-1'values的值：（删掉‘+++$+++’）enumerate修改字典lines[lineObj['lineID']] = lineObj2. `loadConversationsloadLine
复制链接

扫一扫