核心知识点:
- NLP 基础知识:标记化、词干提取、词袋
- 如何预处理数据并将
nltk
其输入到神经网络 - 如何在 Pytorch 中实现前馈神经网络并进行训练
- 该实现应该对初学者来说很容易理解,并提供对聊天机器人的基本理解。
- 使用具有 2 个隐藏层的前馈神经网络可以很简单地实现。
- 自定义您自己的数据集非常简单。只需修改
intents.json
可能的模式和响应并重新运行训练即可
1)设置你的环境
让我们首先设置虚拟环境并安装PyTorch
和nltk
。
创建环境
随你喜欢(例如conda
或venv
)
mkdir myproject
$ cd myproject
$ python3 -m venv venv
激活它
Mac / Linux:
. venv/bin/activate
视窗:
venv\Scripts\activate
安装 PyTorch 和依赖项
有关 PyTorch 的安装请参阅官方网站。
您还需要nltk
:
pip install nltk
如果在第一次运行期间出现错误,您还需要安装nltk.tokenize.punkt
:在终端中运行一次:
$ python
>>> import nltk
>>> nltk.download('punkt')
2)创建训练数据
我们需要在 json 文件中创建训练数据(intents.json
)。它具有以下结构:
{
"intents": [
{
"tag": "greeting",
"patterns": [
"Hi",
"Hey",
"How are you",
"Is anyone there?",
"Hello",
"Good day"
],
"responses": [
"Hey :-)",
"Hello, thanks for visiting",
"Hi there, what can I do for you?",
"Hi there, how can I help?"
]
},
...
]
}
您可以根据自己的用例进行自定义。只需为聊天机器人定义一个新的tag
、可能的patterns
和可能的。每当修改此文件时,您都必须重新运行训练。responses
3)NLP 基础
我们不能直接将输入句子传递给我们的神经网络。我们必须以某种方式将模式字符串转换为网络可以理解的数字。为此,我们将每个句子转换为所谓的词袋(bow)。为此,我们需要收集训练词,即我们的机器人可以在训练数据中查看的所有单词。基于所有这些单词,我们可以计算每个新句子的词袋。词袋的大小与所有单词数组相同,如果单词在传入的句子中可用,则每个位置包含 1,否则包含 0。这是一个直观的示例:
在我们计算弓之前,我们应用了另外两种 NLP 技术:标记化和词干提取。
- 标记化:将字符串拆分为有意义的单元(例如单词、标点符号、数字)
例子:
"what would you do with 1000000$?"
[ “what”, “would”, “you”, “do”, “with”, “1000000”, “$”, “?”]
- 词干提取 生成单词的词根形式。这是一种粗略的启发式方法,可以切断单词的结尾
例子:
[“organize”, “organizes”, “organizing”]
[ “organ”, “organ”, “organ”]
对于标签,我们按字母顺序对其进行排序,然后使用索引作为类标签。我们的整个预处理流程如下所示:
4)实现 NLP 实用程序
为此,我们使用nltk
模块。NLTK(自然语言工具包)是用于构建处理人类语言数据的 Python 程序的领先平台。它提供了许多有用的方法供我们使用。
# nltk_utils.py
import numpy as np
import nltk
# nltk.download('punkt')
from nltk.stem.porter import PorterStemmer
stemmer = PorterStemmer()
def tokenize(sentence):
"""
split sentence into array of words/tokens
a token can be a word or punctuation character, or number
"""
return nltk.word_tokenize(sentence)
def stem(word):
"""
stemming = find the root form of the word
examples:
words = ["organize", "organizes", "organizing"]
words = [stem(w) for w in words]
-> ["organ", "organ", "organ"]
"""
return stemmer.stem(word.lower())
def bag_of_words(tokenized_sentence, words):
"""
return bag of words array:
1 for each known word that exists in the sentence, 0 otherwise
example:
sentence = ["hello", "how", "are", "you"]
words = ["hi", "hello", "I", "you", "bye", "thank", "cool"]
bog = [ 0 , 1 , 0 , 1 , 0 , 0 , 0]
"""
# stem each word
sentence_words = [stem(word) for word in tokenized_sentence]
# initialize bag with 0 for each word
bag = np.zeros(len(words), dtype=np.float32)
for idx, w in enumerate(words):
if w in sentence_words:
bag[idx] = 1
return bag
5)实现神经网络
使用具有 2 个隐藏层的前馈神经网络可以很容易地实现:
# model.py
import torch
import torch.nn as nn
class NeuralNet(nn.Module):
def __init__(self, input_size, hidden_size, num_classes):
super(NeuralNet, self).__init__()
self.l1 = nn.Linear(input_size, hidden_size)
self.l2 = nn.Linear(hidden_size, hidden_size)
self.l3 = nn.Linear(hidden_size, num_classes)
self.relu = nn.ReLU()
def forward(self, x):
out = self.l1(x)
out = self.relu(out)
out = self.l2(out)
out = self.relu(out)
out = self.l3(out)
# no activation and no softmax at the end
return out
6)实施训练流程
把所有内容放在一起:
# train.py
import numpy as np
import random
import json
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
from nltk_utils import bag_of_words, tokenize, stem
from model import NeuralNet
with open('intents.json', 'r') as f:
intents = json.load(f)
all_words = []
tags = []
xy = []
# loop through each sentence in our intents patterns
for intent in intents['intents']:
tag = intent['tag']
# add to tag list
tags.append(tag)
for pattern in intent['patterns']:
# tokenize each word in the sentence
w = tokenize(pattern)
# add to our words list
all_words.extend(w)
# add to xy pair
xy.append((w, tag))
# stem and lower each word
ignore_words = ['?', '.', '!']
all_words = [stem(w) for w in all_words if w not in ignore_words]
# remove duplicates and sort
all_words = sorted(set(all_words))
tags = sorted(set(tags))
# create training data
X_train = []
y_train = []
for (pattern_sentence, tag) in xy:
# X: bag of words for each pattern_sentence
bag = bag_of_words(pattern_sentence, all_words)
X_train.append(bag)
# y: PyTorch CrossEntropyLoss needs only class labels, not one-hot
label = tags.index(tag)
y_train.append(label)
X_train = np.array(X_train)
y_train = np.array(y_train)
# Hyper-parameters
num_epochs = 1000
batch_size = 8
learning_rate = 0.001
input_size = len(X_train[0])
hidden_size = 8
output_size = len(tags)
class ChatDataset(Dataset):
def __init__(self):
self.n_samples = len(X_train)
self.x_data = X_train
self.y_data = y_train
# support indexing such that dataset[i] can be used to get i-th sample
def __getitem__(self, index):
return self.x_data[index], self.y_data[index]
# we can call len(dataset) to return the size
def __len__(self):
return self.n_samples
dataset = ChatDataset()
train_loader = DataLoader(dataset=dataset,
batch_size=batch_size,
shuffle=True,
num_workers=2)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = NeuralNet(input_size, hidden_size, output_size).to(device)
# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
# Train the model
for epoch in range(num_epochs):
for (words, labels) in train_loader:
words = words.to(device)
labels = labels.to(device)
# Forward pass
outputs = model(words)
# if y would be one-hot, we must apply
# labels = torch.max(labels, 1)[1]
loss = criterion(outputs, labels)
# Backward and optimize
optimizer.zero_grad()
loss.backward()
optimizer.step()
if (epoch+1) % 100 == 0:
print (f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')
print(f'final loss: {loss.item():.4f}')
data = {
"model_state": model.state_dict(),
"input_size": input_size,
"hidden_size": hidden_size,
"output_size": output_size,
"all_words": all_words,
"tags": tags
}
FILE = "data.pth"
torch.save(data, FILE)
print(f'training complete. file saved to {FILE}')
7)实现聊天功能
加载训练好的模型并对新句子进行预测:
# chat.py
import random
import json
import torch
from model import NeuralNet
from nltk_utils import bag_of_words, tokenize
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
with open('intents.json', 'r') as json_data:
intents = json.load(json_data)
FILE = "data.pth"
data = torch.load(FILE)
input_size = data["input_size"]
hidden_size = data["hidden_size"]
output_size = data["output_size"]
all_words = data['all_words']
tags = data['tags']
model_state = data["model_state"]
model = NeuralNet(input_size, hidden_size, output_size).to(device)
model.load_state_dict(model_state)
model.eval()
bot_name = "Sam"
print("Let's chat! (type 'quit' to exit)")
while True:
# sentence = "do you use credit cards?"
sentence = input("You: ")
if sentence == "quit":
break
sentence = tokenize(sentence)
X = bag_of_words(sentence, all_words)
X = X.reshape(1, X.shape[0])
X = torch.from_numpy(X).to(device)
output = model(X)
_, predicted = torch.max(output, dim=1)
tag = tags[predicted.item()]
probs = torch.softmax(output, dim=1)
prob = probs[0][predicted.item()]
if prob.item() > 0.75:
for intent in intents['intents']:
if tag == intent["tag"]:
print(f"{bot_name}: {random.choice(intent['responses'])}")
else:
print(f"{bot_name}: I do not understand...")
8)使用
恭喜!您已实现聊天机器人!现在只需运行训练并开始聊天😊。
跑步
python train.py
这将转储data.pth
文件。然后运行
python chat.py
正如开头提到的,您可以根据自己的需求进行自定义。只需修改intents.json
可能的模式和响应,然后重新运行训练即可。