一、前馈网络概要
1、定义
前馈神经网络(FNN)是一种最简单的神经网络结构,其中信息在一个方向上流动,从输入层到输出层,不会有反馈环路。该网络由多个神经元组成,这些神经元通常被组织成几个层,包括输入层、隐藏层和输出层。
2、特点
简单直观:前馈神经网络的结构比较简单明了,信息只朝一个方向传播,便于理解和实现。
高效性能:尽管简单,前馈神经网络在许多任务中表现出色,尤其在处理大规模数据和复杂模式时有很高的性能表现。
可扩展性:前馈神经网络可以很容易地扩展到更深、更宽的网络结构,以适应不同的任务和数据。
并行计算:由于前馈神经网络的结构,其各个神经元之间的计算是独立的,可以实现高效的并行计算加速。
适用性广泛:前馈神经网络广泛应用于图像识别、语音识别、文本分类、回归分析等多个领域,具有很好的适用性和灵活性。
易于训练:前馈神经网络通常使用反向传播算法进行训练,这是一种有效的优化算法,可以通过调整网络的权重和偏差来最小化损失函数,提高网络的性能。
二、自然语言处理与前馈神经网络
前馈神经网络的结构简单,易于实现和训练并且能够处理高维数据,而在自然语言处理中文本数据通常是高维的,故前馈神经网络十分适用于完成各种自然语言处理的任务。现在前馈神经网络被广泛应用于各种自然语言处理任务,包括但不限于以下几个方面:
-
文本分类:前馈神经网络可以用于文本分类任务,例如情感分析、垃圾邮件过滤、新闻分类等。输入文本数据经过处理后,通过前馈神经网络可以实现对文本进行自动分类。
-
语言模型:前馈神经网络可以用于构建语言模型,从而实现自然语言生成和语言理解任务。通过训练前馈神经网络,可以学习文本序列之间的模式和关系,从而生成自然流畅的文本或对文本进行理解。
-
序列标注:在词性标注、命名实体识别、文本分类等序列标注任务中,前馈神经网络也被广泛应用。通过将输入序列数据经过前馈神经网络的处理,可以实现对序列数据进行标注或分类。
-
机器翻译:前馈神经网络在机器翻译任务中也有应用。通过构建编码器-解码器结构的前馈神经网络,可以实现将一种语言的文本翻译成另一种语言的功能。
-
文本生成:前馈神经网络还可用于文本生成任务,如对话生成、摘要生成等。通过学习文本序列的模式和关系,前馈神经网络可以生成自然语言文本。
三、基于pytorch的一个简单实现
任务描述:我们将多层感知机应用于将姓氏分类到其原籍国的任务。
代码实现思路:
1、数据预处理
2、构建多层感知机模型
3、训练模型
4、预测结果
以下是具体代码部分:
1、数据预处理
数据集名为surname.csv,它从互联网上不同的姓名来源收集了了来自18个不同国家的10,000个姓氏。数据预处理的目的一是为了平衡数据集中18个国家的姓氏在数据集中的比例,均匀的比例分布有利于训练有效的模型。另外要将数据集分为三个部分:70%到训练数据集,15%到验证数据集,最后15%到测试数据集,以便跨这些部分的类标签分布具有可比性。
from argparse import Namespace from collections import Counter import json import os import string import numpy as np import pandas as pd import torch import torch.nn as nn import torch.nn.functional as F import torch.optim as optim from torch.utils.data import Dataset, DataLoader from tqdm import tqdm_notebook class Vocabulary(object): """Class to process text and extract vocabulary for mapping""" def __init__(self, token_to_idx=None, add_unk=True, unk_token="<UNK>"): """ Args: token_to_idx (dict): a pre-existing map of tokens to indices add_unk (bool): a flag that indicates whether to add the UNK token unk_token (str): the UNK token to add into the Vocabulary """ if token_to_idx is None: token_to_idx = {} self._token_to_idx = token_to_idx self._idx_to_token = {idx: token for token, idx in self._token_to_idx.items()} self._add_unk = add_unk self._unk_token = unk_token self.unk_index = -1 if add_unk: self.unk_index = self.add_token(unk_token) def to_serializable(self): """ returns a dictionary that can be serialized """ return {'token_to_idx': self._token_to_idx, 'add_unk': self._add_unk, 'unk_token': self._unk_token} @classmethod def from_serializable(cls, contents): """ instantiates the Vocabulary from a serialized dictionary """ return cls(**contents) def add_token(self, token): """Update mapping dicts based on the token. Args: token (str): the item to add into the Vocabulary Returns: index (int): the integer corresponding to the token """ try: index = self._token_to_idx[token] except KeyError: index = len(self._token_to_idx) self._token_to_idx[token] = index self._idx_to_token[index] = token return index def add_many(self, tokens): """Add a list of tokens into the Vocabulary Args: tokens (list): a list of string tokens Returns: indices (list): a list of indices corresponding to the tokens """ return [self.add_token(token) for token in tokens] def lookup_token(self, token): """Retrieve the index associated with the token or the UNK index if token isn't present. Args: token (str): the token to look up Returns: index (int): the index corresponding to the token Notes: `unk_index` needs to be >=0 (having been added into the Vocabulary) for the UNK functionality """ if self.unk_index >= 0: return self._token_to_idx.get(token, self.unk_index) else: return self._token_to_idx[token] def lookup_index(self, index): """Return the token associated with the index Args: index (int): the index to look up Returns: token (str): the token corresponding to the index Raises: KeyError: if the index is not in the Vocabulary """ if index not in self._idx_to_token: raise KeyError("the index (%d) is not in the Vocabulary" % index) return self._idx_to_token[index] def __str__(self): return "<Vocabulary(size=%d)>" % len(self) def __len__(self): return len(self._token_to_idx) class SurnameVectorizer(object): """ The Vectorizer which coordinates the Vocabularies and puts them to use""" def __init__(self, surname_vocab, nationality_vocab): """ Args: surname_vocab (Vocabulary): maps characters to integers nationality_vocab (Vocabulary): maps nationalities to integers """ self.surname_vocab = surname_vocab self.nationality_vocab = nationality_vocab def vectorize(self, surname): """ Args: surname (str): the surname Returns: one_hot (np.ndarray): a collapsed one-hot encoding """ vocab = self.surname_vocab one_hot = np.zeros(len(vocab), dtype=np.float32) for token in surname: one_hot[vocab.lookup_token(token)] = 1 return one_hot @classmethod def from_dataframe(cls, surname_df): """Instantiate the vectorizer from the dataset dataframe Args: surname_df (pandas.DataFrame): the surnames dataset Returns: an instance of the SurnameVectorizer """ surname_vocab = Vocabulary(unk_token="@") nationality_vocab = Vocabulary(add_unk=False) for index, row in surname_df.iterrows(): for letter in row.surname: surname_vocab.add_token(letter) nationality_vocab.add_token(row.nationality) return cls(surname_vocab, nationality_vocab) @classmethod def from_serializable(cls, contents): surname_vocab = Vocabulary.from_serializable(contents['surname_vocab']) nationality_vocab = Vocabulary.from_serializable(contents['nationality_vocab']) return cls(surname_vocab=surname_vocab, nationality_vocab=nationality_vocab) def to_serializable(self): return {'surname_vocab': self.surname_vocab.to_serializable(), 'nationality_vocab': self.nationality_vocab.to_serializable()} class SurnameDataset(Dataset): def __init__(self, surname_df, vectorizer): """ Args: surname_df (pandas.DataFrame): the dataset vectorizer (SurnameVectorizer): vectorizer instatiated from dataset """ self.surname_df = surname_df self._vectorizer = vectorizer self.train_df = self.surname_df[self.surname_df.split=='train'] self.train_size = len(self.train_df) self.val_df = self.surname_df[self.surname_df.split=='val'] self.validation_size = len(self.val_df) self.test_df = self.surname_df[self.surname_df.split=='test'] self.test_size = len(self.test_df) self._lookup_dict = {'train': (self.train_df, self.train_size), 'val': (self.val_df, self.validation_size), 'test': (self.test_df, self.test_size)} self.set_split('train') # Class weights class_counts = surname_df.nationality.value_counts().to_dict() def sort_key(item): return self._vectorizer.nationality_vocab.lookup_token(item[0]) sorted_counts = sorted(class_counts.items(), key=sort_key) frequencies = [count for _, count in sorted_counts] self.class_weights = 1.0 / torch.tensor(frequencies, dtype=torch.float32) @classmethod def load_dataset_and_make_vectorizer(cls, surname_csv): """Load dataset and make a new vectorizer from scratch Args: surname_csv (str): location of the dataset Returns: an instance of SurnameDataset """ surname_df = pd.read_csv(surname_csv) train_surname_df = surname_df[surname_df.split=='train'] return cls(surname_df, SurnameVectorizer.from_dataframe(train_surname_df)) @classmethod def load_dataset_and_load_vectorizer(cls, surname_csv, vectorizer_filepath): """Load dataset and the corresponding vectorizer. Used in the case in the vectorizer has been cached for re-use Args: surname_csv (str): location of the dataset vectorizer_filepath (str): location of the saved vectorizer Returns: an instance of SurnameDataset """ surname_df = pd.read_csv(surname_csv) vectorizer = cls.load_vectorizer_only(vectorizer_filepath) return cls(surname_df, vectorizer) @staticmethod def load_vectorizer_only(vectorizer_filepath): """a static method for loading the vectorizer from file Args: vectorizer_filepath (str): the location of the serialized vectorizer Returns: an instance of SurnameVectorizer """ with open(vectorizer_filepath) as fp: return SurnameVectorizer.from_serializable(json.load(fp)) def save_vectorizer(self, vectorizer_filepath): """saves the vectorizer to disk using json Args: vectorizer_filepath (str): the location to save the vectorizer """ with open(vectorizer_filepath, "w") as fp: json.dump(self._vectorizer.to_serializable(), fp) def get_vectorizer(self): """ returns the vectorizer """ return self._vectorizer def set_split(self, split="train"): """ selects the splits in the dataset using a column in the dataframe """ self._target_split = split self._target_df, self._target_size = self._lookup_dict[split] def __len__(self): return self._target_size def __getitem__(self, index): """the primary entry point method for PyTorch datasets Args: index (int): the index to the data point Returns: a dictionary holding the data point's: features (x_surname) label (y_nationality) """ row = self._target_df.iloc[index] surname_vector = \ self._vectorizer.vectorize(row.surname) nationality_index = \ self._vectorizer.nationality_vocab.lookup_token(row.nationality) return {'x_surname': surname_vector, 'y_nationality': nationality_index} def get_num_batches(self, batch_size): """Given a batch size, return the number of batches in the dataset Args: batch_size (int) Returns: number of batches in the dataset """ return len(self) // batch_size def generate_batches(dataset, batch_size, shuffle=True, drop_last=True, device="cpu"): """ A generator function which wraps the PyTorch DataLoader. It will ensure each tensor is on the write device location. """ dataloader = DataLoader(dataset=dataset, batch_size=batch_size, shuffle=shuffle, drop_last=drop_last) for data_dict in dataloader: out_data_dict = {} for name, tensor in data_dict.items(): out_data_dict[name] = data_dict[name].to(device) yield out_data_dict
2、构建多层感知机模型
class SurnameClassifier(nn.Module): """ A 2-layer Multilayer Perceptron for classifying surnames """ def __init__(self, input_dim, hidden_dim, output_dim): """ Args: input_dim (int): the size of the input vectors hidden_dim (int): the output size of the first Linear layer output_dim (int): the output size of the second Linear layer """ super(SurnameClassifier, self).__init__() self.fc1 = nn.Linear(input_dim, hidden_dim) self.fc2 = nn.Linear(hidden_dim, output_dim) def forward(self, x_in, apply_softmax=False): """The forward pass of the classifier Args: x_in (torch.Tensor): an input data tensor. x_in.shape should be (batch, input_dim) apply_softmax (bool): a flag for the softmax activation should be false if used with the Cross Entropy losses Returns: the resulting tensor. tensor.shape should be (batch, output_dim) """ intermediate_vector = F.relu(self.fc1(x_in)) prediction_vector = self.fc2(intermediate_vector) if apply_softmax: prediction_vector = F.softmax(prediction_vector, dim=1) return prediction_vector def make_train_state(args): return {'stop_early': False, 'early_stopping_step': 0, 'early_stopping_best_val': 1e8, 'learning_rate': args.learning_rate, 'epoch_index': 0, 'train_loss': [], 'train_acc': [], 'val_loss': [], 'val_acc': [], 'test_loss': -1, 'test_acc': -1, 'model_filename': args.model_state_file} def update_train_state(args, model, train_state): """Handle the training state updates. Components: - Early Stopping: Prevent overfitting. - Model Checkpoint: Model is saved if the model is better :param args: main arguments :param model: model to train :param train_state: a dictionary representing the training state values :returns: a new train_state """ # Save one model at least if train_state['epoch_index'] == 0: torch.save(model.state_dict(), train_state['model_filename']) train_state['stop_early'] = False # Save model if performance improved elif train_state['epoch_index'] >= 1: loss_tm1, loss_t = train_state['val_loss'][-2:] # If loss worsened if loss_t >= train_state['early_stopping_best_val']: # Update step train_state['early_stopping_step'] += 1 # Loss decreased else: # Save the best model if loss_t < train_state['early_stopping_best_val']: torch.save(model.state_dict(), train_state['model_filename']) # Reset early stopping step train_state['early_stopping_step'] = 0 # Stop early ? train_state['stop_early'] = \ train_state['early_stopping_step'] >= args.early_stopping_criteria return train_state def compute_accuracy(y_pred, y_target): _, y_pred_indices = y_pred.max(dim=1) n_correct = torch.eq(y_pred_indices, y_target).sum().item() return n_correct / len(y_pred_indices) * 100 def set_seed_everywhere(seed, cuda): np.random.seed(seed) torch.manual_seed(seed) if cuda: torch.cuda.manual_seed_all(seed) def handle_dirs(dirpath): if not os.path.exists(dirpath): os.makedirs(dirpath)
3、训练模型
args = Namespace( # Data and path information surname_csv="surnames_with_splits.csv", vectorizer_file="vectorizer.json", model_state_file="model.pth", save_dir="model_storage/ch4/surname_mlp", # Model hyper parameters hidden_dim=300, # Training hyper parameters seed=1337, num_epochs=100, early_stopping_criteria=5, learning_rate=0.001, batch_size=64, # Runtime options cuda=False, reload_from_files=False, expand_filepaths_to_save_dir=True, ) if args.expand_filepaths_to_save_dir: args.vectorizer_file = os.path.join(args.save_dir, args.vectorizer_file) args.model_state_file = os.path.join(args.save_dir, args.model_state_file) print("Expanded filepaths: ") print("\t{}".format(args.vectorizer_file)) print("\t{}".format(args.model_state_file)) # Check CUDA if not torch.cuda.is_available(): args.cuda = False args.device = torch.device("cuda" if args.cuda else "cpu") print("Using CUDA: {}".format(args.cuda)) # Set seed for reproducibility set_seed_everywhere(args.seed, args.cuda) # handle dirs handle_dirs(args.save_dir) if args.reload_from_files: # training from a checkpoint print("Reloading!") dataset = SurnameDataset.load_dataset_and_load_vectorizer(args.surname_csv, args.vectorizer_file) else: # create dataset and vectorizer print("Creating fresh!") dataset = SurnameDataset.load_dataset_and_make_vectorizer(args.surname_csv) dataset.save_vectorizer(args.vectorizer_file) vectorizer = dataset.get_vectorizer() classifier = SurnameClassifier(input_dim=len(vectorizer.surname_vocab), hidden_dim=args.hidden_dim, output_dim=len(vectorizer.nationality_vocab)) classifier = classifier.to(args.device) dataset.class_weights = dataset.class_weights.to(args.device) loss_func = nn.CrossEntropyLoss(dataset.class_weights) optimizer = optim.Adam(classifier.parameters(), lr=args.learning_rate) scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer=optimizer, mode='min', factor=0.5, patience=1) train_state = make_train_state(args) epoch_bar = tqdm_notebook(desc='training routine', total=args.num_epochs, position=0) dataset.set_split('train') train_bar = tqdm_notebook(desc='split=train', total=dataset.get_num_batches(args.batch_size), position=1, leave=True) dataset.set_split('val') val_bar = tqdm_notebook(desc='split=val', total=dataset.get_num_batches(args.batch_size), position=1, leave=True) try: for epoch_index in range(args.num_epochs): train_state['epoch_index'] = epoch_index # Iterate over training dataset # setup: batch generator, set loss and acc to 0, set train mode on dataset.set_split('train') batch_generator = generate_batches(dataset, batch_size=args.batch_size, device=args.device) running_loss = 0.0 running_acc = 0.0 classifier.train() for batch_index, batch_dict in enumerate(batch_generator): # the training routine is these 5 steps: # -------------------------------------- # step 1. zero the gradients optimizer.zero_grad() # step 2. compute the output y_pred = classifier(batch_dict['x_surname']) # step 3. compute the loss loss = loss_func(y_pred, batch_dict['y_nationality']) loss_t = loss.item() running_loss += (loss_t - running_loss) / (batch_index + 1) # step 4. use loss to produce gradients loss.backward() # step 5. use optimizer to take gradient step optimizer.step() # ----------------------------------------- # compute the accuracy acc_t = compute_accuracy(y_pred, batch_dict['y_nationality']) running_acc += (acc_t - running_acc) / (batch_index + 1) # update bar train_bar.set_postfix(loss=running_loss, acc=running_acc, epoch=epoch_index) train_bar.update() train_state['train_loss'].append(running_loss) train_state['train_acc'].append(running_acc) # Iterate over val dataset # setup: batch generator, set loss and acc to 0; set eval mode on dataset.set_split('val') batch_generator = generate_batches(dataset, batch_size=args.batch_size, device=args.device) running_loss = 0. running_acc = 0. classifier.eval() for batch_index, batch_dict in enumerate(batch_generator): # compute the output y_pred = classifier(batch_dict['x_surname']) # step 3. compute the loss loss = loss_func(y_pred, batch_dict['y_nationality']) loss_t = loss.to("cpu").item() running_loss += (loss_t - running_loss) / (batch_index + 1) # compute the accuracy acc_t = compute_accuracy(y_pred, batch_dict['y_nationality']) running_acc += (acc_t - running_acc) / (batch_index + 1) val_bar.set_postfix(loss=running_loss, acc=running_acc, epoch=epoch_index) val_bar.update() train_state['val_loss'].append(running_loss) train_state['val_acc'].append(running_acc) train_state = update_train_state(args=args, model=classifier, train_state=train_state) scheduler.step(train_state['val_loss'][-1]) if train_state['stop_early']: break train_bar.n = 0 val_bar.n = 0 epoch_bar.update() except KeyboardInterrupt: print("Exiting loop")
4、预测结果
# compute the loss & accuracy on the test set using the best available model
classifier.load_state_dict(torch.load(train_state['model_filename']))
classifier = classifier.to(args.device)
dataset.class_weights = dataset.class_weights.to(args.device)
loss_func = nn.CrossEntropyLoss(dataset.class_weights)
dataset.set_split('test')
batch_generator = generate_batches(dataset,
batch_size=args.batch_size,
device=args.device)
running_loss = 0.
running_acc = 0.
classifier.eval()
for batch_index, batch_dict in enumerate(batch_generator):
# compute the output
y_pred = classifier(batch_dict['x_surname'])
# compute the loss
loss = loss_func(y_pred, batch_dict['y_nationality'])
loss_t = loss.item()
running_loss += (loss_t - running_loss) / (batch_index + 1)
# compute the accuracy
acc_t = compute_accuracy(y_pred, batch_dict['y_nationality'])
running_acc += (acc_t - running_acc) / (batch_index + 1)
train_state['test_loss'] = running_loss
train_state['test_acc'] = running_acc
print("Test loss: {};".format(train_state['test_loss']))
print("Test Accuracy: {}".format(train_state['test_acc']))
def predict_nationality(surname, classifier, vectorizer): """Predict the nationality from a new surname Args: surname (str): the surname to classifier classifier (SurnameClassifer): an instance of the classifier vectorizer (SurnameVectorizer): the corresponding vectorizer Returns: a dictionary with the most likely nationality and its probability """ vectorized_surname = vectorizer.vectorize(surname) vectorized_surname = torch.tensor(vectorized_surname).view(1, -1) result = classifier(vectorized_surname, apply_softmax=True) probability_values, indices = result.max(dim=1) index = indices.item() predicted_nationality = vectorizer.nationality_vocab.lookup_index(index) probability_value = probability_values.item() return {'nationality': predicted_nationality, 'probability': probability_value} new_surname = input("Enter a surname to classify: ") classifier = classifier.to("cpu") prediction = predict_nationality(new_surname, classifier, vectorizer) print("{} -> {} (p={:0.2f})".format(new_surname, prediction['nationality'], prediction['probability']))
vectorizer.nationality_vocab.lookup_index(8) def predict_topk_nationality(name, classifier, vectorizer, k=5): vectorized_name = vectorizer.vectorize(name) vectorized_name = torch.tensor(vectorized_name).view(1, -1) prediction_vector = classifier(vectorized_name, apply_softmax=True) probability_values, indices = torch.topk(prediction_vector, k=k) # returned size is 1,k probability_values = probability_values.detach().numpy()[0] indices = indices.detach().numpy()[0] results = [] for prob_value, index in zip(probability_values, indices): nationality = vectorizer.nationality_vocab.lookup_index(index) results.append({'nationality': nationality, 'probability': prob_value}) return results new_surname = input("Enter a surname to classify: ") classifier = classifier.to("cpu") k = int(input("How many of the top predictions to see? ")) if k > len(vectorizer.nationality_vocab): print("Sorry! That's more than the # of nationalities we have.. defaulting you to max size :)") k = len(vectorizer.nationality_vocab) predictions = predict_topk_nationality(new_surname, classifier, vectorizer, k=k) print("Top {} predictions:".format(k)) print("===================") for prediction in predictions: print("{} -> {} (p={:0.2f})".format(new_surname, prediction['nationality'], prediction['probability']))