深度学习入门 | 李宏毅作业 HW4（NLP）

任务描述：Speaker Identification 这是一个语音识别任务，要识别出语音的发出者。

我们先来研究下数据集，有3个json和很多个语音pt文件。

一、构造Data集

PyTorch数据读入是通过Dataset+DataLoader的方式完成的，Dataset定义好数据的格式和数据变换形式，

DataLoader用iterative的方式不断读入批次数据。

自定义Dataset类

我们可以定义自己的Dataset类来实现灵活的数据读取，定义的类需要继承PyTorch自身的Dataset类。主要包含三个函数：

__init__: 根据mapping.json和metadata.json返回self.data，存储的是语音地址和对应训练集label

__getitem__: 根据语音地址加载pt数据,并且规范长度(太长就随机截断),返回mel, speaker,代表语音数据和对应label

__len__: 用于返回数据集的样本数,1人有多段语音

import os
import json
import torch
import random
from pathlib import Path
from torch.utils.data import Dataset
from torch.nn.utils.rnn import pad_sequence


class myDataset(Dataset):
  def __init__(self, data_dir, segment_len=128):
    self.data_dir = data_dir
    self.segment_len = segment_len

    # Load the mapping from speaker neme to their corresponding id. 
    mapping_path = Path(data_dir) / "mapping.json"
    mapping = json.load(mapping_path.open())
    self.speaker2id = mapping["speaker2id"]

    # Load metadata of training data.
    metadata_path = Path(data_dir) / "metadata.json"
    metadata = json.load(open(metadata_path))["speakers"]

    # Get the total number of speaker.
    self.speaker_num = len(metadata.keys())
    self.data = []
    for speaker in metadata.keys():
      for utterances in metadata[speaker]:
        self.data.append([utterances["feature_path"], self.speaker2id[speaker]])

  def __len__(self):
    return len(self.data)

  def __getitem__(self, index):
    feat_path, speaker = self.data[index]
    # Load preprocessed mel-spectrogram.
    mel = torch.load(os.path.join(self.data_dir, feat_path))

    # Segmemt mel-spectrogram into "segment_len" frames.
    if len(mel) > self.segment_len:
      # Randomly get the starting point of the segment.
      start = random.randint(0, len(mel) - self.segment_len)
      # Get a segment with "segment_len" frames.
      mel = torch.FloatTensor(mel[start:start+self.segment_len])
    else:
      mel = torch.FloatTensor(mel)
    # Turn the speaker id into long for computing loss later.
    speaker = torch.FloatTensor([speaker]).long()
    return mel, speaker

  def get_speaker_number(self):
    return self.speaker_num

构造Dataloader

参考：https://wenku.baidu.com/view/a8f0c6195427a5e9856a561252d380eb629423b2.html

这里先补充下Dataloader的基础知识

所以我们需要定义其中的collate_fn等于如下的collate_batch函数，目的是为了统一特征大小。

将dataset 分为 训练集(90%) 和验证集(10%)，返回 train_loader, valid_loader, speaker_num

import torch
from torch.utils.data import DataLoader, random_split
from torch.nn.utils.rnn import pad_sequence


def collate_batch(batch):
  # Process features within a batch.
  """Collate a batch of data."""
  mel, speaker = zip(*batch)
  # Because we train the model batch by batch, we need to pad the features in the same batch to make their lengths the same.
  mel = pad_sequence(mel, batch_first=True, padding_value=-20)    # pad log 10^(-20) which is very small value.
  # mel: (batch size, length, 40)
  return mel, torch.FloatTensor(speaker).long()


def get_dataloader(data_dir, batch_size, n_workers):
  """Generate dataloader"""
  dataset = myDataset(data_dir)
  speaker_num = dataset.get_speaker_number()
  # Split dataset into training dataset and validation dataset
  trainlen = int(0.9 * len(dataset))
  lengths = [trainlen, len(dataset) - trainlen]
  trainset, validset = random_split(dataset, lengths)

  train_loader = DataLoader(
    trainset,
    batch_size=batch_size,
    shuffle=True,
    drop_last=True,
    num_workers=n_workers,
    pin_memory=True,
    collate_fn=collate_batch,
  )
  valid_loader = DataLoader(
    validset,
    batch_size=batch_size,
    num_workers=n_workers,
    drop_last=True,
    pin_memory=True,
    collate_fn=collate_batch,
  )

  return train_loader, valid_loader, speaker_num

报错：broken pipe

需要额外注意的是，构造Dataloader时，必须这样设置num_workers=0，因为我们没有GPU进行多线程，如果设置>0的数字则会报错broken pipe

下面，我们来查看下构建的数据集维度

data_dir= "./Dataset",
batch_size= 32 
n_workers= 0  #没有多进程的话要用0

train_loader, valid_loader, speaker_num = get_dataloader(data_dir, batch_size, n_workers)
mels, labels = next(iter(train_loader))
print(speaker_num)
print(mels.shape)
print(labels.shape)

我们可以看到，返回的是一共有600个人说了话，我们每次取32bacth，长度为128，有40个feature,即(batch size, length, d_model)

二、构建模型

2.1 Base模型

模型结构是：
全连接(y=wx+b)->TransformerEncoderLayer->pred_layer(全连接->ReLU->全连接)

我们这里用到了Transformer层，这是self attention的重点，但我们可以用一行代码就可以搞定。

留下作业：进阶难度，自己架构Transformer。

Transformer的具体使用可以看这篇参考文献：

迟到的transformer encoder代码详解 https://blog.csdn.net/jokerxsy/article/details/108757220

import torch
import torch.nn as nn
import torch.nn.functional as F


class Classifier(nn.Module):
  # 全连接(y=wx+b)->TransformerEncoderLayer->pred_layer(全连接->ReLU->全连接)
  def __init__(self, d_model=80, n_spks=600, dropout=0.1):
    super().__init__()
    
    self.prenet = nn.Linear(40, d_model)        #全连接,输入是(batch size, length, 40)
    self.encoder_layer = nn.TransformerEncoderLayer(
      d_model=d_model, dim_feedforward=256, nhead=2
    )
    # self.encoder = nn.TransformerEncoder(self.encoder_layer, num_layers=2)

    self.pred_layer = nn.Sequential(
      nn.Linear(d_model, d_model),
      nn.ReLU(),
      nn.Linear(d_model, n_spks),
    )

  def forward(self, mels):
    """
    输入:
      mels: (batch size, length, 40)
    输出:
      out: (batch size, n_spks)
    """
    # out: (batch size, length, d_model)
    out = self.prenet(mels)
    # out: (length, batch size, d_model)
    out = out.permute(1, 0, 2)
    # The encoder layer expect features in the shape of (length, batch size, d_model).
    out = self.encoder_layer(out)
    # out: (batch size, length, d_model)
    out = out.transpose(0, 1)
    # mean pooling
    stats = out.mean(dim=1)

    # out: (batch, n_spks)
    out = self.pred_layer(stats)
    return out

注意：这里使用全连接的时候即nn.Linear ，一般来说是输入二维矩阵，但是也可以输入三维，比如[32, 128, 192]形状的张量，通过nn.Linear(192, 256)的线性层，其输出的形状是[32, 128, 256]。具体见链接

https://blog.csdn.net/qq_36258516/article/details/117333424

2.2 验证模型

2.2.1 model_fn：输入一组bacth数据进入模型，输出损失和准确率

model_fn：输入一组bacth数据进入模型，输出损失和准确率

bacth：为了计算方便，我们将所有数据划分成了一组一组的batch，每组bacth有数据和label，数据维度大小是(batch size, length, 40)，更具体的是（32,128,40），lable维度是（batch ,1）。batch = next(iter(train_loader))

model：就是上面构造的模型，model = Classifier(n_spks=speaker_num).to(device)，输出是预测的label

criterion：损失函数这里用交叉损失，criterion = nn.CrossEntropyLoss()

device：是否用GPU，device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)

import torch

def model_fn(batch, model, criterion, device):
  """Forward a batch through the model."""

  mels, labels = batch
  mels = mels.to(device)
  labels = labels.to(device)

  outs = model(mels)

  loss = criterion(outs, labels)

  # Get the speaker id with highest probability.
  preds = outs.argmax(1)
  # Compute accuracy.
  accuracy = torch.mean((preds == labels).float())

  return loss, accuracy

2.2.2 Validate：计算验证集上的损失

这里tqdm是可视化进度条

valid：计算验证集上的损失率

from tqdm import tqdm
import torch


def valid(dataloader, model, criterion, device): 
  """Validate on validation set."""

  model.eval()
  running_loss = 0.0
  running_accuracy = 0.0
  pbar = tqdm(total=len(dataloader.dataset), ncols=0, desc="Valid", unit=" uttr")

  for i, batch in enumerate(dataloader):
    with torch.no_grad():
      loss, accuracy = model_fn(batch, model, criterion, device)
      running_loss += loss.item()
      running_accuracy += accuracy.item()

    pbar.update(dataloader.batch_size)
    pbar.set_postfix(
      loss=f"{running_loss / (i+1):.2f}",
      accuracy=f"{running_accuracy / (i+1):.2f}",
    )

  pbar.close()
  model.train()

  return running_accuracy / len(dataloader)

2.3 更新模型，存储模型最优参数

主要包含以下部分

1. 前期准备：包括加载训练集和验证集数据，加载model、criterion 、optimizer 、device

2. 开始epoch循环，每次循环下再一组组取bacth，在训练集上，计算梯度反向传播；在验证集上，计算验证集准确率

3. 存储最优的模型

from tqdm import tqdm

import torch
import torch.nn as nn
from torch.optim import AdamW
from torch.utils.data import DataLoader, random_split


def parse_args():
  """arguments"""
  config = {
    "data_dir": "./Dataset",
    "save_path": "model.ckpt",
    "batch_size": 32,
    "n_workers": 0, #没有多进程的话要用0
    #"valid_steps": 2000,
    #"warmup_steps": 1000,
    #"save_steps": 10000,
    #"total_steps": 70000,
    "valid_steps": 2,
    "warmup_steps": 1000,#lr调整的耐心系数
    "save_steps": 5,
    "total_steps": 100,
      
  }

  return config


def main(
  data_dir,
  save_path,
  batch_size,
  n_workers,
  valid_steps,
  warmup_steps,
  total_steps,
  save_steps,
):
  """Main function."""
  # 1.前期准备：包括加载训练集和验证集数据，加载model、criterion 、optimizer 、device 
  device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
  print(f"[Info]: Use {device} now!")
 
  train_loader, valid_loader, speaker_num = get_dataloader(data_dir, batch_size, n_workers)
  train_iterator = iter(train_loader)
  print(f"[Info]: Finish loading data!",flush = True)

  model = Classifier(n_spks=speaker_num).to(device)
  criterion = nn.CrossEntropyLoss()
  optimizer = AdamW(model.parameters(), lr=1e-3)
  scheduler = get_cosine_schedule_with_warmup(optimizer, warmup_steps, total_steps)
  print(f"[Info]: Finish creating model!",flush = True)

  best_accuracy = -1.0
  best_state_dict = None

  pbar = tqdm(total=valid_steps, ncols=0, desc="Train", unit=" step")
  
  # 2. 开始总循环
  for step in range(total_steps):
    # 一组组取batch
    try:
      batch = next(train_iterator)
    except StopIteration:
      train_iterator = iter(train_loader)
      batch = next(train_iterator)

    loss, accuracy = model_fn(batch, model, criterion, device)
    batch_loss = loss.item()
    batch_accuracy = accuracy.item()

    # 计算梯度,更新模型
    loss.backward()
    optimizer.step()
    scheduler.step()
    optimizer.zero_grad()
    
    # Log
    pbar.update()
    pbar.set_postfix(
      loss=f"{batch_loss:.2f}",
      accuracy=f"{batch_accuracy:.2f}",
      step=step + 1,
    )

    # Do validation
    if (step + 1) % valid_steps == 0:
      pbar.close()

      valid_accuracy = valid(valid_loader, model, criterion, device)

      # keep the best model
      if valid_accuracy > best_accuracy:
        best_accuracy = valid_accuracy
        best_state_dict = model.state_dict()
        

      pbar = tqdm(total=valid_steps, ncols=0, desc="Train", unit=" step")
    
    # Save the best model so far.
    if (step + 1) % save_steps == 0 and best_state_dict is not None:
      torch.save(best_state_dict, save_path)
      pbar.write(f"Step {step + 1}, best model saved. (accuracy={best_accuracy:.4f})")

  pbar.close()


if __name__ == "__main__":
  main(**parse_args())

三、预测数据

3.1 定义预测集Data

这里不同的是，没有标签label了，只有预测数据，我们需要根据数据预测出label

import os
import json
import torch
from pathlib import Path
from torch.utils.data import Dataset


class InferenceDataset(Dataset):
  def __init__(self, data_dir):
    testdata_path = Path(data_dir) / "testdata.json"
    metadata = json.load(testdata_path.open())
    self.data_dir = data_dir
    self.data = metadata["utterances"]

  def __len__(self):
    return len(self.data)

  def __getitem__(self, index):
    utterance = self.data[index]
    feat_path = utterance["feature_path"]
    mel = torch.load(os.path.join(self.data_dir, feat_path))

    return feat_path, mel


def inference_collate_batch(batch):
  """Collate a batch of data."""
  feat_paths, mels = zip(*batch)

  return feat_paths, torch.stack(mels)

3.2 加载最优模型，预测label

import json
import csv
from pathlib import Path
#from tqdm.notebook import tqdm
from tqdm import tqdm

import torch
from torch.utils.data import DataLoader

def parse_args():
  """arguments"""
  config = {
    "data_dir": "./Dataset",
    "model_path": "./model.ckpt",
    "output_path": "./output.csv",
  }

  return config


def main(
  data_dir,
  model_path,
  output_path,
):
  """Main function."""
  device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
  print(f"[Info]: Use {device} now!")

  mapping_path = Path(data_dir) / "mapping.json"
  mapping = json.load(mapping_path.open())

  dataset = InferenceDataset(data_dir)
  dataloader = DataLoader(
    dataset,
    batch_size=1,
    shuffle=False,
    drop_last=False,
    num_workers=0,
    collate_fn=inference_collate_batch,
  )
  print(f"[Info]: Finish loading data!",flush = True)

  speaker_num = len(mapping["id2speaker"])
  model = Classifier(n_spks=speaker_num).to(device)
  model.load_state_dict(torch.load(model_path))
  model.eval()
  print(f"[Info]: Finish creating model!",flush = True)

  results = [["Id", "Category"]]
  for feat_paths, mels in tqdm(dataloader):
    with torch.no_grad():
      mels = mels.to(device)
      outs = model(mels)
      preds = outs.argmax(1).cpu().numpy()
      for feat_path, pred in zip(feat_paths, preds):
        results.append([feat_path, mapping["id2speaker"][str(pred)]])
  
  with open(output_path, 'w', newline='') as csvfile:
    writer = csv.writer(csvfile)
    writer.writerows(results)


if __name__ == "__main__":
  main(**parse_args())