自然语言处理NLP：使用DisBert模型完成文本二分类任务(Pytorch)

项目来源于kaggle竞赛，地址为：Natural Language Processing with Disaster Tweets | Kaggle

        本文主要是对本人学习NLP的过程做个总结和记录，以方便日后复习，当然如果本文能帮助到阅读该文的读者，我会感到很开心。

        该项目通过建立一个机器学习模型，预测哪些推文是关于真实灾难的，哪些不是。

        首先读取数据，看一下数据的样子，数据在kaggle上可直接下载。可以看到有用的是text列和target列，我们只需要这两列进行模型训练即可，target为1表示text是真实灾难相关的文本，反之则不是。

import pandas as pd

train_df = pd.read_csv('/kaggle/input/nlpdataset/train.csv')
test_df = pd.read_csv('/kaggle/input/nlpdataset/test.csv')
train_df

         然后定义缩写字符变扩写字符的字典，以便将文本中的缩写字符扩展为全称，在NLP预处理数据过程中，将英文缩写扩展为全称的作用是为了将缩写与其全称等效地表示，以便于算法处理。例如，将"I'm" 扩展为"I am"，将"don't" 扩展为"do not"，将"it's" 扩展为"it is" 或 "it has"。

        缩写扩展的作用主要是为了消除歧义：同一个缩写可能有多种可能的解释，例如"AI"可以表示"Artificial Intelligence"或者"Air India"。将缩写扩展为全称可以消除这种歧义，确保模型可以正确地理解数据。

        日后如果有文本缩写扩展处理的需要，可直接复制该字典，然后进行替换操作。

# 缩写变扩写
contractions = {
    "ain't": "am not",
    "aren't": "are not",
    "can't": "cannot",
    "can't've": "cannot have",
    "'cause": "because",
    "could've": "could have",
    "couldn't": "could not",
    "couldn't've": "could not have",
    "didn't": "did not",
    "doesn't": "does not",
    "don't": "do not",
    "hadn't": "had not",
    "hadn't've": "had not have",
    "hasn't": "has not",
    "haven't": "have not",
    "he'd": "he would",
    "he'd've": "he would have",
    "he'll": "he will",
    "he'll've": "he will have",
    "he's": "he is",
    "how'd": "how did",
    "how'd'y": "how do you",
    "how'll": "how will",
    "how's": "how is",
    "i'd": "I would",
    "i'd've": "I would have",
    "i'll": "I will",
    "i'll've": "I will have",
    "i'm": "I am",
    "i've": "I have",
    "isn't": "is not",
    "it'd": "it would",
    "it'd've": "it would have",
    "it'll": "it will",
    "it'll've": "it will have",
    "it's": "it is",
    "let's": "let us",
    "ma'am": "madam",
    "mayn't": "may not",
    "might've": "might have",
    "mightn't": "might not",
    "mightn't've": "might not have",
    "must've": "must have",
    "mustn't": "must not",
    "mustn't've": "must not have",
    "needn't": "need not",
    "needn't've": "need not have",
    "o'clock": "of the clock",
    "oughtn't": "ought not",
    "oughtn't've": "ought not have",
    "shan't": "shall not",
    "sha'n't": "shall not",
    "shan't've": "shall not have",
    "she'd": "she would",
    "she'd've": "she would have",
    "she'll": "she will",
    "she'll've": "she will have",
    "she's": "she is",
    "should've": "should have",
    "shouldn't": "should not",
    "shouldn't've": "should not have",
    "so've": "so have",
    "so's": "so is",
    "that'd": "that would",
    "that'd've": "that would have",
    "that's": "that is",
    "there'd": "there would",
    "there's": "there is",
    "they'd": "they would",
    "they'd've": "they would have",
    "they'll": "they will",
    "they'll've": "they will have",
    "they're": "they are",
    "they've": "they have",
    "to've": "to have",
    "wasn't": "was not",
    "we'd": "we would",
    "we'd've": "we would have",
    "we'll": "we will",
    "we'll've": "we will have",
    "we're": "we are",
    "we've": "we have",
    "weren't": "were not",
    "what'll": "what will",
    "what'll've": "what will have",
    "what're": "what are",
    "what's": "what is",
    "what've": "what have",
    "when's": "when is",
    "when've": "when have",
    "where'd": "where did",
    "where's": "where is",
    "where've": "where have",
    "who'll": "who will",
    "who'll've": "who will have",
    "who's": "who is",
    "who've": "who have",
    "why's": "why is",
    "why've": "why have",
    "will've": "will have",
    "won't": "will not",
    "won't've": "will not have",
    "would've": "would have",
    "wouldn't": "would not",
    "wouldn't've": "would not have",
    "y'all": "you all",
    "y'all'd": "you all would",
    "y'all'd've": "you all would have",
    "y'all're": "you all are",
    "y'all've": "you all have",
    "you'd": "you would",
    "you'd've": "you would have",
    "you'll": "you will",
    "you'll've": "you will have",
    "you're": "you are",
    "you've": "you have"
}

country_contractions = {
    'u.s': 'united states',
    'u.s.': 'united states',
    'u.s.a': 'united states',
    'u.k': 'united kingdom',
    'u.k.': 'united kingdom',
    'u.a.e': 'united arab emirates',
    'u.a.e.': 'united arab emirates',
    's.korea': 'south korea',
    'n.korea': 'north korea',
    'czech rep.': 'czech republic',
    'dominican rep.': 'dominican republic',
    'costa rica': 'republic of costa rica',
    'el salvador': 'republic of el salvador',
    'guinea-bissau': 'republic of guinea-bissau',
    'cote d\'ivoire': 'republic of cote d\'ivoire',
    'trinidad & tobago': 'republic of trinidad and tobago',
    'congo-brazzaville': 'republic of the congo',
    'congo-kinshasa': 'democratic republic of the congo',
    'sri lanka': 'democratic socialist republic of sri lanka',
    'central african rep.': 'central african republic',
    'san marino': 'republic of san marino',
    'são tomé & príncipe': 'democratic republic of são tomé and príncipe',
    'timor-leste': 'democratic republic of timor-leste'
}

        定义缩写扩展的函数，需要注意的地方在注释中已经写出。

# 缩写变扩写
def expand_contractions(text,contractions):
    words = text.split()
    expand_contractions = []
    for word in words:
        if word.lower() in contractions:#判断word是否在contractions字典的键中出现
            expand_contraction = contractions[word.lower()]
            expand_contractions.append(expand_contraction)
        else:
            expand_contractions.append(word)
    return ' '.join(expand_contractions)# join函数用于将列表中的字符连接为一个字符串(以空格分割)        

        进行文本清洗，包括缩写字符扩展，以及将URL、数字和标点替换为空格。URL、标点、数字等在自然语言文本中属于噪声，将它们替换为空格可以去除这些噪声，以便模型可以更好的学习到有用的特征。

import re

# 应用expand_contractions并用sub函数进行url、空格、标点、数字字符替换为空格
def clean_text(text):
    # 替换一般缩略字符和城市缩略字符
    text = expand_contractions(text, contractions)
    text = expand_contractions(text, country_contractions)
    
    text = re.sub(r'http\S+|https\S+|www\S+', '', text, flags = re.MULTILINE)
    # \S代表匹配任意非空字符
    # 其中 \S 表示匹配非空白字符，+ 表示匹配一个或多个。
    # \S+ 表示匹配一个或多个非空白字符。| 表示或的关系。
    # \S+ 和 | 组合在一起，表示匹配以 http、www 或 https 开头的字符串。
    # flags=re.MULTILINE这个参数是多余的，可以不加
    
    text = re.sub(r'\W', ' ', text)# 非字母数字字符 空格，标点符号等
    text = re.sub(r'\d', ' ', text)# 数字字符
    text = re.sub(r'\s+', ' ', text).strip()# 匹配一个或多个空格字符，替换为空格,strip函数删除开头和结尾的空格
    
    return text
    
train_df['clean_text'] = train_df['text'].apply(clean_text)
test_df['clean_text'] = test_df['text'].apply(clean_text)
train_df

         分割训练集和验证集，注意原始数据集中本就存在测试集。

import math

split_count = math.floor(len(train_df) / 10 * 9)
val_df = train_df[split_count:]
train_df = train_df[:split_count]

        之后通过Transfoemers库定义DistilBert模型的tokenizer和model，这里使用的是DistilBert模型，它是BERT模型的一种轻量级版本，通过压缩BERT模型的大小和计算量以及进行知识蒸馏，使得DistilBERT在保持高性能的同时，具有更小的模型体积和更快的推理速度。

1. 知识蒸馏：是一种模型压缩的技术，用于将一个较大和复杂的模型的知识传递给一个较小和 单的模型中。具体来说，知识蒸馏将一个“教师”模型（通常为较大、较复杂的模型）的知识“蒸馏”到一个“学生”模型（通常为较小、较简单的模型）中，使得学生模型能够学习到教师模型中的关键特征和知识，从而获得更好的泛化能力和性能。

        Transformers是由Hugging Face公司开发的一个自然语言处理(NLP)库，它提供了一系列基于深度学习的预训练模型，包括BERT、GPT、RoBERTa、DistilBERT等，方便用户进行模型微调以应用到不同的任务中，Transformers库的应用场景广泛，可以用于各种NLP任务，如情感分析、机器翻译、文本分类、问答系统等。

1. 预训练模型：在NLP中，预训练模型通常使用大量文本数据进行无监督式预训练，以学习语言的特征和结构，例如语法、语义和上下文等。
2. 微调：微调是指在已经预训练的模型基础上，进一步训练模型以适应特定任务的过程。微调通常使用有监督学习方法，使用任务特定的数据集进行训练，以调整预训练模型的权重和参数，使其适应特定的NLP任务。

        如果想更进一步了解和学习Transformers库，这里给出pytorch官网关于Transformers库的链接：PyTorch-Transformers | PyTorch

from transformers import DistilBertTokenizer, DistilBertForSequenceClassification
import torch

MODEL_NAME = 'distilbert-base-uncased'
tokenizer = DistilBertTokenizer.from_pretrained(MODEL_NAME)
model = DistilBertForSequenceClassification.from_pretrained(MODEL_NAME, num_labels=2)

        之后将数据集进行格式转换，最终转换为模型输入所需要的torch格式。tokenizer函数的作用是将输入的文本数据进行分词并进行必要的处理，padding和truncation参数用于控制padding和截断的行为。padding参数用于将所有输入的文本数据padding到相同的长度，以便进行批量处理。如果一个样本的长度小于指定的length ，则用pad_token_id进行填充，如果长度大于length ，则根据truncation参数的设置进行截断。

from transformers import TrainingArguments, Trainer

def tokenize(batch):
    return tokenizer(batch['text'], padding=True, truncation=True)

test_df["target"] = 0

#取出文本和标签
train_dataset = train_df[['clean_text', 'target']]
val_dataset = val_df[['clean_text', 'target']]
test_dataset = test_df[['clean_text', 'target']]

#更改列名
train_dataset = train_dataset.rename(columns={'clean_text': 'text', 'target': 'label'})
val_dataset = val_dataset.rename(columns={'clean_text': 'text', 'target': 'label'})
test_dataset = test_dataset.rename(columns={'clean_text': 'text', 'target': 'label'})

#records参数代表将每一行转换为一个字典，并存储在一个列表中
train_dataset = train_dataset.to_dict('records')
val_dataset = val_dataset.to_dict('records')
test_dataset = test_dataset.to_dict('records')

from datasets import Dataset

#其实可以直接将train_dataset转换为Dataset，不需要进行字典转换，多此一举了
train_dataset = Dataset.from_pandas(pd.DataFrame(train_dataset))
val_dataset = Dataset.from_pandas(pd.DataFrame(val_dataset))
test_dataset = Dataset.from_pandas(pd.DataFrame(test_dataset))

# 将batch的大小设置为整个数据集的大小，意味着将整个数据集进行tokenize操作
train_dataset = train_dataset.map(tokenize, batched=True, batch_size=len(train_dataset))
val_dataset = val_dataset.map(tokenize, batched=True, batch_size=len(val_dataset))
test_dataset = test_dataset.map(tokenize, batched=True, batch_size=len(test_dataset))

# 将数据集转换为torch格式
train_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'label'])
val_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'label'])
test_dataset.set_format('torch', columns=['input_ids', 'attention_mask', 'label'])

         设置训练参数和训练器，训练并保存模型。

import os 

os.environ["WANDB_DISABLED"] = "true"

# 设置训练参数
training_args = TrainingArguments(
    output_dir='./results',
    num_train_epochs=10,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir='./logs',
    logging_steps=10,
    load_best_model_at_end=True,
    evaluation_strategy="epoch",
    save_strategy="epoch"
)

# 初始化训练器
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    tokenizer=tokenizer,
)

# 训练模型
trainer.train()

# 保存模型参数
model.save_pretrained("./model")

         加载模型并在测试集上进行预测。

​# 加载模型
loaded_model = DistilBertForSequenceClassification.from_pretrained("./model")
​
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

def predict(text, loaded_model, tokenizer):
    inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)
    inputs.to(device)
    outputs = loaded_model(**inputs)
    logits = outputs.logits
    probabilities = torch.softmax(logits, dim=-1)
    predictions = torch.argmax(probabilities, dim=-1)
    return predictions.item()

test_df["predicted_target"] = test_df["clean_text"].apply(lambda x: predict(x, model, tokenizer))