基于华为MindSpore的预训练语言模型（BERT & GPT）应用实战笔记

引言

预训练语言模型（PLM）是自然语言处理（NLP）领域的核心技术，如BERT、GPT等。华为MindSpore提供了一系列工具，支持高效的BERT和GPT训练与推理。

1. MindSpore进行PLM训练的优势

• 高效计算：MindSpore的图编译优化显著提升训练效率。
• 自动并行：原生支持数据并行与模型并行。
• Ascend NPU优化：针对华为Ascend AI处理器深度优化，提升硬件利用率。

2. MindSpore 预训练语言模型开发流程

2.1 安装MindSpore及NLP模块

pip install mindspore -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install mindnlp

2.2 导入必要库

import mindspore as ms
import mindspore.nn as nn
import mindspore.dataset as ds
from mindspore import Tensor, Model
from mindnlp.transformers import BertTokenizer, BertForSequenceClassification, GPT2Tokenizer, GPT2LMHeadModel

2.3 文本预处理（BERT & GPT）

# BERT 预处理
tokenizer_bert = BertTokenizer.from_pretrained("bert-base-uncased")
text = "MindSpore is an open source deep learning framework."
inputs_bert = tokenizer_bert(text, return_tensors="ms")
print(inputs_bert)

# GPT 预处理
tokenizer_gpt = GPT2Tokenizer.from_pretrained("gpt2")
inputs_gpt = tokenizer_gpt(text, return_tensors="ms")
print(inputs_gpt)

3. 加载预训练BERT和GPT模型并微调

3.1 加载BERT和GPT模型

model_bert = BertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)
model_gpt = GPT2LMHeadModel.from_pretrained("gpt2")

3.2 定义训练参数

loss_fn = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction='mean')
optimizer_bert = nn.Adam(model_bert.trainable_params(), learning_rate=2e-5)
optimizer_gpt = nn.Adam(model_gpt.trainable_params(), learning_rate=2e-5)

3.3 加载数据集并训练（BERT & GPT）

def process_data_bert(example):
    inputs = tokenizer_bert(example["text"], return_tensors="ms", padding='max_length', max_length=128)
    return inputs["input_ids"], Tensor(example["label"], ms.int32)

def process_data_gpt(example):
    inputs = tokenizer_gpt(example["text"], return_tensors="ms", padding='max_length', max_length=128)
    return inputs["input_ids"]

# 加载数据集
dataset_bert = ds.load_dataset("imdb", split="train")
dataset_bert = dataset_bert.map(process_data_bert, input_columns=["text", "label"]).batch(32)

dataset_gpt = ds.load_dataset("wikitext", split="train")
dataset_gpt = dataset_gpt.map(process_data_gpt, input_columns=["text"]).batch(32)

# 训练 BERT
trainer_bert = Model(model_bert, loss_fn=loss_fn, optimizer=optimizer_bert)
trainer_bert.train(epoch=3, train_dataset=dataset_bert)

# 训练 GPT（语言建模）
trainer_gpt = Model(model_gpt, loss_fn=None, optimizer=optimizer_gpt)
trainer_gpt.train(epoch=3, train_dataset=dataset_gpt)

3.4 评估BERT和GPT模型

def evaluate_bert(model, dataset):
    total, correct = 0, 0
    for input_ids, label in dataset.create_tuple_iterator():
        output = model(input_ids)
        pred = output.argmax(axis=1)
        correct += (pred == label).sum().asnumpy()
        total += label.shape[0]
    return correct / total

accuracy_bert = evaluate_bert(model_bert, dataset_bert)
print(f"BERT Model accuracy: {accuracy_bert:.2%}")

# GPT 文本生成示例
input_text = "MindSpore is an innovative AI framework"
input_ids = tokenizer_gpt(input_text, return_tensors="ms")["input_ids"]
output_ids = model_gpt.generate(input_ids, max_length=50)
generated_text = tokenizer_gpt.decode(output_ids[0], skip_special_tokens=True)
print("Generated Text:", generated_text)