1、介绍
预训练任务
1、掩盖语言模型,自编码模型
将一些位置的token替换成[MASK],预测这些被替换的字符
2、因果语言模型,自回归模型
将完整的序列输入,基于上文的token预测当前的token
3、序列到序列模型
任务多样化,只是采用编码器和解码器的方式,预测器部分放在解码器中
2、掩码语言模型预训练代码实战
2.1、导包
from datasets import load_dataset,Dataset
from transformers import (AutoTokenizer,
AutoModelForMaskedLM,
DataCollatorForLanguageModeling,
TrainingArguments,
Trainer)
2.2、加载数据
ds = Dataset.load_from_disk('./wiki_cn_filtered/')
2.4、数据预处理
tokenizer = AutoTokenizer.from_pretrained("../Model/chinese-macbert-base")
def process_func(examples):
return tokenizer(
examples['completion'],
return_tensors="pt",
padding=True,
max_length=128,
truncation=True)
tokenizered_ds = ds.map(function=process_func,
batched=True,
remove_columns=ds.column_names)
tokenizered_ds
from torch.utils.data import DataLoader
dl = DataLoader(tokenizered_ds,
batch_size=2,
collate_fn=DataCollatorForLanguageModeling(tokenizer,mlm=True,mlm_probability=0.15))
2.5、创建模型
model = AutoModelForMaskedLM.from_pretrained('../Model/chinese-macbert-base')
2.6、配置训练参数
args = TrainingArguments(
output_dir='./results', # output directory
per_device_train_batch_size=4,
logging_steps=1,
num_train_epochs=1
)
2.7、创建训练器
trainer = Trainer(
args=args,
model=model,
train_dataset=tokenizered_ds,
data_collator=DataCollatorForLanguageModeling(tokenizer,mlm=True,mlm_probability=0.15)
)
2.8、训练
trainer.train()
2.9、模型推理
from transformers import pipeline
pipe = pipeline('fill-mask',model = model,tokenizer = tokenizer,device = 0)
pipe("李妮娜,籍贯辽宁本[MASK],中国女子自由式滑雪运动员。")