1、介绍
预训练任务
1、掩盖语言模型,自编码模型
将一些位置的token替换成[MASK],预测这些被替换的字符
2、因果语言模型,自回归模型
将完整的序列输入,基于上文的token预测当前的token
3、序列到序列模型
任务多样化,只是采用编码器和解码器的方式,预测器部分放在解码器中
2、代码实战因果语言模型预训练
2.1、导包
from datasets import load_dataset, Dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, DataCollatorForLanguageModeling, TrainingArguments, Trainer, BloomForCausalLM
2.2、加载数据集
ds = Dataset.load_from_disk("./wiki_cn_filtered/")
2.3、数据预处理
tokenizer = AutoTokenizer.from_pretrained("../Model/bloom-389m-zh")
def process_func(examples):
contents = [e + tokenizer.eos_token for e in examples["completion"]]
return tokenizer(contents, max_length=384, truncation=True)
tokenized_ds = ds.map(process_func, batched=True, remove_columns=ds.column_names)
tokenized_ds
from torch.utils.data import DataLoader
dl = DataLoader(tokenized_ds, batch_size=2, collate_fn=DataCollatorForLanguageModeling(tokenizer, mlm=False))
2.4、创建模型
model = AutoModelForCausalLM.from_pretrained("../Model/bloom-389m-zh")
2.5、配置训练参数
args = TrainingArguments(
output_dir="./causal_lm",
per_device_train_batch_size=4,
gradient_accumulation_steps=8,
logging_steps=10,
num_train_epochs=1
)
2.6、创建训练器
trainer = Trainer(
args=args,
model=model,
train_dataset=tokenized_ds,
data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False)
)
2.7、训练
trainer.train()
2.8、推理
from transformers import pipeline
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, device=0)
pipe("西安交通大学博物馆(Xi'an Jiaotong University Museum)是一座位于西安", max_length=128, do_sample=True)
pipe("下面是一则游戏新闻。小编报道,近日,游戏产业发展的非常", max_length=128, do_sample=True)