qq_34418352的博客

在使用transformers里的GPT2Tokenizer时，看到一句话：GPT-2 BPE tokenizer. Peculiarities: Byte-level Byte-Pair-Encoding Requires a space to start the input string => the encoding methods should be called with theadd_prefix_spaceflag set toTrue. Otherwis...

采用一种简单的方式，截取每个样本前512个字符。随机mask一些词，其中80%被mask掉的词使用特殊符号代替，如[MASK]，10%使用随机词替代，10%使用原本的词替代。参考transformers开源代码，如下：def mask_tokens(self, inputs: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: labels = inputs.clone() # We sample a few tokens in

https://huggingface.co/transformers/glossary.html1Input IDs模型的输入，为序列经过tokenize之后的数字表示。推荐使用encode 或encode_plus方法。这两个方法会自动加上[CLS]、[SEP]等标记。The input ids are often the only required parameters to be passed to the model as input.They are token indices,..

https://github.com/huggingface/transformers1BERT exampleBertTokenizer.from_pretrained:Instantiate a :class:`~transformers.PreTrainedTokenizer` (or a derived class) from a predefined tokenizer.实例化一个tokenizer。tokenizer.tokenize(text)：Converts a stri.