HuggingFace模型训练流程搭建准确性验证

1.训练流程搭建

使用HuggingFace工具中的transformers模型搭建训练流程，传说核心训练代码5行搞定。今天我们简单搭建一个二分类的文本分类任务。

搭建训练流程demo.py，如下：

整理中

2.验证Loss是否收敛、模型参数是否改变

关键：针对分类任务来说，训练loss会变小。简单实用几个样本，100epoch，看看loss是否收敛，如果收敛，那么训练流程正确，否则，你需要仔细的check你的代码逻辑(一定要忘记之前的逻辑，重新开始梳理。。因为自己总觉得自己是对的，需要摒弃这种想法，重新来过，以一个新的视角去排查，才可能发现错误)。

运行demo.py，打印信息的结果如下：

Truncation was not explicitely activated but `max_length` is provided a specific value, please use `truncation=True` to explicitely truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
/data/deploy/wang/bertt/lib/python3.6/site-packages/transformers/tokenization_utils_base.py:1944: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).
  FutureWarning,
Some weights of the model checkpoint at bert-base-chinese were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-chinese and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Let's use 4 GPUs!
357
<torch.utils.data.dataloader.DataLoader object at 0x7f557a48c940>
Epoch [1/10]
/data/deploy/wang/bertt/lib/python3.6/site-packages/torch/nn/parallel/_functions.py:64: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
Iter:      0,  Train Loss:   1.8,  Train Acc: 14.84%,  Val Loss:   1.9,  Val Acc: 16.43%,  Time: 0:00:43 *
Iter:    100,  Train Loss:  0.64,  Train Acc: 82.03%,  Val Loss:  0.63,  Val Acc: 79.52%,  Time: 0:02:14 *
Iter:    200,  Train Loss:   0.4,  Train Acc: 85.94%,  Val Loss:  0.31,  Val Acc: 87.70%,  Time: 0:03:45 *
Iter:    300,  Train Loss:  0.34,  Train Acc: 85.16%,  Val Loss:  0.27,  Val Acc: 88.87%,  Time: 0:05:17 *
Epoch [2/10]
Iter:    400,  Train Loss:  0.17,  Train Acc: 92.19%,  Val Loss:  0.26,  Val Acc: 89.30%,  Time: 0:06:48 *
Iter:    500,  Train Loss:  0.24,  Train Acc: 92.19%,  Val Loss:  0.24,  Val Acc: 90.36%,  Time: 0:08:19 *
Iter:    600,  Train Loss:  0.19,  Train Acc: 92.19%,  Val Loss:  0.24,  Val Acc: 90.58%,  Time: 0:09:51 *
Iter:    700,  Train Loss:  0.18,  Train Acc: 94.53%,  Val Loss:  0.23,  Val Acc: 90.55%,  Time: 0:11:22 *
Epoch [3/10]
Iter:    800,  Train Loss:  0.17,  Train Acc: 93.75%,  Val Loss:  0.24,  Val Acc: 90.58%,  Time: 0:12:52
Iter:    900,  Train Loss:  0.15,  Train Acc: 96.09%,  Val Loss:  0.24,  Val Acc: 90.62%,  Time: 0:14:22
Iter:   1000,  Train Loss:  0.27,  Train Acc: 89.06%,  Val Loss:  0.23,  Val Acc: 90.76%,  Time: 0:15:53 *
Epoch [4/10]
Iter:   1100,  Train Loss:  0.15,  Train Acc: 95.31%,  Val Loss:  0.24,  Val Acc: 91.02%,  Time: 0:17:23
Iter:   1200,  Train Loss:  0.12,  Train Acc: 94.53%,  Val Loss:  0.25,  Val Acc: 90.82%,  Time: 0:18:53
Iter:   1300,  Train Loss: 0.059,  Train Acc: 98.44%,  Val Loss:  0.24,  Val Acc: 90.87%,  Time: 0:20:24
Iter:   1400,  Train Loss:  0.16,  Train Acc: 92.97%,  Val Loss:  0.25,  Val Acc: 90.72%,  Time: 0:21:54
Epoch [5/10]
Iter:   1500,  Train Loss: 0.094,  Train Acc: 96.88%,  Val Loss:  0.27,  Val Acc: 91.02%,  Time: 0:23:24
Iter:   1600,  Train Loss:  0.12,  Train Acc: 96.09%,  Val Loss:  0.28,  Val Acc: 90.70%,  Time: 0:24:55
Iter:   1700,  Train Loss:  0.13,  Train Acc: 96.09%,  Val Loss:  0.28,  Val Acc: 90.84%,  Time: 0:26:25
Epoch [6/10]
Iter:   1800,  Train Loss: 0.053,  Train Acc: 97.66%,  Val Loss:  0.28,  Val Acc: 90.85%,  Time: 0:27:55
Iter:   1900,  Train Loss: 0.075,  Train Acc: 97.66%,  Val Loss:   0.3,  Val Acc: 91.24%,  Time: 0:29:25
Iter:   2000,  Train Loss: 0.097,  Train Acc: 96.09%,  Val Loss:  0.31,  Val Acc: 90.45%,  Time: 0:30:56
Iter:   2100,  Train Loss: 0.041,  Train Acc: 98.44%,  Val Loss:  0.31,  Val Acc: 90.58%,  Time: 0:32:26
No optimization for a long time, auto-stopping...
train total time is : 1970.5445399284363

可见train loss是收敛的，训练流程没有问题。

看看训练参数是否改变：

模型训练前：
<torch.utils.data.dataloader.DataLoader object at 0x7fe49e13ca20>
----name : module.bert.encoder.layer.11.output.LayerNorm.bias
 parameter : Parameter containing:
tensor([-7.5394e-02, -5.3339e-02, -8.4475e-02,  1.3116e-01, -7.9671e-03,
         2.0331e-01, -1.0460e-01,  9.0587e-02, -1.6743e-01,  1.6022e-02,
        -4.2025e-02, -2.6452e-02,  5.9012e-02, -1.6719e-02, -2.2786e-01,
        -3.5230e-02,  3.8843e-02, -6.1143e-02, -1.8396e-03, -5.0667e-02,
        -9.7109e-02,  9.6873e-02,  4.0096e-02, -9.9374e-02, -3.5263e-02,
        -6.1067e-02, -1.9375e-01, -1.2152e-01, -2.8524e-02, -1.5492e-01,
         1.6237e-02, -8.2809e-02, -1.4949e-01,  1.4481e-01, -8.8371e-02,

训练一轮后：
tensor([-7.5270e-02, -5.3138e-02, -8.5101e-02,  1.3082e-01, -8.3281e-03,
         2.0305e-01, -1.0474e-01,  9.1127e-02, -1.6659e-01,  1.5437e-02,
        -4.1256e-02, -2.7110e-02,  5.8693e-02, -1.5990e-02, -2.2822e-01,
        -3.4954e-02,  3.8195e-02, -6.1271e-02, -1.4476e-03, -5.0210e-02,
        -9.7067e-02,  9.6599e-02,  4.0670e-02, -9.8674e-02, -3.5899e-02,
        -6.1480e-02, -1.9452e-01, -1.2167e-01, -2.8671e-02, -1.5562e-01,

module.classifier.weight
module.classifier.bias

3.打印模型结构

print(model) 即可打印模型结构，如下：

---models ： BertForSequenceClassification(
  (bert): BertModel(
    (embeddings): BertEmbeddings(
      (word_embeddings): Embedding(21128, 768, padding_idx=0)
      (position_embeddings): Embedding(512, 768)
      (token_type_embeddings): Embedding(2, 768)
      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (encoder): BertEncoder(
      (layer): ModuleList(
        (0): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (1): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (2): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (3): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (4): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (5): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (6): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (7): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (8): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (9): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (10): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (11): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
      )
    )
    (pooler): BertPooler(
      (dense): Linear(in_features=768, out_features=768, bias=True)
      (activation): Tanh()
    )
  )
  (dropout): Dropout(p=0.1, inplace=False)
  (classifier): Linear(in_features=768, out_features=2, bias=True)
)

参考：

1.