上一篇记录到训练了双向语言模型，时间代价太高了。
已经花了很长时间了，索性把实验复现完毕。
（6）# construct vocabulary on classification datasets
命令：sh construct_vocab_classification.sh
结果：执行命令后，进程状态总是变成D+，无法运行
解决：用sudo权限执行命令。sudo sh construct_vocab_classification.sh
提示：
constract vocabulary based on frequency
Traceback (most recent call last):
File "construct_vocab_classification.py", line 47, in <module>
main()
File "construct_vocab_classification.py", line 20, in main
construct(args)
File "construct_vocab_classification.py", line 40, in construct
os.mkdir('vocabs')
OSError: [Errno 13] Permission denied: 'vocabs'
权限问题：
将目录权限放开。
提示（成功）：
$ sudo sh construct_vocab_classification.sh
['/root/.chainer/dataset/_dl_cache/c581bb3dc8835a95fade699c3ff6a395']
['/root/.chainer/dataset/_dl_cache/c581bb3dc8835a95fade699c3ff6a395', '/root/.chainer/dataset/_dl_cache/9db74a4b60ce66508feb44cf78339432']
constract vocabulary based on frequency
['/root/.chainer/dataset/_dl_cache/e5eb72af713d09fd07dc67d56b41d4a1']
['/root/.chainer/dataset/_dl_cache/e5eb72af713d09fd07dc67d56b41d4a1', '/root/.chainer/dataset/_dl_cache/c803e809e0cbc9d7227559391e37408e']
constract vocabulary based on frequency
['/root/.chainer/dataset/_dl_cache/0322898ec8ca6422238ad7ec330de85d']
['/root/.chainer/dataset/_dl_cache/0322898ec8ca6422238ad7ec330de85d', '/root/.chainer/dataset/_dl_cache/7988b14081fc008bce44e271d8ff5c4b']
constract vocabulary based on frequency
['/root/.chainer/dataset/_dl_cache/a2beeab41c565a113e818708e932942b']
constract vocabulary based on frequency
['/root/.chainer/dataset/_dl_cache/4d87a87bbfab1e2eaf49373130946045']
constract vocabulary based on frequency
['/root/.chainer/dataset/_dl_cache/78f766b05ebed43f21d8737900346a84']
constract vocabulary based on frequency
['/root/.chainer/dataset/_dl_cache/6e176e199408a4b72b849a5dadb55ad3']
constract vocabulary based on frequency
构建分类数据词汇表成功。
得到一个目录下基于这几个数据集的词汇表。
（7）从这一步开始，需要根据每个数据集进行微调。
# modify the model file for a different vocabulary
命令：
python alt_model_to_another_vocab.py --vocab datasets/wikitext-103-raw/spacy_wikitext-103-raw.train.vocab.t50 --new-vocab vocabs/stsa.fine.vocab.json -u 1024 --layer 1 --resume trained_bilm/best_model.npz --suffix stsa.fine
提示：
{
"gpu": -1,
"unit": 1024,
"layer": 1,
"vocab": "datasets/wikitext-103-raw/spacy_wikitext-103-raw.train.vocab.t50",
"new_vocab": "vocabs/stsa.fine.vocab.json",
"resume": "trained_bilm/best_model.npz",
"suffix": "stsa.fine"
}
vocab is loaded vocabs/stsa.fine.vocab.json
vocab = 16581
old vocab is loaded datasets/wikitext-103-raw/spacy_wikitext-103-raw.train.vocab.t50
old vocab = 49873
load
alter
(16581, 1024)
(16581, 1024)
11448/49873 word embs are exported
11448/16581 word embs are imported
save
（8） # finetune the model as a conditional version
python -u train.py -g 0 --labeled-dataset stsa.fine --vocab vocabs/stsa.fine.vocab.json -u 1024 --layer 1 --dropout 0.5 --batchsize 64 --out trained_bilm/stsa.fine --resume trained_bilm/best_model.npz.stsa.fine --epoch 5
提示：
{
"gpu": 0,
"out": "trained_bilm/stsa.fine",
"batchsize": 64,
"epoch": 5,
"gradclip": 10,
"lr": 0.0001,
"unit": 1024,
"layer": 1,
"dropout": 0.5,
"vocab": "vocabs/stsa.fine.vocab.json",
"train_path": null,
"valid_path": null,
"resume": "trained_bilm/best_model.npz.stsa.fine",
"labeled_dataset": "stsa.fine",
"no_label": false,
"validation": false
}
Downloading from https://raw.githubusercontent.com/harvardnlp/sent-conv-torch/master/data/stsa.fine.train...
['/*******************/python-3.6/lib/python3.6/site-packages/chainer/_dl_cache/0322898ec8ca6422238ad7ec330de85d']
Downloading from https://raw.githubusercontent.com/harvardnlp/sent-conv-torch/master/data/stsa.fine.test...
['/*******************/python-3.6/lib/python3.6/site-packages/chainer/_dl_cache/0322898ec8ca6422238ad7ec330de85d', '/*******************/python-3.6/lib/python3.6/site-packages/chainer/_dl_cache/7988b14081fc008bce44e271d8ff5c4b']
do not use test dataset. pls use validation split.
#train = 8544
#valid = 2210
#vocab = 16581
load trained_bilm/best_model.npz.stsa.fine
# labels = 5
133 iters per epoch
log and eval are scheduled at every (66, 'iteration') and (132, 'iteration')
iter/epoch 133
Training start
epoch iteration main/perp validation/main/perp elapsed_time
0 66 2.09423e+15 4761.39
0 132 7.99731e+12 7.9428e+06 4785.23
1 198 1.87301e+12 4819.74
1 264 2.40841e+11 209938 4843.53
2 330 1.10612e+11 4877.8
2 396 1.63296e+10 41288.2 4901.52
3 462 1.02435e+10 4936.95
3 528 3.02223e+09 17573.2 4960.59
4 594 5.56232e+08 4994.28
4 660 1.00142e+09 9616.01 5017.95
(9) Train a classifier
Baseline
python train_text_classifier.py -g 0 --dataset stsa.fine --resume-vocab vocabs/stsa.fine.vocab.json --out results_base --model rnn
提示：
{
"batchsize": 64,
"epoch": 100,
"gpu": 1,
"out": "SST/results_base",
"unit": 300,
"layer": 1,
"dropout": 0.4,
"learning_rate": 0.0001,
"dataset": "stsa.fine",
"model": "rnn",
"bilm": null,
"bilm_unit": 1024,
"bilm_layer": 1,
"bilm_dropout": 0.0,
"bilm_ratio": 0.25,
"bilm_temp": 1.0,
"bilm_mode": "sampling",
"bilm_gumbel": false,
"bilm_wordwise": true,
"bilm_add_original": 0.0,
"bilm_residual": 0.0,
"resume_vocab": "/********/contextual_augmentation-master/vocabs/stsa.fine.vocab.json",
"validation": false,
"seed": 2018,
"save_model": false,
"stop_epoch": 10,
"no_label": false
}
load vocab from /*******/contextual_augmentation-master/vocabs/stsa.fine.vocab.json
['/root/.chainer/dataset/_dl_cache/0322898ec8ca6422238ad7ec330de85d']
['/root/.chainer/dataset/_dl_cache/0322898ec8ca6422238ad7ec330de85d', '/root/.chainer/dataset/_dl_cache/7988b14081fc008bce44e271d8ff5c4b']
# train data: 8544
# test data: 2210
# vocab: 16581
# class: 5
epoch main/loss validation/main/loss main/accuracy validation/main/accuracy test/main/loss test/main/accuracy elapsed_time
1 1.57305 1.57781 0.267141 0.239819 5.53294
2 1.56486 1.57374 0.281485 0.258824 9.81271
3 1.54463 1.5258 0.310868 0.317195 14.1039
4 1.48117 1.48303 0.348097 0.328507 18.3488
5 1.41038 1.42105 0.383862 0.365611 22.6179
6 1.33871 1.39367 0.417411 0.371041 26.8669
7 1.28669 1.38074 0.439016 0.386878 31.1487
8 1.22635 1.37844 0.464991 0.374661 35.3999
9 1.17776 1.38003 0.491371 0.388688 39.6793
10 1.13627 1.36348 0.519737 0.39457 43.9109
11 1.088 1.39846 0.540695 0.390045 48.2005
12 1.05911 1.40114 0.564497 0.391403 52.4379
13 1.01027 1.40574 0.584422 0.38914 56.7045
14 0.975209 1.41473 0.603383 0.396833 60.9252
15 0.930594 1.44508 0.615905 0.400905 65.1977
16 0.902207 1.47048 0.629934 0.388235 69.4647
17 0.870874 1.48448 0.651236 0.402715 73.7276
18 0.826494 1.50877 0.666471 0.39095 78.0921
19 0.797559 1.5439 0.68097 0.386425 82.404
20 0.767177 1.58223 0.695724 0.397285 86.6515
21 0.73499 1.60273 0.709655 0.382805 90.9344
22 0.699356 1.62306 0.724859 0.387783 95.1913
23 0.682802 1.64172 0.732159 0.385068 99.5035
24 0.661334 1.64112 0.740836 0.395023 103.786
25 0.628351 1.69876 0.755947 0.39638 108.074
26 0.601496 1.70227 0.765273 0.385068 112.307
27 0.578849 1.73248 0.776003 0.397738 116.592
（10）# Using contextual augmentation
原命令：
python train_text_classifier.py -g 0 --dataset stsa.fine --resume-vocab vocabs/stsa.fine.vocab.json --out results_caug --model rnn -bilm trained_bilm/stsa.fine/best_model.npz -bilm-l 1 -bilm-u 1024
提示：
{
"batchsize": 64,
"epoch": 100,
"gpu": 1,
"out": "/*******/contextual_augmentation-master/SST/results_caug",
"unit": 300,
"layer": 1,
"dropout": 0.4,
"learning_rate": 0.0001,
"dataset": "stsa.fine",
"model": "rnn",
"bilm": "/******/contextual_augmentation-master/trained_bilm/stsa.fine/best_model.npz",
"bilm_unit": 1024,
"bilm_layer": 1,
"bilm_dropout": 0.0,
"bilm_ratio": 0.25,
"bilm_temp": 1.0,
"bilm_mode": "sampling",
"bilm_gumbel": false,
"bilm_wordwise": true,
"bilm_add_original": 0.0,
"bilm_residual": 0.0,
"resume_vocab": "/******/contextual_augmentation-master/vocabs/stsa.fine.vocab.json",
"validation": false,
"seed": 2018,
"save_model": false,
"stop_epoch": 10,
"no_label": false
}
load vocab from /******/contextual_augmentation-master/vocabs/stsa.fine.vocab.json
['/root/.chainer/dataset/_dl_cache/0322898ec8ca6422238ad7ec330de85d']
['/root/.chainer/dataset/_dl_cache/0322898ec8ca6422238ad7ec330de85d', '/root/.chainer/dataset/_dl_cache/7988b14081fc008bce44e271d8ff5c4b']
# train data: 8544
# test data: 2210
# vocab: 16581
# class: 5
# labels = 5
add label
epoch main/loss validation/main/loss main/accuracy validation/main/accuracy test/main/loss test/main/accuracy elapsed_time
1 1.57721 1.57962 0.267491 0.241176 26.8906
2 1.56836 1.57728 0.275728 0.252941 46.7234
3 1.56535 1.57369 0.284981 0.259276 66.791
4 1.55118 1.5396 0.305216 0.300453 86.8256
5 1.51495 1.49183 0.336171 0.322172 106.932
6 1.48046 1.44466 0.350681 0.342081 126.932
7 1.43581 1.4323 0.370336 0.357919 147.119
8 1.40233 1.39542 0.387805 0.373303 167.114
9 1.36999 1.38647 0.408699 0.370588 187.296
10 1.34571 1.37571 0.413181 0.390498 207.331
11 1.31885 1.36875 0.43097 0.388235 227.449
12 1.30054 1.36042 0.446194 0.40543 247.46
13 1.27073 1.34711 0.457439 0.39819 267.677
14 1.26248 1.34936 0.460879 0.402715 287.707
15 1.22964 1.36058 0.476446 0.407692 307.863
16 1.22831 1.36657 0.482378 0.408145 327.845
17 1.19779 1.37455 0.496385 0.409955 348.039
18 1.17635 1.35929 0.506109 0.409955 368.094
19 1.15656 1.36487 0.521222 0.40362 388.301
20 1.14338 1.3721 0.520677 0.406335 408.314
21 1.12981 1.36429 0.540812 0.402262 428.531
22 1.10861 1.37445 0.540766 0.402262 448.534
23 1.08891 1.40454 0.551423 0.384615 468.683
24 1.07815 1.36887 0.560033 0.411765 488.679
25 1.04886 1.39302 0.572295 0.416742 508.816
26 1.03461 1.3719 0.582472 0.410407 528.875
27 1.03791 1.38794 0.573111 0.407692 549.084
28 1.00891 1.44629 0.591518 0.409955 569.172
29 0.991101 1.45043 0.605527 0.40543 589.367
30 0.982054 1.41546 0.606908 0.404525 609.458
31 0.96531 1.43088 0.614156 0.413122 629.658
32 0.932189 1.48758 0.628054 0.411765 649.678
33 0.937657 1.43412 0.633512 0.406787 669.828
34 0.923997 1.44117 0.641212 0.40543 689.867
35 0.909982 1.49308 0.63223 0.398643 710.06
截止到现在，复现stsa数据的实验结果完毕，和论文中的结果是可以对照上的。
这个数据增强方法，只在分类上做了结果的输出，没有对数据增强后的结果进行输出，目前，在纠结要不要去尝试改代码，让其输出数据增强后的数据，难度略大。。。
纠结的原因：
1. 要对应中文的话，需要重新训练中文的语言模型。
2. 要改代码，目前还不知道从哪里改起。
3. 不知道是不是可以使用中文数据进行这个实验。如果不可以，改了也是白改。
1296
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
