因为接触机器学习时间比较短，这里就安装一下软件，运行一下案例程序，
速度非常快，能达到几十Mb/s,
完成后按照案例程序进行操作。
还有就是运行了分布式训练，
PaddlePaddle Distributed Training
/word2vec/prepare.py
63ffe52 on 19 Oct
42 lines (32 sloc)
1.09 KB
| import paddle.v2 as paddle |
| import tarfile |
| import os |
| import pickle |
| SPLIT_COUNT = 3 |
| N = 5 |
| def file_len(fd): |
| for i, l in enumerate(fd): |
| pass |
| return i + 1 |
| def split_from_reader_by_line(filename, reader, split_count): |
| fn = open(filename, "w") |
| for batch_id, batch_data in enumerate(reader()): |
| batch_data_str = [str(d) for d in batch_data] |
| fn.write(",".join(batch_data_str)) |
| fn.write("\n") |
| fn.close() |
| fn = open(filename, "r") |
| total_line_count = file_len(fn) |
| fn.close() |
| per_file_lines = total_line_count / split_count + 1 |
| cmd = "split -d -a 5 -l %d %s %s-" % (per_file_lines, filename, filename) |
| os.system(cmd) |
| word_dict = paddle.dataset.imikolov.build_dict() |
| with open("word_dict.pickle", "w") as dict_f: |
| pickle.dump(word_dict, dict_f) |
| split_from_reader_by_line("train.txt", |
| paddle.dataset.imikolov.train(word_dict, N), |
| SPLIT_COUNT) |
| split_from_reader_by_line("test.txt", |
| paddle.dataset.imikolov.test(word_dict, N), |
~/.cache/paddle/dataset/imikolov/simple-examples/3-combination#
这里有一个解说，
Example of training and evaluation of rnnlm model on the Penn Treebank Corpus. If the
training takes more than 20 minutes, it would be good to check section 4 in FAQ.txt in
rnnlm directory (performance issues).
others.
案例程序为book, 但是安装后生成的一个book无法运行，然后又从新下载了一个，
git clone https://github.com/PaddlePaddle/Paddle.git
然后再运行就没有问题了，
root@ng2ee1952-master-instance-09zlnfq3:~/book2/book# cd 01*
root@ng2ee1952-master-instance-09zlnfq3:~/book2/book/01.fit_a_line# lsfit_a_line.tar image index.cn.html index.html infer.py README.cn.md README.md train.py
root@ng2ee1952-master-instance-09zlnfq3:~/book2/book/01.fit_a_line# python train.py
I1210 16:24:09.096017 3256 Util.cpp:166] commandline: --use_gpu=False --trainer_count=1
I1210 16:24:09.118629 3256 GradientMachine.cpp:85] Initing parameters..
I1210 16:24:09.118664 3256 GradientMachine.cpp:92] Init parameters done.
Cache file /root/.cache/paddle/dataset/uci_housing/housing.data not found, downloading https://archive.ics.uci.edu/ml/machine-learning-databases/housing/housing.data
[==================================================]Pass 0, Batch 0, Cost 350.478027
Pass 0, Batch 100, Cost 144.011780
Pass 0, Batch 200, Cost 21.822021
Test 0, Cost 55.322908
Pass 1, Batch 0, Cost 30.754185
Pass 1, Batch 100, Cost 12.422053
Pass 1, Batch 200, Cost 27.100828
Test 1, Cost 22.610712
Pass 2, Batch 0, Cost 113.014809
Pass 2, Batch 100, Cost 2.062238
Pass 2, Batch 200, Cost 8.437082
Test 2, Cost 18.292072
Pass 3, Batch 0, Cost 17.645754
Pass 3, Batch 100, Cost 15.179324
Pass 3, Batch 200, Cost 46.365536
Test 3, Cost 17.722245
Pass 4, Batch 0, Cost 8.160726
Pass 4, Batch 100, Cost 15.524778
Pass 4, Batch 200, Cost 3.599603
Test 4, Cost 17.323252
Pass 5, Batch 0, Cost 1.100273
Pass 5, Batch 100, Cost 5.593287
Pass 5, Batch 200, Cost 7.624557
Test 5, Cost 16.895311
Pass 6, Batch 0, Cost 11.876204
Pass 6, Batch 100, Cost 26.956287
Pass 6, Batch 200, Cost 3.333240
Test 6, Cost 16.453065
Pass 7, Batch 0, Cost 43.242226
Pass 7, Batch 100, Cost 21.981386
Pass 7, Batch 200, Cost 12.113927
Test 7, Cost 16.023801
Pass 8, Batch 0, Cost 10.356236
Pass 8, Batch 100, Cost 408.588074
Pass 8, Batch 200, Cost 244.850235
Test 8, Cost 15.624351
Pass 9, Batch 0, Cost 16.430382
Pass 9, Batch 100, Cost 17.266754
Pass 9, Batch 200, Cost 5.391199
Test 9, Cost 15.282412
Pass 10, Batch 0, Cost 156.690918
Pass 10, Batch 100, Cost 12.585128
Pass 10, Batch 200, Cost 154.717743
Test 10, Cost 14.971308
Pass 11, Batch 0, Cost 20.233416
Pass 11, Batch 100, Cost 9.964501
Pass 11, Batch 200, Cost 229.265411
Test 11, Cost 14.612724
Pass 12, Batch 0, Cost 22.560265
Pass 12, Batch 100, Cost 12.851789
Pass 12, Batch 200, Cost 18.491701
Test 12, Cost 14.351880
Pass 13, Batch 0, Cost 12.568289
Pass 13, Batch 100, Cost 12.654366
Pass 13, Batch 200, Cost 11.971765
Test 13, Cost 14.143503
Pass 14, Batch 0, Cost 208.058945
Pass 14, Batch 100, Cost 23.672329
Pass 14, Batch 200, Cost 36.995781
Test 14, Cost 13.927430
Pass 15, Batch 0, Cost 6.115829
Pass 15, Batch 100, Cost 13.646911
Pass 15, Batch 200, Cost 12.065642
Test 15, Cost 13.807400
Pass 16, Batch 0, Cost 1.972460
Pass 16, Batch 100, Cost 27.642412
Pass 16, Batch 200, Cost 157.392105
Test 16, Cost 13.642984
Pass 17, Batch 0, Cost 28.668180
Pass 17, Batch 100, Cost 13.654858
Pass 17, Batch 200, Cost 291.427246
Test 17, Cost 13.523349
Pass 18, Batch 0, Cost 8.852428
Pass 18, Batch 100, Cost 0.137222
Pass 18, Batch 200, Cost 2.501552
Test 18, Cost 13.413879
Pass 19, Batch 0, Cost 6.659045
Pass 19, Batch 100, Cost 55.985928
Pass 19, Batch 200, Cost 2.636630
Test 19, Cost 13.338899
Pass 20, Batch 0, Cost 18.643250
Pass 20, Batch 100, Cost 34.386089
Pass 20, Batch 200, Cost 38.573933
Test 20, Cost 13.241313
Pass 21, Batch 0, Cost 5.276097
Pass 21, Batch 100, Cost 178.950150
Pass 21, Batch 200, Cost 51.521793
Test 21, Cost 13.203160
Pass 22, Batch 0, Cost 4.773140
Pass 22, Batch 100, Cost 37.894161
Pass 22, Batch 200, Cost 4.663883
Test 22, Cost 13.177613
Pass 23, Batch 0, Cost 97.522957
Pass 23, Batch 100, Cost 3.344421
Pass 23, Batch 200, Cost 20.911604
Test 23, Cost 13.089471
Pass 24, Batch 0, Cost 4.799085
Pass 24, Batch 100, Cost 262.496765
Pass 24, Batch 200, Cost 3.433479
Test 24, Cost 13.060977
Pass 25, Batch 0, Cost 205.243866
Pass 25, Batch 100, Cost 13.707615
Pass 25, Batch 200, Cost 88.187920
Test 25, Cost 13.058446
Pass 26, Batch 0, Cost 21.459761
Pass 26, Batch 100, Cost 13.670927
Pass 26, Batch 200, Cost 3.855029
Test 26, Cost 13.121759
Pass 27, Batch 0, Cost 44.330051
Pass 27, Batch 100, Cost 3.509866
Pass 27, Batch 200, Cost 79.958588
Test 27, Cost 13.185231
Pass 28, Batch 0, Cost 76.851936
Pass 28, Batch 100, Cost 10.062405
Pass 28, Batch 200, Cost 0.222403
Test 28, Cost 13.142699
Pass 29, Batch 0, Cost 20.023430
Pass 29, Batch 100, Cost 16.772104
Pass 29, Batch 200, Cost 8.239898
Test 29, Cost 13.200032
label=8.5, predict=11.7552
label=5.0, predict=13.6842
label=11.9, predict=10.7416
label=27.9, predict=18.0805
label=17.2, predict=13.0336
root@ng2ee1952-master-instance-09zlnfq3:~/book2/book/01.fit_a_line# ls
fit_a_line.tar index.cn.html inference_topology.pkl params_pass_0.tar params_pass_20.tar README.md
image index.html infer.py params_pass_10.tar README.cn.md train.py
root@ng2ee1952-master-instance-09zlnfq3:~/book2/book/01.fit_a_line# cd ..
root@ng2ee1952-master-instance-09zlnfq3:~/book2/book# ls
01.fit_a_line 04.word2vec 07.label_semantic_roles index.html pending serve
02.recognize_digits 05.recommender_system 08.machine_translation mnist-client README.cn.md
03.image_classification 06.understand_sentiment index.cn.html paddle README.md
root@ng2ee1952-master-instance-09zlnfq3:~/book2/book# cd 02*
root@ng2ee1952-master-instance-09zlnfq3:~/book2/book/02.recognize_digits# ls
client image index.cn.html index.html README.cn.md README.md train.py
root@ng2ee1952-master-instance-09zlnfq3:~/book2/book/02.recognize_digits# python train.py
I1210 16:29:03.251523 3296 Util.cpp:166] commandline: --use_gpu=False --trainer_count=1
[INFO 2017-12-10 16:29:03,361 layers.py:2539] output for __conv_pool_0___conv: c = 20, h = 24, w = 24, size = 11520
[INFO 2017-12-10 16:29:03,363 layers.py:2667] output for __conv_pool_0___pool: c = 20, h = 12, w = 12, size = 2880
[INFO 2017-12-10 16:29:03,363 layers.py:2539] output for __conv_pool_1___conv: c = 50, h = 8, w = 8, size = 3200
[INFO 2017-12-10 16:29:03,364 layers.py:2667] output for __conv_pool_1___pool: c = 50, h = 4, w = 4, size = 800
I1210 16:29:03.381567 3296 GradientMachine.cpp:85] Initing parameters..
I1210 16:29:03.383512 3296 GradientMachine.cpp:92] Init parameters done.
Cache file /root/.cache/paddle/dataset/mnist/train-images-idx3-ubyte.gz not found, downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
[==================================================]Cache file /root/.cache/paddle/dataset/mnist/train-labels-idx1-ubyte.gz not found, downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
[==================================================]Pass 0, Batch 0, Cost 3.086941, {'classification_error_evaluator': 0.9140625}
Pass 0, Batch 100, Cost 0.177784, {'classification_error_evaluator': 0.03125}
Pass 0, Batch 200, Cost 0.144659, {'classification_error_evaluator': 0.046875}
Pass 0, Batch 300, Cost 0.094071, {'classification_error_evaluator': 0.0390625}
Pass 0, Batch 400, Cost 0.135413, {'classification_error_evaluator': 0.03125}
Cache file /root/.cache/paddle/dataset/mnist/t10k-images-idx3-ubyte.gz not found, downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
[==================================================]Cache file /root/.cache/paddle/dataset/mnist/t10k-labels-idx1-ubyte.gz not found, downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
[==================================================]Test with Pass 0, Cost 0.070293, {'classification_error_evaluator': 0.0215000007301569}
Pass 1, Batch 0, Cost 0.185338, {'classification_error_evaluator': 0.0546875}
Pass 1, Batch 100, Cost 0.096369, {'classification_error_evaluator': 0.0078125}
Pass 1, Batch 200, Cost 0.037399, {'classification_error_evaluator': 0.015625}
Pass 1, Batch 300, Cost 0.122069, {'classification_error_evaluator': 0.0390625}
Pass 1, Batch 400, Cost 0.025024, {'classification_error_evaluator': 0.015625}
Test with Pass 1, Cost 0.048198, {'classification_error_evaluator': 0.016100000590085983}
Pass 2, Batch 0, Cost 0.069788, {'classification_error_evaluator': 0.03125}
Pass 2, Batch 100, Cost 0.020557, {'classification_error_evaluator': 0.0078125}
Pass 2, Batch 200, Cost 0.035924, {'classification_error_evaluator': 0.0078125}
Pass 2, Batch 300, Cost 0.085978, {'classification_error_evaluator': 0.0234375}
Pass 2, Batch 400, Cost 0.028562, {'classification_error_evaluator': 0.0078125}
Test with Pass 2, Cost 0.041632, {'classification_error_evaluator': 0.013899999670684338}
Pass 3, Batch 0, Cost 0.049302, {'classification_error_evaluator': 0.015625}
Pass 3, Batch 100, Cost 0.082377, {'classification_error_evaluator': 0.0234375}
Pass 3, Batch 200, Cost 0.027198, {'classification_error_evaluator': 0.0078125}
Pass 3, Batch 300, Cost 0.102874, {'classification_error_evaluator': 0.0234375}
Pass 3, Batch 400, Cost 0.006461, {'classification_error_evaluator': 0.0}
Test with Pass 3, Cost 0.037316, {'classification_error_evaluator': 0.013299999758601189}
Pass 4, Batch 0, Cost 0.007065, {'classification_error_evaluator': 0.0}
Pass 4, Batch 100, Cost 0.022764, {'classification_error_evaluator': 0.0078125}
Pass 4, Batch 200, Cost 0.009182, {'classification_error_evaluator': 0.0}
Pass 4, Batch 300, Cost 0.089309, {'classification_error_evaluator': 0.03125}
Pass 4, Batch 400, Cost 0.016304, {'classification_error_evaluator': 0.0078125}
Test with Pass 4, Cost 0.037039, {'classification_error_evaluator': 0.013199999928474426}
Best pass is 4, testing Avgcost is 0.0370385892283
The classification accuracy is 98.68%
Label of image/infer_3.png is: 3
root@ng2ee1952-master-instance-09zlnfq3:~/book2/book/02.recognize_digits# ls
client index.cn.html params_pass_0.tar params_pass_2.tar params_pass_4.tar README.md
image index.html params_pass_1.tar params_pass_3.tar README.cn.md train.py
root@ng2ee1952-master-instance-09zlnfq3:~/book2/book/02.recognize_digits#
扫一扫
5673
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
