分布式tensorflow测试结果及问题

最新推荐文章于 2022-07-12 23:29:52 发布

__Sunny__

最新推荐文章于 2022-07-12 23:29:52 发布

阅读量1.2k

点赞数

分类专栏： TensorFlow 文章标签： TensorFlow

本文链接：https://blog.csdn.net/s_sunnyy/article/details/71172895

版权

TensorFlow 专栏收录该内容

60 篇文章 1 订阅

订阅专栏

分布式TensorFlow+GoogleNet+between-graph模式+异步更新

先写测试结果出来后的问题，希望有经验的大牛或者正在研究的能够解答，或者提供想法和建议：

1. 测试结果是否正确，从测试结果看各个worker节点的性能是不一样的，差异比较明显（相差3倍以上）。

2. 最后的性能（运行时间）是如何统计（我在这里是取的平均值）

3.如果按我算的平均值的结果来看，多个节点没有达到理想的加速比。

测试结果：

Nodes	Batch	sec/batch	images/s
4	128	7.095	72.163
8	128	8.822	116.073
16	128	17.126	119.584
32	128	33.581	121.974

具体输出：

1） 4nodes: 1ps-4worker

7.095 sec/batch （取的平均值->感觉不对啊）

2） 8nodes: 1ps-8worker

8.822 sec/batch

3） 16nodes: 1ps-16worker

17.126 sec/batch

4） 32nodes: 1ps-32worker

33.581 sec/batch

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Running inter_op : 2
Running intra_op : 64
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
task_index=0
2017-05-03 20:47:46.719618: batch_size = 128, step 45 (32.6 examples/sec; 3.925 sec/batch)
2017-05-03 20:47:50.428333: batch_size = 128, step 50 (34.5 examples/sec; 3.708 sec/batch)
2017-05-03 20:48:10.352844: batch_size = 128, step 65 (32.1 examples/sec; 3.985 sec/batch)
2017-05-03 20:48:14.102468: batch_size = 128, step 70 (34.1 examples/sec; 3.749 sec/batch)
2017-05-03 20:48:25.732333: batch_size = 128, step 85 (32.6 examples/sec; 3.930 sec/batch)
2017-05-03 20:48:50.170304: batch_size = 128, step 110 (32.4 examples/sec; 3.952 sec/batch)
2017-05-03 20:48:54.406515: batch_size = 128, step 115 (30.2 examples/sec; 4.235 sec/batch)
2017-05-03 20:48:58.970412: batch_size = 128, step 125 (28.0 examples/sec; 4.563 sec/batch)
2017-05-03 20:49:03.141752: batch_size = 128, step 130 (30.7 examples/sec; 4.171 sec/batch)
2017-05-03 20:50:32.605188: batch_size = 128, step 225 (34.5 examples/sec; 3.712 sec/batch)

task_index=1
2017-05-03 05:48:31.200203: batch_size = 128, step 30 (5.4 examples/sec; 23.689 sec/batch)
2017-05-03 05:49:42.633059: batch_size = 128, step 80 (3.3 examples/sec; 38.349 sec/batch)
2017-05-03 05:50:53.710354: batch_size = 128, step 160 (3.6 examples/sec; 35.776 sec/batch)
2017-05-03 05:54:23.360670: batch_size = 128, step 390 (4.3 examples/sec; 30.052 sec/batch)
2017-05-03 05:56:49.983847: batch_size = 128, step 530 (3.1 examples/sec; 41.558 sec/batch)
2017-05-03 05:57:12.841021: batch_size = 128, step 575 (5.6 examples/sec; 22.857 sec/batch)

task_index=2
2017-05-03 05:51:57.007129: batch_size = 128, step 185 (3.3 examples/sec; 39.060 sec/batch)
2017-05-03 05:56:40.613737: batch_size = 128, step 485 (3.9 examples/sec; 32.600 sec/batch)

task_index=3
2017-05-03 05:55:06.794296: batch_size = 128, step 490 (2.1 examples/sec; 60.772 sec/batch)

task_index=4
2017-05-03 05:45:26.607303: batch_size = 128, step 15 (1.1 examples/sec; 113.131 sec/batch)
2017-05-03 05:49:12.911162: batch_size = 128, step 305 (2.5 examples/sec; 51.150 sec/batch)

task_index=5
2017-05-03 05:49:36.528040: batch_size = 128, step 100 (3.7 examples/sec; 34.204 sec/batch)
2017-05-03 05:50:29.954126: batch_size = 128, step 140 (2.4 examples/sec; 53.426 sec/batch)
2017-05-03 05:54:38.455900: batch_size = 128, step 425 (3.9 examples/sec; 32.787 sec/batch)

task_index=6
2017-05-03 05:44:34.802525: batch_size = 128, step 35 (3.6 examples/sec; 35.366 sec/batch)
2017-05-03 05:46:56.584304: batch_size = 128, step 180 (3.4 examples/sec; 37.440 sec/batch)
2017-05-03 05:49:29.225924: batch_size = 128, step 340 (3.1 examples/sec; 40.679 sec/batch)

task_index=7
2017-05-03 05:55:03.756407: batch_size = 128, step 555 (6.5 examples/sec; 19.801 sec/batch)

task_index=8
2017-05-03 13:44:35.123241: batch_size = 128, step 15 (6.7 examples/sec; 19.243 sec/batch)
2017-05-03 13:47:00.980776: batch_size = 128, step 145 (3.8 examples/sec; 33.695 sec/batch)
2017-05-03 13:50:23.666216: batch_size = 128, step 370 (4.9 examples/sec; 25.957 sec/batch)
2017-05-03 13:50:48.370081: batch_size = 128, step 395 (5.2 examples/sec; 24.703 sec/batch)
2017-05-03 13:52:43.737968: batch_size = 128, step 510 (3.9 examples/sec; 32.465 sec/batch)
2017-05-03 13:53:09.098748: batch_size = 128, step 540 (5.0 examples/sec; 25.360 sec/batch)

task_index=9
2017-05-03 05:49:26.793108: batch_size = 128, step 120 (3.6 examples/sec; 35.827 sec/batch)
2017-05-03 05:50:20.626469: batch_size = 128, step 190 (5.5 examples/sec; 23.267 sec/batch)
2017-05-03 05:54:36.051785: batch_size = 128, step 450 (4.3 examples/sec; 29.934 sec/batch)
2017-05-03 05:55:00.836508: batch_size = 128, step 480 (5.2 examples/sec; 24.784 sec/batch)

task_index=10
2017-05-03 05:46:48.098061: batch_size = 128, step 90 (3.4 examples/sec; 37.158 sec/batch)
2017-05-03 05:47:18.741101: batch_size = 128, step 135 (4.2 examples/sec; 30.643 sec/batch)
2017-05-03 05:49:02.799734: batch_size = 128, step 245 (4.1 examples/sec; 31.470 sec/batch)
2017-05-03 05:54:33.105420: batch_size = 128, step 585 (3.6 examples/sec; 35.395 sec/batch)

task_index=11
2017-05-03 05:44:42.909636: batch_size = 128, step 300 (5.1 examples/sec; 25.243 sec/batch)
2017-05-03 05:48:08.129838: batch_size = 128, step 515 (5.0 examples/sec; 25.644 sec/batch)
2017-05-03 05:48:51.692858: batch_size = 128, step 570 (8.2 examples/sec; 15.561 sec/batch)

task_index=12
2017-05-03 05:47:59.707772: batch_size = 128, step 5 (7.9 examples/sec; 16.248 sec/batch)
2017-05-03 05:48:16.931354: batch_size = 128, step 20 (7.4 examples/sec; 17.223 sec/batch)
2017-05-03 05:49:52.540310: batch_size = 128, step 95 (4.5 examples/sec; 28.506 sec/batch)

task_index=13
2017-05-03 05:47:24.053530: batch_size = 128, step 195 (4.0 examples/sec; 31.822 sec/batch)
2017-05-03 05:48:19.802468: batch_size = 128, step 260 (4.4 examples/sec; 29.275 sec/batch)
2017-05-03 05:49:49.996623: batch_size = 128, step 355 (4.5 examples/sec; 28.262 sec/batch)

task_index=14
2017-05-03 05:48:08.411386: batch_size = 128, step 295 (5.5 examples/sec; 23.423 sec/batch)

task_index=15
2017-05-03 05:45:48.658595: batch_size = 128, step 15 (3.1 examples/sec; 40.729 sec/batch)
2017-05-03 05:46:54.905363: batch_size = 128, step 75 (3.4 examples/sec; 37.579 sec/batch)
2017-05-03 05:48:53.505604: batch_size = 128, step 200 (3.1 examples/sec; 41.327 sec/batch)
2017-05-03 05:50:36.849856: batch_size = 128, step 310 (3.1 examples/sec; 40.790 sec/batch)
2017-05-03 05:51:41.509382: batch_size = 128, step 380 (3.6 examples/sec; 35.182 sec/batch)
2017-05-03 05:52:16.279572: batch_size = 128, step 420 (3.7 examples/sec; 34.770 sec/batch)

task_index=16
2017-05-03 05:45:40.311368: batch_size = 128, step 10 (11.2 examples/sec; 11.407 sec/batch)
2017-05-03 05:48:42.277773: batch_size = 128, step 170 (4.1 examples/sec; 31.530 sec/batch)
2017-05-03 05:50:49.874289: batch_size = 128, step 325 (6.9 examples/sec; 18.427 sec/batch)
2017-05-03 05:51:14.941023: batch_size = 128, step 345 (5.1 examples/sec; 25.066 sec/batch)
2017-05-03 05:51:39.810068: batch_size = 128, step 365 (5.1 examples/sec; 24.869 sec/batch)
2017-05-03 05:52:25.087176: batch_size = 128, step 415 (5.6 examples/sec; 22.814 sec/batch)
2017-05-03 05:53:41.619113: batch_size = 128, step 495 (6.0 examples/sec; 21.230 sec/batch)

task_index=17
2017-05-03 05:50:09.882157: batch_size = 128, step 360 (4.1 examples/sec; 30.934 sec/batch)

task_index=18
2017-05-03 12:48:23.787143: batch_size = 128, step 15 (1.6 examples/sec; 77.810 sec/batch)
2017-05-03 12:50:57.380887: batch_size = 128, step 220 (4.1 examples/sec; 31.324 sec/batch)
2017-05-03 12:51:50.350145: batch_size = 128, step 275 (4.1 examples/sec; 31.343 sec/batch)
2017-05-03 12:54:22.509583: batch_size = 128, step 430 (4.1 examples/sec; 31.472 sec/batch)

task_index=19
2017-05-03 12:50:10.655159: batch_size = 128, step 165 (3.9 examples/sec; 32.472 sec/batch)
2017-05-03 12:54:31.838019: batch_size = 128, step 445 (4.9 examples/sec; 26.069 sec/batch)
2017-05-03 12:55:21.823974: batch_size = 128, step 500 (5.3 examples/sec; 24.304 sec/batch)
2017-05-03 12:55:54.275059: batch_size = 128, step 525 (3.9 examples/sec; 32.451 sec/batch)

task_index=20
2017-05-03 12:50:40.915674: batch_size = 128, step 205 (4.5 examples/sec; 28.568 sec/batch)
2017-05-03 12:51:33.622872: batch_size = 128, step 265 (4.9 examples/sec; 26.191 sec/batch)
2017-05-03 12:51:59.174208: batch_size = 128, step 290 (5.0 examples/sec; 25.551 sec/batch)
2017-05-03 12:55:21.325603: batch_size = 128, step 505 (6.9 examples/sec; 18.655 sec/batch)
2017-05-03 12:56:11.405814: batch_size = 128, step 565 (12.0 examples/sec; 10.680 sec/batch)
2017-05-03 12:56:30.378659: batch_size = 128, step 580 (6.7 examples/sec; 18.972 sec/batch)

task_index=21
2017-05-03 12:51:31.847710: batch_size = 128, step 255 (4.2 examples/sec; 30.397 sec/batch)
2017-05-03 12:56:23.674076: batch_size = 128, step 560 (4.7 examples/sec; 27.067 sec/batch)
2017-05-03 12:56:47.261972: batch_size = 128, step 590 (5.4 examples/sec; 23.587 sec/batch)

task_index=22
2017-05-03 12:48:42.576828: batch_size = 128, step 55 (3.0 examples/sec; 42.159 sec/batch)
2017-05-03 12:50:06.411809: batch_size = 128, step 150 (3.0 examples/sec; 42.724 sec/batch)
2017-05-03 12:51:20.961616: batch_size = 128, step 240 (3.7 examples/sec; 34.329 sec/batch)

task_index=23
2017-05-03 12:48:45.637563: batch_size = 128, step 60 (3.2 examples/sec; 40.447 sec/batch)
2017-05-03 12:50:55.156800: batch_size = 128, step 210 (3.2 examples/sec; 39.540 sec/batch)
2017-05-03 12:54:55.569735: batch_size = 128, step 465 (4.1 examples/sec; 31.572 sec/batch)

task_index=24
2017-05-03 12:49:30.226793: batch_size = 128, step 105 (2.8 examples/sec; 45.144 sec/batch)
2017-05-03 12:50:05.809792: batch_size = 128, step 155 (3.6 examples/sec; 35.583 sec/batch)
2017-05-03 12:52:30.430595: batch_size = 128, step 335 (7.5 examples/sec; 16.981 sec/batch)
2017-05-03 12:53:00.332736: batch_size = 128, step 350 (4.3 examples/sec; 29.902 sec/batch)
2017-05-03 12:53:54.153751: batch_size = 128, step 405 (4.3 examples/sec; 29.591 sec/batch)
2017-05-03 12:54:22.383400: batch_size = 128, step 435 (4.5 examples/sec; 28.229 sec/batch)

task_index=25
2017-05-03 12:51:58.798826: batch_size = 128, step 285 (4.4 examples/sec; 29.191 sec/batch)
2017-05-03 12:52:38.400311: batch_size = 128, step 315 (3.2 examples/sec; 39.601 sec/batch)
2017-05-03 12:55:03.666079: batch_size = 128, step 475 (4.5 examples/sec; 28.373 sec/batch)

task_index=26
2017-05-03 12:50:29.503593: batch_size = 128, step 175 (3.1 examples/sec; 41.548 sec/batch)
2017-05-03 12:51:54.122309: batch_size = 128, step 280 (4.2 examples/sec; 30.238 sec/batch)
2017-05-03 12:53:26.468322: batch_size = 128, step 375 (4.6 examples/sec; 27.600 sec/batch)
2017-05-03 12:54:30.155472: batch_size = 128, step 440 (4.0 examples/sec; 31.646 sec/batch)
2017-05-03 12:55:59.342549: batch_size = 128, step 535 (4.6 examples/sec; 27.935 sec/batch)
2017-05-03 12:57:03.475904: batch_size = 128, step 595 (3.7 examples/sec; 34.229 sec/batch)

task_index=27
2017-05-03 12:48:13.815870: batch_size = 128, step 10 (1.7 examples/sec; 75.328 sec/batch)
2017-05-03 12:50:54.686255: batch_size = 128, step 215 (3.9 examples/sec; 32.586 sec/batch)
2017-05-03 12:51:21.410746: batch_size = 128, step 250 (4.8 examples/sec; 26.724 sec/batch)
2017-05-03 12:54:58.227361: batch_size = 128, step 470 (4.5 examples/sec; 28.602 sec/batch)
2017-05-03 12:57:05.750578: batch_size = 128, step 600 (4.0 examples/sec; 32.095 sec/batch)

task_index=28
2017-05-03 12:51:45.102666: batch_size = 128, step 270 (4.2 examples/sec; 30.402 sec/batch)
2017-05-03 12:52:50.112646: batch_size = 128, step 330 (3.0 examples/sec; 43.102 sec/batch)
2017-05-03 12:55:48.659364: batch_size = 128, step 520 (3.8 examples/sec; 33.699 sec/batch)

task_index=29
2017-05-03 12:53:50.393170: batch_size = 128, step 400 (4.2 examples/sec; 30.300 sec/batch)
2017-05-03 12:54:40.773937: batch_size = 128, step 455 (5.2 examples/sec; 24.409 sec/batch)
2017-05-03 12:56:06.510080: batch_size = 128, step 545 (5.6 examples/sec; 22.695 sec/batch)

task_index=30
2017-05-03 12:51:07.914241: batch_size = 128, step 230 (4.3 examples/sec; 29.816 sec/batch)
2017-05-03 12:56:39.176273: batch_size = 128, step 550 (2.6 examples/sec; 49.355 sec/batch)

task_index=31
2017-05-03 12:47:34.777994: batch_size = 128, step 25 (9.8 examples/sec; 13.091 sec/batch)
2017-05-03 12:48:09.510539: batch_size = 128, step 40 (3.7 examples/sec; 34.732 sec/batch)
2017-05-03 12:51:06.540000: batch_size = 128, step 235 (5.0 examples/sec; 25.382 sec/batch)
2017-05-03 12:52:19.058394: batch_size = 128, step 320 (7.4 examples/sec; 17.213 sec/batch)
2017-05-03 12:53:29.515982: batch_size = 128, step 385 (5.9 examples/sec; 21.523 sec/batch)
2017-05-03 12:53:55.709580: batch_size = 128, step 410 (4.9 examples/sec; 26.193 sec/batch)
2017-05-03 12:54:45.078163: batch_size = 128, step 460 (5.4 examples/sec; 23.792 sec/batch)

__Sunny__

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
1
评论
分布式tensorflow测试结果及问题

分布式TensorFlow+GoogleNet+between-graph模式+异步更新先写测试结果出来后的问题，希望有经验的大牛或者正在研究的能够解答，或者提供想法和建议：1. 测试结果是否正确，从测试结果看各个worker节点的性能是不一样的，差异比较明显（相差3倍以上）。2. 最后的性能（运行时间）是如何统计（我在这里是取的平均值）3.如果按我算的平均值的结果来看
复制链接

扫一扫