对Linux Server进行性能评估,从带宽和时延两个方面,对Linux Server的CPU、内存、操作系统和IO等指标进行评估,主要借鉴lmbench这个框架。各项测量原理见参考链接
目前主要完成
带宽 Bandwidth
Memory
read
write
copy
Pipe
interact with parent and child process
时延 Latency
create Process
fork
fork + execve
调用操作系统接口
system call
getpid
getppid
read/write via VFS
read /dev/zero
write /dev/null
内存子系统时延
L1 Cache Latency
L2 Cache Latency
L3 Cache Latency
Memory 带宽
ThinkdPad-X220,搭配intel i5-2520M CPU,操作系统为Ubuntu 16.10 64bit
L1 Cache 32KB8路组相连 64B Cache line
L2 Cache 256KB8路组相连 64B Cache line
L3 Cache 3072KB12路组相连 64B Cache line
读内存
测试内存带宽的意义在于衡量计算机系统内,从主存中,经过各级存储和缓存将数据读到CPU的能力。在测试内存带宽的过程中,采用不同大小的数据块,将数据块内每个元素读入CPU,进行测试,可以大致得出各级存储的带宽。
读内存带宽
下表是Memory Read的读性能,可以看到当数据块大小为4KB~32KB时,带宽明显高于64KB数据块,因为当前CPU的L1 Cache大小为32KB,使用较小数据块时,尤其在循环读写的过程中,可以充分享受L1 Cache带来的红利;
一旦数据块大小超过L2 Cache,由于L1 Cache发生Conlict,带宽下降;随后在数据块大小达到L2 Cache的256KB和L3 Cache的3072KB时都会有带宽的跳变;
由于L1和L2关系亲密,访问速度和容量的差别在10倍以内,所以L1 miss时L2可以将数据稳定在50000MB/s;
而当数据接近L3 Cache时,由于L3访问速度慢,且容量较大,而且是共享的,因此此时较多的体现的是L3的速度,并且在数据块增大并较多的引起高速缓存失效时,访问带宽接近于从主存中读取数据。
目前存在的顾虑
数据块4KB大小时的测试带宽是74036MB/s,平均每ns读取78Byte,对于睿频达3GHz的CPU而言,相当于每个cycle读取26B,如何解释呢, CPU每次从L1 Cache读取数据以cache line为单位
做bench过程中,各级存储是怎样起作用的,毕竟系统总线PCIE也才10GB/s的带宽,测试达到这个数据量的话一定是cache起到很大作用
写内存带宽
benchmark on Bandwidth of Memory
-------------------------------------
Bandwidth of Memory Read
blocksize: 4KBbandwidth of mem read: 74036.584484 MB/s
blocksize: 8KBbandwidth of mem read: 74990.845844 MB/s
blocksize: 16KBbandwidth of mem read: 73919.006713 MB/s
blocksize: 32KBbandwidth of mem read: 71869.736103 MB/s
blocksize: 48KBbandwidth of mem read: 50845.840161 MB/s
blocksize: 64KBbandwidth of mem read: 50965.558431 MB/s
blocksize: 128KBbandwidth of mem read: 51501.282503 MB/s
blocksize: 192KBbandwidth of mem read: 49226.035775 MB/s
blocksize: 256KBbandwidth of mem read: 41491.085900 MB/s
blocksize: 512KBbandwidth of mem read: 31507.692308 MB/s
blocksize: 1024KBbandwidth of mem read: 31075.503763 MB/s
blocksize: 2048KBbandwidth of mem read: 26670.139341 MB/s
blocksize: 2560KBbandwidth of mem read: 22759.090370 MB/s
blocksize: 3072KBbandwidth of mem read: 19567.377495 MB/s
blocksize: 4096KBbandwidth of mem read: 17341.236240 MB/s
blocksize: 8192KBbandwidth of mem read: 16377.449020 MB/s
blocksize: 16384KBbandwidth of mem read: 16633.637634 MB/s
blocksize: 32768KBbandwidth of mem read: 16737.222340 MB/s
blocksize: 65536KBbandwidth of mem read: 16818.868668 MB/s
blocksize: 131072KBbandwidth of mem read: 16860.962919 MB/s
Bandwidth of Memory Write
blocksize: 4KBbandwidth of mem write: 45464.636150 MB/s
blocksize: 8KBbandwidth of mem write: 45636.866031 MB/s
blocksize: 16KBbandwidth of mem write: 45096.225833 MB/s
blocksize: 32KBbandwidth of mem write: 37205.246521 MB/s
blocksize: 64KBbandwidth of mem write: 27972.027972 MB/s
blocksize: 128KBbandwidth of mem write: 27495.838032 MB/s
blocksize: 256KBbandwidth of mem write: 26442.183546 MB/s
blocksize: 512KBbandwidth of mem write: 21045.276116 MB/s
blocksize: 1024KBbandwidth of mem write: 20297.726417 MB/s
blocksize: 2048KBbandwidth of mem write: 18696.707992 MB/s
blocksize: 4096KBbandwidth of mem write: 11184.045260 MB/s
blocksize: 8192KBbandwidth of mem write: 9729.493477 MB/s
blocksize: 16384KBbandwidth of mem write: 9825.086593 MB/s
blocksize: 32768KBbandwidth of mem write: 9860.375542 MB/s
blocksize: 65536KBbandwidth of mem write: 9885.314901 MB/s
blocksize: 131072KBbandwidth of mem write: 9905.204101 MB/s
Bandwidth of Memory Copy
blocksize: 4KBbandwidth of mem copy: 23679.038039 MB/s
blocksize: 8KBbandwidth of mem copy: 23963.306187 MB/s
blocksize: 16KBbandwidth of mem copy: 22988.505747 MB/s
blocksize: 32KBbandwidth of mem copy: 21537.037816 MB/s
blocksize: 64KBbandwidth of mem copy: 21261.575516 MB/s
blocksize: 128KBbandwidth of mem copy: 20395.968609 MB/s
blocksize: 256KBbandwidth of mem copy: 13415.784509 MB/s
blocksize: 512KBbandwidth of mem copy: 13412.094461 MB/s
blocksize: 1024KBbandwidth of mem copy: 11705.265083 MB/s
blocksize: 2048KBbandwidth of mem copy: 7406.871609 MB/s
blocksize: 4096KBbandwidth of mem copy: 6369.426752 MB/s
blocksize: 8192KBbandwidth of mem copy: 6320.636508 MB/s
blocksize: 16384KBbandwidth of mem copy: 6156.065889 MB/s
blocksize: 32768KBbandwidth of mem copy: 6200.686678 MB/s
blocksize: 65536KBbandwidth of mem copy: 6317.322048 MB/s
blocksize: 131072KBbandwidth of mem copy: 6446.414182 MB/s
Pipe带宽
pipe带宽的瓶颈点主要在于写操作,在单次写数据超过64KB时,带宽出现拐点;
高版本的Linux Kernel中,Pipe的大小可以扩充为最大64KB,所以在单次写入pipe时以64KB为单位可以获得最大性能;
read()数据块大小与否对pipe的性能影响不大;
将pipe通信双方进程绑定到不同CPU上,避免在同一个CPU上调度导致不能时读写pipe。
huang@ThinkPad-X220:~/workspace/my_benchmark$ ./bench
loop cost: 0.001674usclock func cost: 0.031215us
benchmark on Bandwidth of Pipe
-------------------------------------
blocksize: 4KBbandwidth of pipe: 3239.563165 MB/s
blocksize: 8KBbandwidth of pipe: 3456.656765 MB/s
blocksize: 16KBbandwidth of pipe: 3642.831732 MB/s
blocksize: 32KBbandwidth of pipe: 3736.844410 MB/s
blocksize: 64KBbandwidth of pipe: 4000.437548 MB/s
blocksize: 128KBbandwidth of pipe: 2626.610851 MB/s
blocksize: 256KBbandwidth of pipe: 2579.241139 MB/s
blocksize: 512KBbandwidth of pipe: 2800.691421 MB/s
blocksize: 1024KBbandwidth of pipe: 2736.095079 MB/s
blocksize: 2048KBbandwidth of pipe: 2448.027234 MB/s
blocksize: 4096KBbandwidth of pipe: 2414.115029 MB/s
blocksize: 8192KBbandwidth of pipe: 1961.971475 MB/s
blocksize: 16384KBbandwidth of pipe: 1939.511486 MB/s
blocksize: 32768KBbandwidth of pipe: 2402.334769 MB/s
blocksize: 65536KBbandwidth of pipe: 2401.455883 MB/s
blocksize: 131072KBbandwidth of pipe: 2350.025244 MB/s
--------------end--------------------
Operating [System Entry]
system call 系统调用
getpid
严格说来不算系统调用,Linux对它的实现是缓存起来,不用系统调用即可获得自己的pid
getppid
获取父进程pid
write/read
read /dev/zero
从设备/dev/zero读一个字节
write /dev/null
向设备/dev/null写入一个字节
由于大部分操作系统都没有对这两个设备进行优化,所以向/dev/null写入一个字节可以测量纯粹的write调用,从用户进程切入内核态,调用VFS层,无缓冲区的写入设备。
benchmark on Operating system entry
===================================
getpid() cost: 0.006010us
getppid() cost: 0.049570us
write 1 byte to /dev/null cost: 0.086910us
read 1 byte from /dev/zero cost: 0.123290us
--------------end------------------
Process Creation
fork创建进程
fork + exit
创建子进程后立刻exit,用以衡量fork()这一过程用时
fork + exec
创建子进程后,子进程调用execve执行另外一个程序,该程序打印"hello world"后退出
swap_ucontext
采用ucontext保存当前处理器状态,切换到另一个ucontext状态,然后再切换回来
loop cost: 0.006350usclock func cost: 0.030847us
benchmark on Creating Process
===================================
fork+exit cost: 72.923700us
fork+exec cost: 306.169600us
swap_ucontext cost: 0.421900us
--------------end------------------
Memory Latency
目前常见的x64 Linux Server都具有 L1/L2/L3/Main Memory四级存储
L1 Cache Latency
本机具有32KB L1 Cache,采用可以全部填充进L1 Cache的内存块,可以统计计算出L1 Cache的访问速度;
由下表可以看出,内存块大小在32KB和64KB间延时明显增大,结合本机的32KB L1 Cache可以得出结论 L1 Cache的latency为1.338ns,约合4个cpu cycle(本机主频可达3GHz),这与intel芯片手册相符。
benchmark on Cache Latency
-------------------------------------
buflen = 1KBstride = 64L1 Cache Latency: 1.337800ns
buflen = 4KBstride = 64L1 Cache Latency: 1.337700ns
buflen = 16KBstride = 64L1 Cache Latency: 1.337700ns
buflen = 32KBstride = 64L1 Cache Latency: 1.338300ns
buflen = 48KBstride = 64L1 Cache Latency: 4.030900ns
buflen = 64KBstride = 64L1 Cache Latency: 4.042000ns
buflen = 192KBstride = 64L1 Cache Latency: 4.162100ns
buflen = 320KBstride = 64L1 Cache Latency: 4.429900ns
buflen = 448KBstride = 64L1 Cache Latency: 4.574200ns
buflen = 576KBstride = 64L1 Cache Latency: 4.589900ns
buflen = 832KBstride = 64L1 Cache Latency: 4.593400ns
=====================
buflen = 4KBstride = 1024L1 Cache Latency: 1.337700ns
buflen = 16KBstride = 1024L1 Cache Latency: 1.337700ns
buflen = 32KBstride = 1024L1 Cache Latency: 1.337800ns
buflen = 48KBstride = 1024L1 Cache Latency: 4.022400ns
buflen = 64KBstride = 1024L1 Cache Latency: 4.016500ns
buflen = 192KBstride = 1024L1 Cache Latency: 5.033400ns
buflen = 320KBstride = 1024L1 Cache Latency: 7.959200ns
buflen = 448KBstride = 1024L1 Cache Latency: 9.424500ns
buflen = 576KBstride = 1024L1 Cache Latency: 9.396900ns
buflen = 832KBstride = 1024L1 Cache Latency: 9.397800ns
=====================
buflen = 16KBstride = 4096L1 Cache Latency: 1.337800ns
buflen = 32KBstride = 4096L1 Cache Latency: 1.337800ns
buflen = 48KBstride = 4096L1 Cache Latency: 4.021300ns
buflen = 64KBstride = 4096L1 Cache Latency: 4.008100ns
buflen = 192KBstride = 4096L1 Cache Latency: 4.411000ns
buflen = 320KBstride = 4096L1 Cache Latency: 8.380600ns
buflen = 448KBstride = 4096L1 Cache Latency: 9.362800ns
buflen = 576KBstride = 4096L1 Cache Latency: 9.376100ns
buflen = 832KBstride = 4096L1 Cache Latency: 9.402000ns
=====================
buflen = 16KBstride = 8192L1 Cache Latency: 1.337900ns
buflen = 32KBstride = 8192L1 Cache Latency: 1.337700ns
buflen = 48KBstride = 8192L1 Cache Latency: 1.337700ns
buflen = 64KBstride = 8192L1 Cache Latency: 1.337800ns
buflen = 192KBstride = 8192L1 Cache Latency: 4.021700ns
buflen = 320KBstride = 8192L1 Cache Latency: 4.025100ns
buflen = 448KBstride = 8192L1 Cache Latency: 6.283900ns
buflen = 576KBstride = 8192L1 Cache Latency: 8.047600ns
buflen = 832KBstride = 8192L1 Cache Latency: 8.372600ns
=====================
--------------end--------------------
L2 Cache Latency
本机的L2 Cache是256KB,可以从下表中发现,数据块大小大于L1 Cache,小于L2 Cache时,访问速度稳定在4ns,约11个cycle,符合预期。
4KB的stride是非常好用的,对于Cache Line为64B的各级Cache,4KB可以在循环中频繁触发Cache替换,对于我用多种规模数据测试有很大帮助。比如测量L1 Cache时采用总共32KB数据,完全填充在L1 Cache中,测量L1的时延;而在数据块能完全填充在L2 Cache中却大于L1时,由于频繁的Cache miss,CPU需要从L2中将数据读出,因此此时可以认为测试的是L2 Cache。
benchmark on Cache Latency
-------------------------------------
buflen = 32KBstride = 4096L2 Cache Latency: 1.337800ns
buflen = 64KBstride = 4096L2 Cache Latency: 4.022700ns
buflen = 128KBstride = 4096L2 Cache Latency: 4.018500ns
buflen = 160KBstride = 4096L2 Cache Latency: 4.023900ns
buflen = 192KBstride = 4096L2 Cache Latency: 4.999700ns
buflen = 224KBstride = 4096L2 Cache Latency: 5.989300ns
buflen = 256KBstride = 4096L2 Cache Latency: 6.592200ns
buflen = 768KBstride = 4096L2 Cache Latency: 11.040700ns
buflen = 1280KBstride = 4096L2 Cache Latency: 10.422100ns
buflen = 1792KBstride = 4096L2 Cache Latency: 10.119700ns
buflen = 2304KBstride = 4096L2 Cache Latency: 11.141800ns
buflen = 3328KBstride = 4096L2 Cache Latency: 34.722800ns
buflen = 4352KBstride = 4096L2 Cache Latency: 48.896900ns
buflen = 6400KBstride = 4096L2 Cache Latency: 52.685000ns
=====================
buflen = 32KBstride = 8192L2 Cache Latency: 1.337700ns
buflen = 64KBstride = 8192L2 Cache Latency: 1.337700ns
buflen = 128KBstride = 8192L2 Cache Latency: 3.989400ns
buflen = 160KBstride = 8192L2 Cache Latency: 4.022500ns
buflen = 192KBstride = 8192L2 Cache Latency: 4.018800ns
buflen = 224KBstride = 8192L2 Cache Latency: 4.015900ns
buflen = 256KBstride = 8192L2 Cache Latency: 4.020300ns
buflen = 768KBstride = 8192L2 Cache Latency: 10.251700ns
buflen = 1280KBstride = 8192L2 Cache Latency: 9.881300ns
buflen = 1792KBstride = 8192L2 Cache Latency: 9.596500ns
buflen = 2304KBstride = 8192L2 Cache Latency: 9.562000ns
buflen = 3328KBstride = 8192L2 Cache Latency: 22.131600ns
buflen = 4352KBstride = 8192L2 Cache Latency: 47.491200ns
buflen = 6400KBstride = 8192L2 Cache Latency: 53.915800ns
=====================
buflen = 32KBstride = 16384L2 Cache Latency: 1.337700ns
buflen = 64KBstride = 16384L2 Cache Latency: 1.337700ns
buflen = 128KBstride = 16384L2 Cache Latency: 1.337800ns
buflen = 160KBstride = 16384L2 Cache Latency: 3.789700ns
buflen = 192KBstride = 16384L2 Cache Latency: 3.995500ns
buflen = 224KBstride = 16384L2 Cache Latency: 3.989400ns
buflen = 256KBstride = 16384L2 Cache Latency: 4.024100ns
buflen = 768KBstride = 16384L2 Cache Latency: 7.879300ns
buflen = 1280KBstride = 16384L2 Cache Latency: 8.572000ns
buflen = 1792KBstride = 16384L2 Cache Latency: 8.772700ns
buflen = 2304KBstride = 16384L2 Cache Latency: 8.826700ns
buflen = 3328KBstride = 16384L2 Cache Latency: 13.492600ns
buflen = 4352KBstride = 16384L2 Cache Latency: 49.614400ns
buflen = 6400KBstride = 16384L2 Cache Latency: 59.004800ns
=====================
--------------end--------------------
L3 Cache Latency
L3 Cache的测量不会显得那么准确,因此只有参考意义
本机L3 Cache为3072KB,从256KB到3072KB之间可以看到访存时间稳定在10~11ns之间
benchmark on Cache Latency
-------------------------------------
buflen = 32KBstride = 4096L2 Cache Latency: 1.337800ns
buflen = 64KBstride = 4096L2 Cache Latency: 4.009700ns
buflen = 128KBstride = 4096L2 Cache Latency: 4.021200ns
buflen = 160KBstride = 4096L2 Cache Latency: 4.024200ns
buflen = 192KBstride = 4096L2 Cache Latency: 4.983700ns
buflen = 224KBstride = 4096L2 Cache Latency: 5.923500ns
buflen = 256KBstride = 4096L2 Cache Latency: 6.543300ns
buflen = 512KBstride = 4096L2 Cache Latency: 11.435800ns
buflen = 768KBstride = 4096L2 Cache Latency: 11.596700ns
buflen = 1024KBstride = 4096L2 Cache Latency: 11.632400ns
buflen = 1280KBstride = 4096L2 Cache Latency: 11.639400ns
buflen = 1536KBstride = 4096L2 Cache Latency: 11.662100ns
buflen = 1792KBstride = 4096L2 Cache Latency: 12.375600ns
buflen = 2048KBstride = 4096L2 Cache Latency: 14.748800ns
buflen = 2560KBstride = 4096L2 Cache Latency: 21.878800ns
buflen = 3072KBstride = 4096L2 Cache Latency: 32.695100ns
buflen = 3584KBstride = 4096L2 Cache Latency: 43.491900ns
buflen = 4096KBstride = 4096L2 Cache Latency: 50.027200ns
buflen = 5120KBstride = 4096L2 Cache Latency: 53.826100ns
buflen = 6144KBstride = 4096L2 Cache Latency: 55.367800ns
buflen = 7168KBstride = 4096L2 Cache Latency: 55.354000ns
buflen = 8192KBstride = 4096L2 Cache Latency: 54.837900ns
=====================
buflen = 32KBstride = 8192L2 Cache Latency: 1.337800ns
buflen = 64KBstride = 8192L2 Cache Latency: 1.337800ns
buflen = 128KBstride = 8192L2 Cache Latency: 4.018900ns
buflen = 160KBstride = 8192L2 Cache Latency: 4.021400ns
buflen = 192KBstride = 8192L2 Cache Latency: 4.022200ns
buflen = 224KBstride = 8192L2 Cache Latency: 4.022600ns
buflen = 256KBstride = 8192L2 Cache Latency: 4.016900ns
buflen = 512KBstride = 8192L2 Cache Latency: 8.304500ns
buflen = 768KBstride = 8192L2 Cache Latency: 10.898900ns
buflen = 1024KBstride = 8192L2 Cache Latency: 11.564300ns
buflen = 1280KBstride = 8192L2 Cache Latency: 11.574900ns
buflen = 1536KBstride = 8192L2 Cache Latency: 11.617000ns
buflen = 1792KBstride = 8192L2 Cache Latency: 11.363700ns
buflen = 2048KBstride = 8192L2 Cache Latency: 11.030500ns
buflen = 2560KBstride = 8192L2 Cache Latency: 10.727900ns
buflen = 3072KBstride = 8192L2 Cache Latency: 11.567700ns
buflen = 3584KBstride = 8192L2 Cache Latency: 19.558400ns
buflen = 4096KBstride = 8192L2 Cache Latency: 31.960000ns
buflen = 5120KBstride = 8192L2 Cache Latency: 46.668400ns
buflen = 6144KBstride = 8192L2 Cache Latency: 49.114400ns
buflen = 7168KBstride = 8192L2 Cache Latency: 52.456800ns
buflen = 8192KBstride = 8192L2 Cache Latency: 52.631700ns
=====================
buflen = 32KBstride = 16384L2 Cache Latency: 1.337800ns
buflen = 64KBstride = 16384L2 Cache Latency: 1.337800ns
buflen = 128KBstride = 16384L2 Cache Latency: 1.337800ns
buflen = 160KBstride = 16384L2 Cache Latency: 3.898100ns
buflen = 192KBstride = 16384L2 Cache Latency: 4.022800ns
buflen = 224KBstride = 16384L2 Cache Latency: 4.021000ns
buflen = 256KBstride = 16384L2 Cache Latency: 4.021900ns
buflen = 512KBstride = 16384L2 Cache Latency: 6.362800ns
buflen = 768KBstride = 16384L2 Cache Latency: 7.399500ns
buflen = 1024KBstride = 16384L2 Cache Latency: 8.770800ns
buflen = 1280KBstride = 16384L2 Cache Latency: 10.223000ns
buflen = 1536KBstride = 16384L2 Cache Latency: 11.045300ns
buflen = 1792KBstride = 16384L2 Cache Latency: 11.089800ns
buflen = 2048KBstride = 16384L2 Cache Latency: 10.882700ns
buflen = 2560KBstride = 16384L2 Cache Latency: 10.589700ns
buflen = 3072KBstride = 16384L2 Cache Latency: 10.397600ns
buflen = 3584KBstride = 16384L2 Cache Latency: 10.390000ns
buflen = 4096KBstride = 16384L2 Cache Latency: 16.485300ns
buflen = 5120KBstride = 16384L2 Cache Latency: 44.056700ns
buflen = 6144KBstride = 16384L2 Cache Latency: 49.682600ns
buflen = 7168KBstride = 16384L2 Cache Latency: 55.243400ns
buflen = 8192KBstride = 16384L2 Cache Latency: 56.688400ns
=====================
--------------end--------------------
TODO
measure Cache line
measure TLB