利用strace探测cp命令一次拷多少字节 续

目录

引出问题

用数据说话

1. dd命令测试

2. C语言程序测试

结论

---

引出问题

上节《利用strace探测cp命令一次拷多少字节》提到cp使用的是128K的buffer。确实如此吗？如果让读者实现cp命令，你会设置多大的buffer？不是buffer越大速度越快吗？

用数据说话

1. dd命令测试

首先cp源代码注释中有一个办法测试各个buffer下copy的速度：

for i in $(seq 0 10); do
     bs=$((1024*2**$i))
     printf "%7s=" $bs
     timeout --foreground -sINT 2 \
       dd bs=$bs if=/dev/zero of=/dev/null 2>&1 \
         | sed -n 's/.* \([0-9.]* [GM]B\/s\)/\1/p'
   done

在我机器上结果如下：

   1024=943 MB/s
   2048=1.8 GB/s
   4096=3.4 GB/s
   8192=6.0 GB/s
  16384=9.5 GB/s
  32768=12.5 GB/s
  65536=16.8 GB/s
 131072=19.1 GB/s
 262144=21.4 GB/s
 524288=22.0 GB/s
1048576=20.9 GB/s
2097152=20.5 GB/s
4194304=20.9 GB/s
8388608=19.9 GB/s
16777216=16.6 GB/s
33554432=5.1 GB/s
67108864=4.3 GB/s
134217728=4.1 GB/s
268435456=4.1 GB/s
536870912=3.7 GB/s
1073741824=3.2 GB/s

此处我把for循环扩大到了20。 可见 buffer大小=524288时速度最快，=131072时也挺快（我的配置是Intel(R) Xeon(R) Gold 6242R CPU @ 3.10GHz，4G DDR3）。它这个值要考虑到所有配置，cp内注释列出了几种配置组合：

 With the results shown for these systems:
   system #1: 1.7GHz pentium-m with 400MHz DDR2 RAM, arch=i686
   system #2: 2.1GHz i3-2310M with 1333MHz DDR3 RAM, arch=x86_64
   system #3: 3.2GHz i7-970 with 1333MHz DDR3, arch=x86_64
   system #4: 2.20GHz Xeon E5-2660 with 1333MHz DDR3, arch=x86_64
   system #5: 2.30GHz i7-3615QM with 1600MHz DDR3, arch=x86_64
   system #6: 1.30GHz i5-4250U with 1-channel 1600MHz DDR3, arch=x86_64
   system #7: 3.55GHz IBM,8231-E2B with 1066MHz DDR3, POWER7 revision 2.1

                per-system transfer rate (GB/s)
   blksize   #1    #2    #3    #4    #5    #6    #7
   ------------------------------------------------------------------------
      1024  .73   1.7   2.6   .64   1.0   2.5   1.3
      2048  1.3   3.0   4.4   1.2   2.0   4.4   2.5
      4096  2.4   5.1   6.5   2.3   3.7   7.4   4.8
      8192  3.5   7.3   8.5   4.0   6.0  10.4   9.2
     16384  3.9   9.4  10.1   6.3   8.3  13.3  16.8
     32768  5.2   9.9  11.1   8.1  10.7  13.2  28.0
     65536  5.3  11.2  12.0  10.6  12.8  16.1  41.4
    131072  5.5  11.8  12.3  12.1  14.0  16.7  54.8
    262144  5.7  11.6  12.5  12.3  14.7  16.4  40.0
    524288  5.7  11.4  12.5  12.1  14.7  15.5  34.5
   1048576  5.8  11.4  12.6  12.2  14.9  15.7  36.5

 所以不是buffer越大越好，131072是一个不错的选项。

2. C语言程序测试

按块写入1G大小的文件，直到写完。块大小从1G，1G/2，1G/4直到1.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>

#define FILENAME "AA"
#define BLOCK_SIZE 4096
#define FILE_SIZE (1024 * 1024 * 1024)

double timespec_diff(struct timespec start, struct timespec end) {
    return (double)(end.tv_sec - start.tv_sec) + (double)(end.tv_nsec - start.tv_nsec) / 1e9;
}


int main(int argc, char* argv[]) {

    FILE *fp;
    fp = fopen(FILENAME, "wb+");
    if (fp == NULL) {
        printf("Error opening file.\n");
        return 1;
    }

    struct timespec start_ts;
    struct timespec end_ts;
    int BUFFER_SIZE = FILE_SIZE;

    char* buffer;
    for(; BUFFER_SIZE>0; BUFFER_SIZE/=2){
        fseek(fp, 0L, SEEK_SET);
        
        clock_gettime(CLOCK_REALTIME, &start_ts);
        buffer = malloc(BUFFER_SIZE);
        memset(buffer, 'A', BUFFER_SIZE);

        off_t offset = 0;
        for (; offset < FILE_SIZE; offset += BUFFER_SIZE) {
            fwrite(buffer, BUFFER_SIZE, 1, fp);
        }
        free(buffer);
        clock_gettime(CLOCK_REALTIME, &end_ts);
        printf("BUFFER_SIZE=%d time=%.2f \n", BUFFER_SIZE, timespec_diff(start_ts,end_ts ));
    }

    fclose(fp);

    return 0;
}

输出如下：

 * BUFFER_SIZE=1073741824 time=1.89
 * BUFFER_SIZE=536870912 time=1.27
 * BUFFER_SIZE=268435456 time=1.21
 * BUFFER_SIZE=134217728 time=1.05
 * BUFFER_SIZE=67108864 time=1.05
 * BUFFER_SIZE=33554432 time=0.97
 * BUFFER_SIZE=16777216 time=1.04
 * BUFFER_SIZE=8388608 time=0.95
 * BUFFER_SIZE=4194304 time=0.93
 * BUFFER_SIZE=2097152 time=0.88
 * BUFFER_SIZE=1048576 time=0.90
 * BUFFER_SIZE=524288 time=0.86
 * BUFFER_SIZE=262144 time=0.87
 * BUFFER_SIZE=131072 time=0.88
 * BUFFER_SIZE=65536 time=0.95
 * BUFFER_SIZE=32768 time=0.98
 * BUFFER_SIZE=16384 time=1.01
 * BUFFER_SIZE=8192 time=1.02
 * BUFFER_SIZE=4096 time=0.96
 * BUFFER_SIZE=2048 time=0.95
 * BUFFER_SIZE=1024 time=1.00
 * BUFFER_SIZE=512 time=1.04
 * BUFFER_SIZE=256 time=1.11
 * BUFFER_SIZE=128 time=1.13
 * BUFFER_SIZE=64 time=1.27
 * BUFFER_SIZE=32 time=1.35
 * BUFFER_SIZE=16 time=2.01
 * BUFFER_SIZE=8 time=3.07
 

也是524288最快，131072第三。与dd命令测试结果一致。 

结论

写文件不是buffer越大越快，131072到524288个字节效率比较高。