DPDK精准测量时间

---

DPDK实现方式

简单来说就是通过rdtsc指令来获取CPU启动起来的tick值，进行减法，然后结合频率来得到时间差。
对应到spdk里面的话就是spdk_get_ticks和spdk_get_ticks_hz. spdk_get_ticks最终会调用到rte_rdtsc，其实现如下：

//dpdk/lib/librte_eal/common/include/arch/x86/rte_cycles.h
static inline uint64_t
rte_rdtsc(void)
{
        union {
                uint64_t tsc_64;
                RTE_STD_C11
                struct {
                        uint32_t lo_32;
                        uint32_t hi_32;
                };
        } tsc;

#ifdef RTE_LIBRTE_EAL_VMWARE_TSC_MAP_SUPPORT
        if (unlikely(rte_cycles_vmware_tsc_map)) {
                /* ecx = 0x10000 corresponds to the physical TSC for VMware */
                asm volatile("rdpmc" :
                             "=a" (tsc.lo_32),
                             "=d" (tsc.hi_32) :
                             "c"(0x10000));
                return tsc.tsc_64;
        }
#endif

        asm volatile("rdtsc" :
                     "=a" (tsc.lo_32),
                     "=d" (tsc.hi_32));
        return tsc.tsc_64;
}

static inline uint64_t
rte_rdtsc_precise(void)
{
        rte_mb();
        return rte_rdtsc();
}

从以上代码可以看出，其实际就是调用了rdtsc指令。

spdk_get_ticks_hz最终会调用到get_tsc_freq_arch，其实现如下：

//dpdk/lib/librte_eal/common/arch/x86/rte_cycles.c

uint64_t
get_tsc_freq_arch(void)
{
        uint64_t tsc_hz = 0;
        uint32_t a, b, c, d, maxleaf;
        uint8_t mult, model;
        int32_t ret;

        /*
         * Time Stamp Counter and Nominal Core Crystal Clock
         * Information Leaf
         */
        maxleaf = __get_cpuid_max(0, NULL);

        if (maxleaf >= 0x15) {
                __cpuid(0x15, a, b, c, d);

                /* EBX : TSC/Crystal ratio, ECX : Crystal Hz */
                if (b && c)
                        return c * (b / a);
        }

        __cpuid(0x1, a, b, c, d);
        model = rte_cpu_get_model(a);

        if (check_model_wsm_nhm(model))
                mult = 133;
        else if ((c & bit_AVX) || check_model_gdm_dnv(model))
                mult = 100;
        else
                return 0;

        ret = rdmsr(0xCE, &tsc_hz);
        if (ret < 0)
                return 0;

        return ((tsc_hz >> 8) & 0xff) * mult * 1E6;
}
static int32_t
rdmsr(int msr, uint64_t *val)
{
#ifdef RTE_EXEC_ENV_LINUXAPP
        int fd;
        int ret;

        fd = open("/dev/cpu/0/msr", O_RDONLY);
        if (fd < 0)
                return fd;

        ret = pread(fd, val, sizeof(uint64_t), msr);

        close(fd);

        return ret;
#else
        RTE_SET_USED(msr);
        RTE_SET_USED(val);

        return -1;
#endif
}

实际操作

从以上代码可以看出，频率的获取是通过读取/dev/cpu/0/msr的0xCE偏移获取的，以下是对当前机器进行的实验。

$ hexdump -C /dev/cpu/0/msr -s 206 -n 8
000000ce  00 17 81 f3 2c 0a 07 00                           |....,...|
000000d6
$ python
Python 2.7.5 (default, Jul 13 2018, 13:06:57) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> mult=100
>>> tsc_hz=0x70a2cf3811700
>>> ((tsc_hz >> 8) & 0xff) * mult * 1E6;
2300000000.0
>>> 

参考链接

关于TSC 请参考以下文章。

1. https://en.wikipedia.org/wiki/Time_Stamp_Counter
2. https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/ia-32-ia-64-benchmark-code-execution-paper.pdf

值得注意的是以下两点：

1. TSC ticks是以CPU名义频率在跳动，而不是CPU的实际运行频率，所以CPU在power-saving模式下和高负载时候，TSC的速率不会改变。
2. TSC ticks记录的是实际时间的流逝，而不是真正CPU时钟。

With these processors, the TSC ticks at the processor’s nominal frequency, regardless of the actual CPU clock frequency due to turbo or power saving states. Hence TSC ticks are counting the passage of time, not the number of CPU clock cycles elapsed.