查看网卡型号
lspci
lspci -vvv
查看网卡信息
注意这里的rx1/tx1表示low latency模式:
转自:https://support.xilinx.com/s/question/0D54U00005TFyjxSAD/efvi-best-practice-for-low-latency?language=en_US
In terms of settings - our cards that support lowlatency mode ship in that mode by default. You can check this using “ethtool -i”, its should show ‘rx1’. If it shows rx0 then you’re in full-feature mode, nd if you don’t need those features then a small latency improvement (about 100-200ns) can be gained by disabling it. This can be done using: sfboot --firmware-variant=low-latency. (Sfboot is available in our utiltiies package, from https://www.xilinx.com/support/download/nic-software-and-drivers.html#linux )
查看网卡bad/drop/discard的数量
From Onload User Guide:
- port_rx_nodesc_drops:Number of packets dropped when there are no further descriptors in the rx ring buffer to receive them.
- port_rx_dp_di_dropped_packets: Number of packets dropped because filters indicate the packets should be dropped - this can happen when packets don’t match any filter or the matched filter indicates the packet should be dropped.
netstat -i 检查网卡丢包
netstat -i 显示的网卡丢包和ethtool -S 是一样的:
ethtool -g/-G 获取/设置 网卡读写环形缓冲区大小
ethtool -G enp130s0f0 rx 4096 #设置为4096
注意: 机器重启后可能会恢复到默认的1024, 需要重新设置。 我们发生过重启后恢复到了1024, 导致efvi从网卡获取数据时丢包的情况
numa node号
查看网卡的numa node, 然后查看cpu所在的numa node
NUMA架构显著的特点就是CPU访问本地内存快,访问远程内存慢。所以我们在NUMA架构下编写程序,要扬长避短,多核多线程编程中,我们要尽可能的利用CPU Core的亲和性,将线程绑定到对应的CPU上,并且该线程从该CPU对应的本地内存上去申请内存,这样才能最大限度发挥NUMA架构的优势,达到比较好的处理性能。
简单来说,就是本地的处理器、本地的内存来处理本地的设备上产生的数据。如果有一个PCI设备(比如网卡)在Node0上,就用Node0上的核来处理该设备,处理该设备用到的数据结构和数据缓冲区都从Node0上分配。
参考: https://zhuanlan.zhihu.com/p/643610982
使用命令cat /etc/default/grub 查看核心隔离情况:
onload_stackdump lots
sfreport
参考资料:
【1】Onload_User_Guide:
https://www.xilinx.com/content/dam/xilinx/publications/solarflare/onload/enterprise-onload/SF-104474-CD-34_Onload_User_Guide.pdf
【2】efvi user guide
https://docs.xilinx.com/v/u/en-US/SF-114063-CD-ef_vi_User_Guide
【3】How can I detect packet drops due to ef_vi specifically on Solarflare XtremeScale X2522-25G Adapter?
https://support.xilinx.com/s/question/0D54U000076eThKSAU/how-can-i-detect-packet-drops-due-to-efvi-specifically-on-solarflare-xtremescale-x252225g-adapter?language=en_US
【4】multicast packet loss in ef_vi processes while onload processes are OK
https://support.xilinx.com/s/question/0D54U00005wU25ySAC/multicast-packet-loss-in-efvi-processes-while-onload-processes-are-ok?language=en_US
【5】efvi best practice for low latency
https://support.xilinx.com/s/question/0D54U00005TFyjxSAD/efvi-best-practice-for-low-latency?language=en_US
【6】UDP multicast drops
https://support.xilinx.com/s/question/0D52E00006hpSOoSAM/udp-multicast-drops?language=en_US
【7】Lost multicast packets troubleshooting guide
https://ref.onixs.biz/lost-multicast-packets-troubleshooting.html
【8】Learning Path of ef_vi and Onload
https://support.xilinx.com/s/question/0D54U00005Sh49sSAB/learning-path-of-efvi-and-onload?language=en_US