网卡软中断绑定–网卡中断亲和力

最新推荐文章于 2024-08-13 18:12:03 发布

GoRustNeverStop

最新推荐文章于 2024-08-13 18:12:03 发布

阅读量3.8k

点赞数

分类专栏： linux性能调优

linux性能调优专栏收录该内容

5 篇文章 0 订阅

订阅专栏

在这里先解释下啥是中断亲和力

中断亲和力是指将一个或多个中断源绑定到特定的 CPU 上运行

具体如何操作我这里就不说了,网络上一大片一大片的.我今天这篇文档是用来吐槽的..具体操作请见链接

http://liuxin1982.blog.51cto.com/4338970/1019825

这个链接的文章已经说的很清楚了在操作方面, 只要看过这个,跟着做一边就知道怎么去操作了,

我这里要吐槽的是这个文章的脚本

# setting up irq affinity according to /proc/interrupts
# 2008-11-25 Robert Olsson
# 2009-02-19 updated by Jesse Brandeburg
#
# > Dave Miller:
# (To get consistent naming in /proc/interrups)
# I would suggest that people use something like:
# char buf[IFNAMSIZ+6];
#
# sprintf(buf, "%s-%s-%d",
#         netdev->name,
#  (RX_INTERRUPT ? "rx" : "tx"),
#  queue->index);
#
#  Assuming a device with two RX and TX queues.
#  This script will assign: 
#
# eth0-rx-0  CPU0
# eth0-rx-1  CPU1
# eth0-tx-0  CPU0
# eth0-tx-1  CPU1
#
set_affinity()
{
    MASK=$((1<<$VEC))
    printf "%s mask=%X for /proc/irq/%d/smp_affinity\n" $DEV $MASK $IRQ
    printf "%X" $MASK > /proc/irq/$IRQ/smp_affinity
    #echo $DEV mask=$MASK for /proc/irq/$IRQ/smp_affinity
    #echo $MASK > /proc/irq/$IRQ/smp_affinity
}
if [ "$1" = "" ] ; then
 echo "Description:"
 echo "    This script attempts to bind each queue of a multi-queue NIC"
 echo "    to the same numbered core, ie tx0¦rx0 --> cpu0, tx1¦rx1 --> cpu1"
 echo "usage:"
 echo "    $0 eth0 [eth1 eth2 eth3]"
fi

# check for irqbalance running
IRQBALANCE_ON=`ps ax ¦ grep -v grep ¦ grep -q irqbalance; echo $?`
if [ "$IRQBALANCE_ON" == "0" ] ; then
 echo " WARNING: irqbalance is running and will"
 echo "          likely override this script's affinitization."
 echo "          Please stop the irqbalance service and/or execute"
 echo "          'killall irqbalance'"
fi
#
# Set up the desired devices.
#
for DEV in $*
do
  for DIR in rx tx TxRx
  do
     MAX=`grep $DEV-$DIR /proc/interrupts ¦ wc -l`
     if [ "$MAX" == "0" ] ; then
       MAX=`egrep -i "$DEV:.*$DIR" /proc/interrupts ¦ wc -l`
     fi
     if [ "$MAX" == "0" ] ; then
       echo no $DIR vectors found on $DEV
       continue
       #exit 1
     fi
     for VEC in `seq 0 1 $MAX`
     do
        IRQ=`cat /proc/interrupts ¦ grep -i $DEV-$DIR-$VEC"$"  ¦ cut  -d:  -f1 ¦ sed "s/ //g"`
        if [ -n  "$IRQ" ]; then
          set_affinity
        else
           IRQ=`cat /proc/interrupts ¦ egrep -i $DEV:v$VEC-$DIR"$"  ¦ cut  -d:  -f1 ¦ sed "s/ //g"`
           if [ -n  "$IRQ" ]; then
             set_affinity
           fi
        fi
     done
  done
done

“ linux network子系统的负责人David Miller提供了一个脚本 ”

据说这个是某个大牛写的脚本.. 看上去很好很强大,但是这里有一个大坑, 非常大的坑.. 今天本人就被它坑到了…

它的坑在

MASK=$((1<<$VEC))

这个位置, 这里是计算cpu掩码的, 比如网卡eth0 第一个队列eth0-0 那么这里的结果就是MASK=1 将0左移一位得到2进制0b10 十进制1 ,

这样看是很正常.. 因为根据网络上大片的文章显示计算cpu掩码就是第几个网卡队列就位移几为, 比如一个4核4队列网卡, 第4队列的cpu掩码为 1<<3 等于8 反推回去可以得到前面三个队列的cpu掩码, 然后将这个cpu掩码分别写入每个队列中断号的smp_affinity.类似这样

echo $((1<<3)) > /proc/irq/xx/smp_affinity 这样就将xx中断绑定到第4个cpu上,

这样看还是很符合规律的,但是假设我们的cpu是8核, 网卡队列也是8个呢..

根据$((1<<7))得到的cpu掩码将是128 ,然后将128写入xx中断的smp_affinity中,观察发现:尼玛说好的绑定到第8个cpu上的呢.. 怎么跑到第4个cpu上了?

33: 73905753 0 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-4
34: 0 5596608 0 0 0 0 0 0 IR-PCI-MSI-edge eth0-5
35: 0 0 5590023 0 0 0 0 0 IR-PCI-MSI-edge eth0-6
36: 0 0 0 5574803 0 0 0 0 IR-PCI-MSI-edge eth0-7

然后我又放狗..找到这么一句话”计算cpu的方法第一颗为00000001换算成16进制为1，第2颗cpu为00000010换算成16进制为2，依次类推得出，第8颗cpu为80″

这里有一个重点就是” 将2进制转换成16进制 ” 看到这里再看上面的脚本,尼玛这不是坑爹么… $((1<<n)) 直接是将2进制给转成10进制了哇.. 假如n = 0-3的话,还好. 结果还是正确的,但是一旦超过了3结果就开始偏差了..这样就直接导致我8核cpu8队列网卡,在绑定中断的时候产生重叠…. 即队列0-3绑定到cpu0-3,队列4-7绑定到cpu0-3.. 坑爹呢….

看着上面的脚本是某国外大牛所写,自己也认为合情合理,直接用在自己服务器上.. 结果却有这么一个大坑…

无奈自己重写了一遍,用py写的,shell的字符处理方面我用的实在不顺手..

#!/usr/bin/python
#coding=utf8

import os,re

def irq():
    #return irq number and network interface number
    #exp:
    #irq iface
    #61  0
    #62  1
    cpunum = os.popen("cat /proc/cpuinfo¦grep \"model name\"¦wc -l").read().replace("\n","")
    r = os.popen("cat /proc/interrupts ¦grep -E \"eth[0-9]-\"¦awk '{sub(\"eth[0-9]-\",\"\",$%s);print $1,$%s}'"%(int(cpunum)+3,int(cpunum)+3)).readlines()
    return [ (i.split()[0].split(":")[0],re.sub("[a-zA-Z]", "",i.replace("\n","").split()[1])) for i in r ]

def main(irq_queuenum):

    # if exists irqbalance process,will killed"
    irqbalance = int(os.popen("ps axu¦grep irqbalance¦grep -v grep¦wc -l").read())
    if irqbalance > 0:os.popen("pkill irqbalance");print "irqbalance is kill"

    # set irq_affinity
    for i in irq_queuenum:
        set_irq_affinity(i[0],hex(1 << int(i[1])).replace('0x',''))

def set_irq_affinity(IRQ,MASK):
    print 'echo %s to /proc/irq/%s/smp_affinity'%(MASK,IRQ)
    fp = open('/proc/irq/%s/smp_affinity'%IRQ,'w')
    fp.write(str(MASK))
    fp.close()

main(irq())