DPDK系列之十四：在kubernetes运行Intel的vhost-user-net-plugin插件以支持dpdk

最新推荐文章于 2023-03-22 08:44:29 发布

cloudvtech

最新推荐文章于 2023-03-22 08:44:29 发布

阅读量4.4k

点赞数 1

分类专栏： DPDK系列文章标签： DPDK OVS vhostuser Kubernetes 容器

本文链接：https://blog.csdn.net/cloudvtech/article/details/80505572

版权

kubernetes 同时被 3 个专栏收录

56 篇文章 11 订阅

订阅专栏

container

25 篇文章 0 订阅

订阅专栏

DPDK

24 篇文章 23 订阅

订阅专栏

一、前言

根据文章《容器云的数据通道加速方案概览》的描述，借助Intel提供的vhost-user-net-plugin CNI，可以为基于Kubernetes的容器配置使用virtio前端和vhost-user后端的数据通道加速方案，vhost-user后端port可以由OVS-DPDK或者VPP提供。

本文就将使用multus绑定flannel和vhost-user-net-plugin两种网络类型，为容器提供除了传统网络以外DPDK的支持。

multus的配置可以参照文章《kubernetes系列之十三：POD多网卡方案multus-cni之通过CRD给POD配置自由组合网络》。

转载自https://blog.csdn.net/cloudvtech

二、编译vhost-user-net-plugin CNI

2.1 参照git文档

github地址：https://github.com/intel/vhost-user-net-plugin

https://github.com/intel/vhost-user-net-plugin.git

git clone https://github.com/intel/vhost-user-net-plugin.git
cd vhost-user-net-plugin
./build

build出来的binary就在bin目录

[root@k8s-master vhost-user-net-plugin]# ls -l bin
总用量 3108
-rwxr-xr-x 1 root root 3179579 5月  29 07:56 vhostuser

2.2 安装vhost-user-net-plugin CNI

将vhost-user-net-plugin目录下bin/vhostuser文件和tests目录拷贝到node的/opt/cni/bin目录

2.3 node上的CNI配置文件

本测试只使用一个master和一个node

rm -f /etc/cni/net.d/*

创建新的multus CRD conf如下：

/etc/cni/net.d/10-multus.conf 
{  
  "name": "multus-cni-network",  
  "type": "multus",  
  "kubeconfig": "/etc/kubernetes/kubelet.conf"  
}

重启kubelet

systemctl restart kubelet

转载自https://blog.csdn.net/cloudvtech

三、编译并运行OVS-DPDK

参见文章《DPD系列之一：DPDK 17.08.1在CentoS 7.2.1511的安装》和文章《DPDK系列之七：OVS-DPDK的在CentOS安装和测试》。

转载自https://blog.csdn.net/cloudvtech

四、编译容器镜像

参见文章《DPDK系列之十三：容器基于OVS-DPDK的dpdkvhostuser端口的pktgen和testpmd数据包测试》。

这里只要使用testpmd容器就可以了。

转载自https://blog.csdn.net/cloudvtech

五、部署vhost-user-net-plugin网络

5.1 multus Network CRD yml文件

ovs-dpdk-network.yaml 
apiVersion: "kubernetes.com/v1"  
kind: Network  
metadata:  
  name: ovs-dpdk-networkobj  
plugin: vhostuser  
args: '[  
{
  "type": "vhostuser",
  "name": "vhostuser-network",
  "if0name": "net0",
  "vhost": {
    "vhost_tool": "/opt/cni/bin/tests/ovs-config.py"
  },
  "ipam": {
    "type": "host-local",
    "subnet": "192.168.166.0/24",
    "rangeStart": "192.168.166.220",
    "rangeEnd": "192.168.166.240",
    "routes": [
      {
        "dst": "0.0.0.0/0"
      }
    ],
    "gateway": "192.168.166.2"
  }
}
]'

使用vhost-user-net-plugin tests目录下ovs-config.py，这个tool的主要功能就是

使用vs-vsctl为POD创建一个dpdkvhostuser port并且将socket移动到/var/lib/cni/vhostuser/目录

5.2 部署vhost-user-net CNI网络

[root@k8s-master CDR]# kubectl get crd
NAME                      AGE
networks.kubernetes.com   21d
[root@k8s-master CDR]# kubectl get networks
NAME                  AGE
flannel-networkobj    21d
macvlan-networkobj    21d
ovs-dpdk-networkobj   3h
[root@k8s-master CDR]# kubectl describe networks ovs-dpdk-networkobj
Name:         ovs-dpdk-networkobj
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"kubernetes.com/v1","args":"[ { \"type\": \"vhostuser\", \"name\": \"vhostuser-network\", \"if0name\": \"net0\", \"vhost\": { \"vhost_too...
API Version:  kubernetes.com/v1
Args:         [ { "type": "vhostuser", "name": "vhostuser-network", "if0name": "net0", "vhost": { "vhost_tool": "/opt/cni/bin/tests/ovs-config.py" }, "ipam": { "type": "host-local", "subnet": "192.168.166.0/24", "rangeStart": "192.168.166.220", "rangeEnd": "192.168.166.240", "routes": [ { "dst": "0.0.0.0/0" } ], "gateway": "192.168.166.2" } } ]
Kind:         Network
Metadata:
  Cluster Name:        
  Creation Timestamp:  2018-05-29T12:20:31Z
  Generation:          1
  Resource Version:    356234
  Self Link:           /apis/kubernetes.com/v1/namespaces/default/networks/ovs-dpdk-networkobj
  UID:                 adfbbf3a-633a-11e8-8aca-000c29d3e746
Plugin:                vhostuser
Events:                <none>
[root@k8s-master CDR]#

5.3 POD配置

cat testpmd_pod_flannel_vhostuser.yml 
apiVersion: v1
kind: Pod
metadata:
  name: nginx-flannel-vhostuser
  annotations:
    networks: '[  
        { "name": "flannel-networkobj" },
        { "name": "ovs-dpdk-networkobj" }
    ]'
spec:
  containers:
      - name: ovs-dpdk
        image: dpdk_testpmd_1708
        imagePullPolicy: Never
        securityContext:
          privileged: true
        command: [ "/bin/bash", "-c", "--" ]
        args: [ "while true; do sleep 30; done;" ]
        volumeMounts:
         - name: hugepage
           mountPath: /mnt/huge
         - name: socketdir
           mountPath: /var/run/openvswitch
  volumes:
  - name: hugepage
    hostPath:
     path: /mnt/huge
  - name: socketdir
    hostPath:
     path: /var/lib/cni/vhostuser

部署两个POD

[root@k8s-master CDR]# kubectl apply -f testpmd_pod_flannel_vhostuser.yml 
pod "nginx-flannel-vhostuser" created
[root@k8s-master CDR]# kubectl apply -f testpmd_pod_flannel_vhostuser1.yml 
pod "nginx-flannel-vhostuser1" created

转载自https://blog.csdn.net/cloudvtech

六、查看部署结果

6.1 log可以看到建立了两个port

May 29 12:12:57 k8s-node1 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl add-port br0 70dc0863789f-net0 -- set Interface 70dc0863789f-net0 type=dpdkvhostuser
May 29 12:12:59 k8s-node1 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl add-port br0 d7157cc4f590-net0 -- set Interface d7157cc4f590-net0 type=dpdkvhostuser

2018-05-29T16:12:57.496Z|00185|netdev_dpdk|INFO|Socket /usr/local/var/run/openvswitch/70dc0863789f-net0 created for vhost-user port 70dc0863789f-net0
2018-05-29T16:12:57.496Z|00186|dpdk|INFO|VHOST_CONFIG: bind to /usr/local/var/run/openvswitch/70dc0863789f-net0
2018-05-29T16:12:57.497Z|00187|dpif_netdev|INFO|PMD thread on numa_id: 0, core id:  2 created.
2018-05-29T16:12:57.497Z|00188|dpif_netdev|INFO|There are 1 pmd threads on numa node 0
2018-05-29T16:12:57.578Z|00192|bridge|INFO|bridge br0: added interface 70dc0863789f-net0 on port 13
2018-05-29T16:12:59.288Z|00193|dpdk|INFO|VHOST_CONFIG: vhost-user server: socket created, fd: 51
2018-05-29T16:12:59.288Z|00194|netdev_dpdk|INFO|Socket /usr/local/var/run/openvswitch/d7157cc4f590-net0 created for vhost-user port d7157cc4f590-net0
2018-05-29T16:12:59.288Z|00195|dpdk|INFO|VHOST_CONFIG: bind to /usr/local/var/run/openvswitch/d7157cc4f590-net0
2018-05-29T16:12:59.288Z|00196|bridge|INFO|bridge br0: added interface d7157cc4f590-net0 on port 14

6.2 POD状态

[root@k8s-master CDR]# kubectl get pod -o wide
NAME                                READY     STATUS    RESTARTS   AGE       IP            NODE
nginx-flannel-vhostuser             1/1       Running   0          10m       10.244.1.14   k8s-node1
nginx-flannel-vhostuser1            1/1       Running   0          10m       10.244.1.13   k8s-node1

[root@k8s-master CDR]# ping 10.244.1.14
PING 10.244.1.14 (10.244.1.14) 56(84) bytes of data.
64 bytes from 10.244.1.14: icmp_seq=1 ttl=63 time=0.504 ms
64 bytes from 10.244.1.14: icmp_seq=2 ttl=63 time=0.466 ms
64 bytes from 10.244.1.14: icmp_seq=3 ttl=63 time=0.512 ms
^C
--- 10.244.1.14 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.466/0.494/0.512/0.020 ms

6.3 node上容器状态

[root@k8s-node1 ~]# docker ps | grep bash
4663c3f744ff        284b78f4e48f                 "/bin/bash -c -- '..."   3 minutes ago       Up 3 minutes                            k8s_ovs-dpdk_nginx-flannel-vhostuser_default_31243b57-635a-11e8-8aca-000c29d3e746_0
668df5db09c6        284b78f4e48f                 "/bin/bash -c -- '..."   3 minutes ago       Up 3 minutes                            k8s_ovs-dpdk_nginx-flannel-vhostuser1_default_2fefba60-635a-11e8-8aca-000c29d3e746_0

[root@k8s-node1 ~]# docker exec -it 4663c3f744ff bash
[root@nginx-flannel-vhostuser dpdk-17.05]# ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.244.1.14  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::c25:caff:fe9d:5b8d  prefixlen 64  scopeid 0x20<link>
        ether 0a:58:0a:f4:01:0e  txqueuelen 0  (Ethernet)
        RX packets 13  bytes 962 (962.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 648 (648.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

6.4 vhost socket

POD的网络容器pause状态如下：

d7157cc4f590        k8s.gcr.io/pause-amd64:3.1   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_nginx-flannel-vhostuser_default_31243b57-635a-11e8-8aca-000c29d3e746_0
70dc0863789f        k8s.gcr.io/pause-amd64:3.1   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_nginx-flannel-vhostuser1_default_2fefba60-635a-11e8-8aca-000c29d3e746_0

所以POD的vhost socket应该在/var/lib/cni/vhostuser目录，对应两个pause容器ID

[root@k8s-node1 ~]# ls -l /var/lib/cni/vhostuser/
总用量 0
drwx------ 2 root root 95 5月  29 12:12 70dc0863789fd13a61ad14fc333a50175992cc31727e204c14c6fa53f241885a
drwx------ 2 root root 95 5月  29 12:12 d7157cc4f590a84c0e460daf8062b44a1c4b80d4ec6213551915dd35527caf68

[root@k8s-node1 70dc0863789fd13a61ad14fc333a50175992cc31727e204c14c6fa53f241885a]# ls -l
总用量 8
srwxr-xr-x 1 root root   0 5月  29 12:12 70dc0863789f-net0
-rw-r--r-- 1 root root 127 5月  29 12:12 70dc0863789f-net0-ip4.conf
-rw-r--r-- 1 root root 156 5月  29 12:12 70dc0863789f-net0.json

[root@k8s-node1 70dc0863789fd13a61ad14fc333a50175992cc31727e204c14c6fa53f241885a]# cat 70dc0863789f-net0-ip4.conf 
{
	"ipAddr": "192.168.166.228/32",
	"macAddr": "8e:10:52:bf:e5:70",
	"gateway": "169.254.1.1",
	"gwMac": "00:00:00:00:00:00"
}
[root@k8s-node1 70dc0863789fd13a61ad14fc333a50175992cc31727e204c14c6fa53f241885a]# cat 70dc0863789f-net0.json 
{"vhostname":"70dc0863789f-net0","vhostmac":"00:00:00:00:00:00","ifname":"net0","ifmac":"8e:10:52:bf:e5:70","vhost_tool":"/opt/cni/bin/tests/ovs-config.py"}

6.5 OVS-DPDK port

/root/openvswitch-2.8.1/utilities/ovs-vsctl show
590c0cb0-706c-4d36-a9ed-611a824a5a9e
    Bridge "br0"
        Port "70dc0863789f-net0"
            Interface "70dc0863789f-net0"
                type: dpdkvhostuser
        Port "d7157cc4f590-net0"
            Interface "d7157cc4f590-net0"
                type: dpdkvhostuser
        Port "br0"
            Interface "br0"
                type: internal

转载自https://blog.csdn.net/cloudvtech

cloudvtech

关注

1
点赞
踩
7

收藏

觉得还不错? 一键收藏
0
评论
DPDK系列之十四：在kubernetes运行Intel的vhost-user-net-plugin插件以支持dpdk

一、前言根据文章《容器云的数据通道加速方案概览》的描述，借助Intel提供的vhost-user-net-plugin CNI，可以为基于Kubernetes的容器配置使用virtio前端和vhost-user后端的数据通道加速方案，vhost-user后端port可以由OVS-DPDK或者VPP提供。本文就将使用multus绑定flannel和vhost-user-net-plugin两种网络类...
复制链接

扫一扫

专栏目录