Kubernetes 部署contiv-vpp

简介

Contiv/VPP is a Kubernetes network plugin that uses FD.io VPP to provide network connectivity between PODs in a k8s cluster. It deploys itself as a set of system PODs in the kube-system namespace, some of them (contiv-ksr, contiv-crd, contiv-etcd) on the master node, and some of them (contiv-cni, contiv-vswitch, contiv-stn) on each node in the cluster.

Contiv/VPP is fully integrated with k8s via its components, and it automatically reprograms itself upon each change in the cluster via k8s API.

The main component of the solution, VPP, runs within the contiv-vswitch POD on each node in the cluster and provides POD-to-POD connectivity across the nodes in the cluster, as well as host-to-POD and outside-to-POD connectivity. While doing that, it leverages VPP’s fast data processing that runs completely in userspace and uses DPDK for fast access to the network IO layer.

Kubernetes services and policies are also reflected into VPP configuration, which means they are fully supported on VPP, without the need of forwarding packets into the Linux network stack (Kube Proxy), which makes them very effective and scalable.

Contiv-VPP架构

Contiv/VPP consists of several components, each of them packed and shipped as a Docker container. Three of them deploy on Kubernetes master node only:

• Contiv KSR
• Contiv CRD + netctl
• Contiv ETCD
  and the rest of them deploy on all nodes within the k8s cluster (including the master node):
• Contiv vSwitch
• Contiv CNI
• Contiv STN

Contiv-VPP架构图：

Contiv-VPP工作时序图：

K8S环境部署Contiv-VPP

1、配置Contiv-VPP运行环境：

使用Contiv-VPP提供的脚本setup-node.sh配置运行环境。使用DPDK项目提供的dpdk-setup.sh脚本配置大页内存。

[root@develop contiv]# ls
contiv-vpp.yaml  dpdk-setup.sh  pull-images.sh  setup-node.sh
[root@develop contiv]# bash dpdk-setup.sh 
------------------------------------------------------------------------------
 RTE_SDK exported as /data/k8s
------------------------------------------------------------------------------
----------------------------------------------------------
 Step 1: Select the DPDK environment to build
----------------------------------------------------------
[1] *

----------------------------------------------------------
 Step 2: Setup linuxapp environment
----------------------------------------------------------
[2] Insert IGB UIO module
[3] Insert VFIO module
[4] Insert KNI module
[5] Setup hugepage mappings for non-NUMA systems
[6] Setup hugepage mappings for NUMA systems
[7] Display current Ethernet/Crypto device settings
[8] Bind Ethernet/Crypto device to IGB UIO module
[9] Bind Ethernet/Crypto device to VFIO module
[10] Setup VFIO permissions

----------------------------------------------------------
 Step 3: Run test application for linuxapp environment
----------------------------------------------------------
[11] Run test application ($RTE_TARGET/app/test)
[12] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)

----------------------------------------------------------
 Step 4: Other tools
----------------------------------------------------------
[13] List hugepage info from /proc/meminfo

----------------------------------------------------------
 Step 5: Uninstall and system cleanup
----------------------------------------------------------
[14] Unbind devices from IGB UIO or VFIO driver
[15] Remove IGB UIO module
[16] Remove VFIO module
[17] Remove KNI module
[18] Remove hugepage mappings

[19] Exit Script

Option: 6

Removing currently reserved hugepages
Unmounting /mnt/huge and removing directory

  Input the number of 2048kB hugepages for each node
  Example: to have 128MB of hugepages available per node in a 2MB huge page system,
  enter '64' to reserve 64 * 2MB pages on each node
Number of pages for node0: 1024
Number of pages for node1: 1024
Reserving hugepages
Creating /mnt/huge and mounting as hugetlbfs

Press enter to continue ...
----------------------------------------------------------
 Step 1: Select the DPDK environment to build
----------------------------------------------------------
[1] *

----------------------------------------------------------
 Step 2: Setup linuxapp environment
----------------------------------------------------------
[2] Insert IGB UIO module
[3] Insert VFIO module
[4] Insert KNI module
[5] Setup hugepage mappings for non-NUMA systems
[6] Setup hugepage mappings for NUMA systems
[7] Display current Ethernet/Crypto device settings
[8] Bind Ethernet/Crypto device to IGB UIO module
[9] Bind Ethernet/Crypto device to VFIO module
[10] Setup VFIO permissions

----------------------------------------------------------
 Step 3: Run test application for linuxapp environment
----------------------------------------------------------
[11] Run test application ($RTE_TARGET/app/test)
[12] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)

----------------------------------------------------------
 Step 4: Other tools
----------------------------------------------------------
[13] List hugepage info from /proc/meminfo

----------------------------------------------------------
 Step 5: Uninstall and system cleanup
----------------------------------------------------------
[14] Unbind devices from IGB UIO or VFIO driver
[15] Remove IGB UIO module
[16] Remove VFIO module
[17] Remove KNI module
[18] Remove hugepage mappings

[19] Exit Script

Option: q

[root@develop contiv]# bash setup-node.sh 
#########################################
#   Contiv - VPP                        #
#########################################
Do you want to setup multinode cluster? [Y/n] y
PCI UIO driver is loaded
The following network devices were found
1) enp2s0f0 0000:02:00.0
2) enp2s0f1 0000:02:00.1
3) ens15f0 0000:04:00.0
4) ens15f1 0000:04:00.1
5) ens14f0 0000:81:00.0
6) ens14f1 0000:81:00.1
7) ens14f2 0000:81:00.2
8) ens14f3 0000:81:00.3
9) ens6f0 0000:83:00.0
10) ens6f1 0000:83:00.1
11) ens6f2 0000:83:00.2
12) ens6f3 0000:83:00.3
Select interface for node interconnect [1-12]:1
Device 'enp2s0f0' must be shutdown, do you want to proceed? [Y/n] y

unix {
   nodaemon
   cli-listen /run/vpp/cli.sock
   cli-no-pager
   poll-sleep-usec 100
}
nat {
   endpoint-dependent
   translation hash buckets 1048576
   translation hash memory 268435456
   user hash buckets 1024
   max translations per user 10000
}
acl-plugin {
   use tuple merge 0
}
api-trace {
    on
    nitems 5000
}
dpdk {
   dev 0000:02:00.0
}

File /etc/vpp/contiv-vswitch.conf will be modified, do you want to proceed? [Y/n] y
Do you want to pull the latest images? [Y/n] n
Do you want to install STN Daemon? [Y/n] n
Configuration of the node finished successfully.
[root@develop contiv]# 

部署K8S环境（Contos7系统）

[root@develop k8s]#kubeadm reset 
[root@develop k8s]#swapoff -a
[root@develop k8s]#kubeadm init --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.13.2 --pod-network-cidr=192.168.0.0/16 
[root@develop k8s]#mkdir -p $HOME/.kube
[root@develop k8s]#cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@develop k8s]#chown $(id -u):$(id -g) $HOME/.kube/config
[root@develop k8s]#kubectl taint nodes --all node-role.kubernetes.io/master-
[root@develop k8s]# kubectl get pod --all-namespaces
NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE
kube-system   coredns-78d4cf999f-zgxqb          0/1     Pending   0          108s
kube-system   coredns-78d4cf999f-zkwg8          0/1     Pending   0          108s
kube-system   etcd-develop                      1/1     Running   0          51s
kube-system   kube-apiserver-develop            1/1     Running   0          62s
kube-system   kube-controller-manager-develop   1/1     Running   0          68s
kube-system   kube-proxy-25zlj                  1/1     Running   0          108s
kube-system   kube-scheduler-develop            1/1     Running   0          56s
[root@develop k8s]# 

部署Contiv-VPP到K8S环境

1、下载Contiv-VPP编排文件

[root@develop contiv]# wget https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
--2019-02-08 15:15:38--  https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 24303 (24K) [text/plain]
Saving to: ‘contiv-vpp.yaml.1’

100%[==============================================================================================================================================================>] 24,303      88.0KB/s   in 0.3s   

2019-02-08 15:15:40 (88.0 KB/s) - ‘contiv-vpp.yaml.1’ saved [24303/24303]

[root@develop contiv]# 

2、使用contiv-vpp.yaml编排文件部署Contiv-VPP

[root@develop contiv]# kubectl apply -f contiv-vpp.yaml 
configmap/contiv-agent-cfg created
configmap/vpp-agent-cfg created
statefulset.apps/contiv-etcd created
service/contiv-etcd created
configmap/contiv-ksr-http-cfg created
configmap/contiv-etcd-cfg created
configmap/contiv-etcd-withcompact-cfg created
daemonset.extensions/contiv-vswitch created
daemonset.extensions/contiv-ksr created
clusterrole.rbac.authorization.k8s.io/contiv-ksr created
serviceaccount/contiv-ksr created
clusterrolebinding.rbac.authorization.k8s.io/contiv-ksr created
daemonset.extensions/contiv-crd created
clusterrole.rbac.authorization.k8s.io/contiv-crd created
serviceaccount/contiv-crd created
clusterrolebinding.rbac.authorization.k8s.io/contiv-crd created
configmap/contiv-crd-http-cfg created
[root@develop contiv]#

等待5分钟左右部署完成：

[root@develop contiv]# kubectl get pod --all-namespaces
NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE
kube-system   contiv-crd-qzv8j                  1/1     Running   0          2m56s
kube-system   contiv-etcd-0                     1/1     Running   0          2m57s
kube-system   contiv-ksr-tcxts                  1/1     Running   0          2m56s
kube-system   contiv-vswitch-kp9cd              1/1     Running   0          2m57s
kube-system   coredns-78d4cf999f-zgxqb          1/1     Running   0          6m49s
kube-system   coredns-78d4cf999f-zkwg8          1/1     Running   0          6m49s
kube-system   etcd-develop                      1/1     Running   0          5m52s
kube-system   kube-apiserver-develop            1/1     Running   0          6m3s
kube-system   kube-controller-manager-develop   1/1     Running   0          6m9s
kube-system   kube-proxy-25zlj                  1/1     Running   0          6m49s
kube-system   kube-scheduler-develop            1/1     Running   0          5m57s
[root@develop contiv]# kubectl get daemonset --all-namespaces
NAMESPACE     NAME             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
kube-system   contiv-crd       1         1         1       1            1           node-role.kubernetes.io/master=   3m8s
kube-system   contiv-ksr       1         1         1       1            1           node-role.kubernetes.io/master=   3m8s
kube-system   contiv-vswitch   1         1         1       1            1           <none>                            3m8s
kube-system   kube-proxy       1         1         1       1            1           <none>                            7m14s
[root@develop contiv]# kubectl get configmap --all-namespaces
NAMESPACE     NAME                                 DATA   AGE
kube-public   cluster-info                         2      7m25s
kube-system   contiv-agent-cfg                     3      3m18s
kube-system   contiv-crd-http-cfg                  2      3m18s
kube-system   contiv-etcd-cfg                      1      3m18s
kube-system   contiv-etcd-withcompact-cfg          1      3m18s
kube-system   contiv-ksr-http-cfg                  1      3m18s
kube-system   coredns                              1      7m25s
kube-system   extension-apiserver-authentication   6      7m28s
kube-system   kube-proxy                           2      7m24s
kube-system   kubeadm-config                       2      7m26s
kube-system   kubelet-config-1.13                  1      7m26s
kube-system   vpp-agent-cfg                        9      3m18s
[root@develop contiv]# vppctl show int 
              Name               Idx    State  MTU (L3/IP4/IP6/MPLS)     Counter          Count     
TenGigabitEthernet2/0/0           1      up          9000/0/0/0     
local0                            0     down          0/0/0/0       
loop0                             2      up          9000/0/0/0     
loop1                             4      up          9000/0/0/0     
tap0                              3      up          1450/0/0/0     rx packets                   712
                                                                    rx bytes                  275063
                                                                    tx packets                   709
                                                                    tx bytes                   57569
                                                                    drops                         11
                                                                    ip4                          701
                                                                    ip6                            9
tap1                              5      up          1450/0/0/0     rx packets                   407
                                                                    rx bytes                   33798
                                                                    tx packets                   350
                                                                    tx bytes                  137136
                                                                    drops                         49
                                                                    ip4                          398
                                                                    ip6                            9
tap2                              6      up          1450/0/0/0     rx packets                   398
                                                                    rx bytes                   33043
                                                                    tx packets                   351
                                                                    tx bytes                  137105
                                                                    drops                         49
                                                                    ip4                          389
                                                                    ip6                            9
[root@develop contiv]# 

注意事项：

1、如果先部署K8S环境，再部署Contiv-VPP，需要重启一下kubelet，不然容器识别不到宿主机的大页内存配置（虽然在容器内可以看到大页内存已经配置好），会导致contiv-vswitch-kp9cd容器启动失败(vpp启动失败），失败情况如下：

/usr/bin/vpp[46013]: dpdk: EAL init args: -c 2 -n 4 --huge-dir /run/vpp/hugepages --file-prefix vpp --master-lcore 1 --socket-mem 64,64
[New Thread 0x7ffea7a7a700 (LWP 46713)]
[New Thread 0x7ffea7279700 (LWP 46716)]

Thread 1 "vpp" hit Breakpoint 1, 0x00007fffb3f87ea0 in rte_eal_memory_init () from /usr/lib/vpp_plugins/dpdk_plugin.so
(gdb) s
Single stepping until exit from function rte_eal_memory_init,
which has no line number information.

Thread 1 "vpp" received signal SIGBUS, Bus error.
0x00007fffb3f8147e in ?? () from /usr/lib/vpp_plugins/dpdk_plugin.so
(gdb)