k8s之multus cni

最新推荐文章于 2024-03-24 09:31:54 发布

分享放大价值

最新推荐文章于 2024-03-24 09:31:54 发布

阅读量1.7k

点赞数 1

分类专栏： kubernetes 文章标签： k8s multus cni

本文链接：https://blog.csdn.net/fengcai_ke/article/details/125716785

版权

kubernetes 专栏收录该内容

44 篇文章 9 订阅

订阅专栏

通常情况下在k8s中，一个pod只有一个接口，用于集群网络中pod和pod通信，
而multus定义了一种crd(Kubernetes Network Custom Resource Definition)-NetworkAttachmentDefinition,可用来定义其他网络接口，使pod可以生成多个接口。
如下图，eth0为默认的集群网络中的接口，net0和net1是自定义的其他接口。

image.png

安装multus

安装multus前，k8s必须已经安装了一种cni用于连接pod集群网络，比如calico。
再安装multus，multus会使用之前的cni作为默认网络，将之前cni的配置放在/etc/cni/net.d/00-multus.conf。
再通过NetworkAttachmentDefinition定义其他的cni接口。
在pod的yaml的annotation中使用"k8s.v1.cni.cncf.io/networks" 引用，如果用逗号分开，同时添加多个接口。

参考官网，安装multus有两种方法，一种是手动安装(手动下载multus binary，手动创建sa，crd等)，另一种是通过daemonset自动安装。

下面采用自动安装方法

#下载源码
root@master:~/multus# git clone https://github.com/intel/multus-cni.git
#安装multus
root@master:~/multus# kubectl apply -f multus-cni-master/images/multus-daemonset.yml
customresourcedefinition.apiextensions.k8s.io/network-attachment-definitions.k8s.cni.cncf.io created
clusterrole.rbac.authorization.k8s.io/multus created
clusterrolebinding.rbac.authorization.k8s.io/multus created
serviceaccount/multus created
configmap/multus-cni-config created
daemonset.apps/kube-multus-ds-amd64 created
daemonset.apps/kube-multus-ds-ppc64le created

查看multus pod已经处于running状态

root@master:~/multus# kubectl get pod -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-5b644bc49c-vrlmw   1/1     Running   0          7h45m
calico-node-5fhft                          1/1     Running   0          7h45m
calico-node-8jpzr                          1/1     Running   0          7h45m
calico-node-p8wxx                          1/1     Running   0          7h45m
coredns-9d85f5447-4znmx                    1/1     Running   4          42d
coredns-9d85f5447-fh667                    1/1     Running   2          42d
etcd-master                                1/1     Running   8          184d
kube-apiserver-master                      1/1     Running   0          36h
kube-controller-manager-master             1/1     Running   8          184d
kube-multus-ds-amd64-469ls                 1/1     Running   0          2s
kube-multus-ds-amd64-bzkts                 1/1     Running   0          2s
kube-multus-ds-amd64-pj2p4                 1/1     Running   0          2s
kube-proxy-l4wn7                           1/1     Running   5          184d
kube-proxy-prhcm                           1/1     Running   5          184d
kube-proxy-psxqt                           1/1     Running   8          184d
kube-scheduler-master                      1/1     Running   8          184d

network-attachment-definitions.k8s.cni.cncf.io 为multus创建的crd，用于定义其他网络接口

root@master:~/multus# kubectl describe CustomResourceDefinition network-attachment-definitions.k8s.cni.cncf.io
Name:         network-attachment-definitions.k8s.cni.cncf.io
Namespace:
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"apiextensions.k8s.io/v1","kind":"CustomResourceDefinition","metadata":{"annotations":{},"name":"network-attachment-definiti...
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2020-08-16T21:28:15Z
  Generation:          1
  Resource Version:    12551106
  Self Link:           /apis/apiextensions.k8s.io/v1/customresourcedefinitions/network-attachment-definitions.k8s.cni.cncf.io
  UID:                 71df5215-ce14-41b3-bd6b-bf6bfa8198f5
Spec:
  Conversion:
    Strategy:  None
  Group:       k8s.cni.cncf.io
  Names:
    Kind:       NetworkAttachmentDefinition
    List Kind:  NetworkAttachmentDefinitionList
    Plural:     network-attachment-definitions
    Short Names:
      net-attach-def
    Singular:  network-attachment-definition
  Scope:       Namespaced
  Versions:
    Name:  v1
    Schema:
      openAPIV3Schema:
        Description:  NetworkAttachmentDefinition is a CRD schema specified by the Network Plumbing Working Group to express the intent for attaching pods to one or more logical or physical networks. More information available at: https://github.com/k8snetworkplumbingwg/multi-net-spec
        Properties:
          Spec:
            Description:  NetworkAttachmentDefinition spec defines the desired state of a network attachment
            Properties:
              Config:
                Description:  NetworkAttachmentDefinition config is a JSON-formatted CNI configuration
                Type:         string
            Type:             object
        Type:                 object
    Served:                   true
    Storage:                  true
Status:
  Accepted Names:
    Kind:       NetworkAttachmentDefinition
    List Kind:  NetworkAttachmentDefinitionList
    Plural:     network-attachment-definitions
    Short Names:
      net-attach-def
    Singular:  network-attachment-definition
  Conditions:
    Last Transition Time:  2020-08-16T21:28:15Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  2020-08-16T21:28:15Z
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
  Stored Versions:
    v1
Events:  <none>

使用multus创建多个接口

首先创建一个NetworkAttachmentDefinition，指定macvlan类型

cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "macvlan",
      "master": "ens3",
      "mode": "bridge",
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.1.0/24",
        "rangeStart": "192.168.1.200",
        "rangeEnd": "192.168.1.216",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ],
        "gateway": "192.168.1.1"
      }
    }'
EOF

root@master:~/multus# kubectl get net-attach-def
NAME           AGE
macvlan-conf   25s

作为对比，先创建一个默认的pod

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: samplepod

spec:
  containers:
  - name: samplepod
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine
EOF

如下为pod默认的网络接口。
root@master:~/multus# kubectl exec -it samplepod -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if46: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1440 qdisc noqueue state UP
    link/ether b2:95:21:e8:80:63 brd ff:ff:ff:ff:ff:ff
    inet 10.24.166.141/32 scope global eth0
       valid_lft forever preferred_lft forever

再使用macvlan-conf 创建一个pod

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
  containers:
  - name: samplepod
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine
EOF

查看pod网络接口，多出来的net1为macvlan接口

root@master:~/multus#  kubectl exec -it samplepod -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if43: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1440 qdisc noqueue state UP
    link/ether 66:5c:86:29:da:d2 brd ff:ff:ff:ff:ff:ff
    inet 10.24.166.139/32 scope global eth0
       valid_lft forever preferred_lft forever
5: net1@tunl0: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether ce:b5:43:8e:50:b2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.200/24 scope global net1
       valid_lft forever preferred_lft forever

可以指定多次macvlan-conf来创建更多接口

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf,macvlan-conf
spec:
  containers:
  - name: samplepod
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine
EOF

多出来的net1和net2即为macvlan接口

root@master:~/multus# kubectl exec -it samplepod -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if45: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1440 qdisc noqueue state UP
    link/ether 52:f8:ed:35:c0:77 brd ff:ff:ff:ff:ff:ff
    inet 10.24.166.140/32 scope global eth0
       valid_lft forever preferred_lft forever
5: net1@tunl0: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 6a:1a:1f:0f:0d:6e brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.201/24 scope global net1
       valid_lft forever preferred_lft forever
6: net2@tunl0: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 2a:6b:f3:08:d3:d2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.202/24 scope global net2
       valid_lft forever preferred_lft forever