K8S 网络插件详解 CNI Container Network Interface（容器网络接口）

cni 是什么?

CNI（Container Network Interface）是 CNCF 旗下的一个项目，由一组用于配置 Linux 容器的网络接口的规范和库组成，同时还包含了一些插件。CNI 仅关心容器创建时的网络分配，和当容器被删除时释放网络资源。通过此链接浏览该项目：https://github.com/containernetworking/cni。由于cni 被k8s的kubelet使用所以，就存在我们进一步了解的必要。

k8s 是怎么使用cni 的

k8s （kubernetes）使用cni 为容器创建网络，当下发deployment、ststateful、pod 的时候由调度器（kube-scheduler）调度到选择出来的计算节点上，然后pod的创建任务交给了对应计算节点的kubelet，当创建pod的时候k8s会通过cni 定义的规范回调实现cni规范的网络插件（比如 flannel、calico、ovs）。这些插件都实现了cni 向cni 库里面注册了自己的回调函数。在各个计算节点的/opt/cni/bin 放着这些插件如下图，同时cni配置文件指定了创建网络使用什么插件，配置文件存放在/etc/cni/net.d如下cni配置文件截图。

特别注意： 如果目录下存在多个配置文件是读取 文件名字排序的第一个配置文件。
通过各个节点放同样的插件和配置文件，k8s就知道使用什么网络插件，但是k8s并不关心具体网络是以什么形式实现。当pod 删除时也会再调用一次cni的网络回收命令（容器的命名空间也回收了）。

如何自定义cni 插件

自定义cni 插件，需要引入cni的依赖包实现skel.PluginMain 的方法中注册自己的cmdAdd 和cmdDel。废话不多说上demo。

package main

import (
	"encoding/json"
	"fmt"

	"github.com/containernetworking/cni/pkg/skel"
	"github.com/containernetworking/cni/pkg/types"
	current "github.com/containernetworking/cni/pkg/types/100"
	"github.com/containernetworking/cni/pkg/version"

	bv "github.com/containernetworking/plugins/pkg/utils/buildversion"
)

// PluginConf is whatever you expect your configuration json to be. This is whatever
// is passed in on stdin. Your plugin may wish to expose its functionality via
// runtime args, see CONVENTIONS.md in the CNI spec.
type PluginConf struct {
	// This embeds the standard NetConf structure which allows your plugin
	// to more easily parse standard fields like Name, Type, CNIVersion,
	// and PrevResult.
	types.NetConf

	RuntimeConfig *struct {
		SampleConfig map[string]interface{} `json:"sample"`
	} `json:"runtimeConfig"`

	// Add plugin-specifc flags here
	MyAwesomeFlag     bool   `json:"myAwesomeFlag"`
	AnotherAwesomeArg string `json:"anotherAwesomeArg"`
}

// parseConfig parses the supplied configuration (and prevResult) from stdin.
func parseConfig(stdin []byte) (*PluginConf, error) {
	conf := PluginConf{}

	if err := json.Unmarshal(stdin, &conf); err != nil {
		return nil, fmt.Errorf("failed to parse network configuration: %v", err)
	}

	// Parse previous result. This will parse, validate, and place the
	// previous result object into conf.PrevResult. If you need to modify
	// or inspect the PrevResult you will need to convert it to a concrete
	// versioned Result struct.
	if err := version.ParsePrevResult(&conf.NetConf); err != nil {
		return nil, fmt.Errorf("could not parse prevResult: %v", err)
	}
	// End previous result parsing

	// Do any validation here
	if conf.AnotherAwesomeArg == "" {
		return nil, fmt.Errorf("anotherAwesomeArg must be specified")
	}

	return &conf, nil
}

// cmdAdd is called for ADD requests
func cmdAdd(args *skel.CmdArgs) error {
	conf, err := parseConfig(args.StdinData)
	if err != nil {
		return err
	}

	// A plugin can be either an "originating" plugin or a "chained" plugin.
	// Originating plugins perform initial sandbox setup and do not require
	// any result from a previous plugin in the chain. A chained plugin
	// modifies sandbox configuration that was previously set up by an
	// originating plugin and may optionally require a PrevResult from
	// earlier plugins in the chain.

	// START chained plugin code
	if conf.PrevResult == nil {
		return fmt.Errorf("must be called as chained plugin")
	}

	// Convert the PrevResult to a concrete Result type that can be modified.
	prevResult, err := current.GetResult(conf.PrevResult)
	if err != nil {
		return fmt.Errorf("failed to convert prevResult: %v", err)
	}

	if len(prevResult.IPs) == 0 {
		return fmt.Errorf("got no container IPs")
	}

	// Pass the prevResult through this plugin to the next one
	result := prevResult

	// END chained plugin code

	// START originating plugin code
	// if conf.PrevResult != nil {
	//	return fmt.Errorf("must be called as the first plugin")
	// }

	// Generate some fake container IPs and add to the result
	// result := &current.Result{CNIVersion: current.ImplementedSpecVersion}
	// result.Interfaces = []*current.Interface{
	// 	{
	// 		Name:    "intf0",
	// 		Sandbox: args.Netns,
	// 		Mac:     "00:11:22:33:44:55",
	// 	},
	// }
	// result.IPs = []*current.IPConfig{
	// 	{
	// 		Address:   "1.2.3.4/24",
	// 		Gateway:   "1.2.3.1",
	// 		// Interface is an index into the Interfaces array
	// 		// of the Interface element this IP applies to
	// 		Interface: current.Int(0),
	// 	}
	// }
	// END originating plugin code

	// Implement your plugin here

	// Pass through the result for the next plugin
	return types.PrintResult(result, conf.CNIVersion)
}

// cmdDel is called for DELETE requests
func cmdDel(args *skel.CmdArgs) error {
	conf, err := parseConfig(args.StdinData)
	if err != nil {
		return err
	}
	_ = conf

	// Do your delete here

	return nil
}

func main() {
	// replace TODO with your plugin name
	skel.PluginMain(cmdAdd, cmdCheck, cmdDel, version.All, bv.BuildString("TODO"))
}

func cmdCheck(args *skel.CmdArgs) error {
	// TODO: implement
	return fmt.Errorf("not implemented")
}

如下代码为注册自己的接口逻辑给k8s cni，然后在对应的方法实现自己的网络逻辑。

func main() {
	// replace TODO with your plugin name
	skel.PluginMain(cmdAdd, cmdCheck, cmdDel, version.All, bv.BuildString("TODO"))
}
在这个方法注册自定义网络的创建
// 创建自己的网络 方法逻辑
func cmdAdd(args *skel.CmdArgs) error
// 回收自己的网络 方法逻辑
func cmdDel(args *skel.CmdArgs) error

multus 介绍

在Kubernetes中，网络是非常重要的一个领域。 Kubernetes本身不提供网络解决方案，但是提供了CNI规范。这些规范被许多CNI插件（例如WeaveNet，Flannel，Calico等）遵守。这些插件中任何一个都可以在集群上使用和部署以提供网络解决方案。该网络称为集群的默认网络。此默认网络使Pods不仅可以在同一节点上而且可以在群集中的各个节点之间相互通信。
随着发展，Kubernetes 缺乏支持VNF中多个网络接口的所需功能。传统上，网络功能使用多个网络接口分离控制，管理和控制用户/数据的网络平面。他们还用于支持不同的协议，满足不同的调整和配置要求。
为了解决这个需求，英特尔实现了MULTUS的CNI插件，其中提供了将多个接口添加到Pod的功能。这允许POD通过不同的接口连接到多个网络，并且每个接口都将使用其自己的CNI插件（CNI插件可以相同或不同。这完全取决于需求和实现）。

multus 工作原理

Kubernetes当前没有提供为POD添加额外的接口选项的规定，或支持多个CNI插件同时工作的规定，但是它确实提供了一种由api服务器扩展受支持的API的机制。使用“自定义资源定义”可以做到这一点。 MULTUS依赖于“自定义资源定义”来存储其他接口和CNI插件所需的信息。

multus 配置文件

{
    "cniVersion": "0.3.1",
    "name": "node-cni-network",
    "type": "multus",
    "kubeconfig": "/etc/kubernetes/node-kubeconfig.yaml",
    "confDir": "/etc/cni/multus/net.d",
    "cniDir": "/var/lib/cni/multus",
    "binDir": "/opt/cni/bin",
    "logFile": "/var/log/multus.log",
    "logLevel": "debug",
    "capabilities": {
        "portMappings": true
    },    
    "readinessindicatorfile": "",
    "namespaceIsolation": false,
"Note1":"NOTE: you can set clusterNetwork+defaultNetworks OR delegates!!",
    "clusterNetwork": "defaultCRD",
    "defaultNetworks": ["sidecarCRD", "flannel"],
    "systemNamespaces": ["kube-system", "admin"],
    "multusNamespace": "kube-system",
"Note2":"NOTE: If you use clusterNetwork/defaultNetworks, delegates is ignored",
    "delegates": [{
        "type": "weave-net",
        "hairpinMode": true
    }, {
        "type": "macvlan",
        ... (snip)
    }]
}

部署文件

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: network-attachment-definitions.k8s.cni.cncf.io
spec:
  group: k8s.cni.cncf.io
  scope: Namespaced
  names:
    plural: network-attachment-definitions
    singular: network-attachment-definition
    kind: NetworkAttachmentDefinition
    shortNames:
    - net-attach-def
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          description: 'NetworkAttachmentDefinition is a CRD schema specified by the Network Plumbing
            Working Group to express the intent for attaching pods to one or more logical or physical
            networks. More information available at: https://github.com/k8snetworkplumbingwg/multi-net-spec'
          type: object
          properties:
            apiVersion:
              description: 'APIVersion defines the versioned schema of this represen
                tation of an object. Servers should convert recognized schemas to the
                latest internal value, and may reject unrecognized values. More info:
                https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
              type: string
            kind:
              description: 'Kind is a string value representing the REST resource this
                object represents. Servers may infer this from the endpoint the client
                submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
              type: string
            metadata:
              type: object
            spec:
              description: 'NetworkAttachmentDefinition spec defines the desired state of a network attachment'
              type: object
              properties:
                config:
                  description: 'NetworkAttachmentDefinition config is a JSON-formatted CNI configuration'
                  type: string
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: multus
rules:
  - apiGroups: ["k8s.cni.cncf.io"]
    resources:
      - '*'
    verbs:
      - '*'
  - apiGroups:
      - ""
    resources:
      - pods
      - pods/status
    verbs:
      - get
      - update
  - apiGroups:
      - ""
      - events.k8s.io
    resources:
      - events
    verbs:
      - create
      - patch
      - update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: multus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: multus
subjects:
- kind: ServiceAccount
  name: multus
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: multus
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: multus-cni-config
  namespace: kube-system
  labels:
    tier: node
    app: multus
data:
  # NOTE: If you'd prefer to manually apply a configuration file, you may create one here.
  # In the case you'd like to customize the Multus installation, you should change the arguments to the Multus pod
  # change the "args" line below from
  # - "--multus-conf-file=auto"
  # to:
  # "--multus-conf-file=/tmp/multus-conf/70-multus.conf"
  # Additionally -- you should ensure that the name "70-multus.conf" is the alphabetically first name in the
  # /etc/cni/net.d/ directory on each node, otherwise, it will not be used by the Kubelet.
  cni-conf.json: |
    {
      "name": "multus-cni-network",
      "type": "multus",
      "capabilities": {
        "portMappings": true
      },
      "delegates": [
        {
          "cniVersion": "0.3.1",
          "name": "default-cni-network",
          "plugins": [
            {
              "type": "flannel",
              "name": "flannel.1",
                "delegate": {
                  "isDefaultGateway": true,
                  "hairpinMode": true
                }
              },
              {
                "type": "portmap",
                "capabilities": {
                  "portMappings": true
                }
              }
          ]
        }
      ],
      "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-multus-ds
  namespace: kube-system
  labels:
    tier: node
    app: multus
    name: multus
spec:
  selector:
    matchLabels:
      name: multus
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        tier: node
        app: multus
        name: multus
    spec:
      hostNetwork: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: multus
      containers:
      - name: kube-multus
        image: ghcr.io/k8snetworkplumbingwg/multus-cni:stable
        command: ["/entrypoint.sh"]
        args:
        - "--multus-conf-file=auto"
        - "--cni-version=0.3.1"
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: true
        volumeMounts:
        - name: cni
          mountPath: /host/etc/cni/net.d
        - name: cnibin
          mountPath: /host/opt/cni/bin
        - name: multus-cfg
          mountPath: /tmp/multus-conf
      terminationGracePeriodSeconds: 10
      volumes:
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: cnibin
          hostPath:
            path: /opt/cni/bin
        - name: multus-cfg
          configMap:
            name: multus-cni-config
            items:
            - key: cni-conf.json
              path: 70-multus.conf

参考：
https://jimmysong.io/kubernetes-handbook/concepts/flannel.html
https://github.com/containernetworking/cni
https://github.com/containernetworking/plugins/blob/f14ff6687ad98cb5d58254dbf754fbbb8d77dc99/plugins/sample/main.go#L34https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/configuration.md
https://github.com/k8snetworkplumbingwg/multus-cni
https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/quickstart.md
https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md
https://zhuanlan.zhihu.com/p/73863683