文章目录
本文描述基于 KubeOVN + KubeVirt 实现网卡热插的实践方法。
基于 containerd。
版本
kubeovn 基于 feature 提交
kubevirt 基于 commit 3bee1de7b4f8bab059817cf44d636d2521dec8a9
部署
kubeovn + kubevirt
kubeovn 基于社区 install.sh 脚本部署,然后 kube-ovn-controller 镜像替换
kubevirt 可以使用 镜像 quay.io/kubevirt/virt-operator:20230320_3bee1de7b
kubevirt 需要开启 feature-gate
---
apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata:
name: kubevirt
namespace: kubevirt
spec:
configuration:
developerConfiguration:
featureGates:
- HotplugNICs
dynamic-networks-controller
git clone https://github.com/k8snetworkplumbingwg/multus-dynamic-networks-controller.git
cd multus-dynamic-networks-controller
kubectl apply -f manifests/dynamic-networks-controller.yaml
multus
git clone https://github.com/k8snetworkplumbingwg/multus-cni.git && cd multus-cni
cat ./deployments/multus-daemonset-thick.yml | kubectl apply -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: net1
namespace: default
spec:
config: '{
"cniVersion": "0.3.1",
"name": "kube-ovn",
"plugins":[
{
"type":"kube-ovn",
"server_socket":"/run/openvswitch/kube-ovn-daemon.sock",
"provider": "net1.default.ovn"
},
{
"type":"portmap",
"capabilities":{
"portMappings":true
}
}
]
}'
使用
创建虚拟机
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
name: testvmi-nocloud
spec:
terminationGracePeriodSeconds: 30
domain:
resources:
requests:
memory: 1024M
devices:
disks:
- name: containerdisk
disk:
bus: virtio
- name: emptydisk
disk:
bus: virtio
- disk:
bus: virtio
name: cloudinitdisk
volumes:
- name: containerdisk
containerDisk:
image: kubevirt/fedora-cloud-container-disk-demo:latest
- name: emptydisk
emptyDisk:
capacity: "2Gi"
- name: cloudinitdisk
cloudInitNoCloud:
userData: |-
#cloud-config
password: fedora
chpasswd: { expire: False }
检查 pod
$ kubectl get pods -owide
virt-launcher-testvmi-nocloud-sl55d 2/2 Running 0 61s 10.16.0.49 node113 <none> 1/1
$ kubectl exec -it virt-launcher-testvmi-nocloud-sl55d ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default
link/ether 66:6d:26:df:ab:7c brd ff:ff:ff:ff:ff:ff
inet 10.16.0.49/16 brd 10.16.255.255 scope global eth0
valid_lft forever preferred_lft forever
3: k6t-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
link/ether 3e:4c:d9:14:6f:1f brd ff:ff:ff:ff:ff:ff
inet 169.254.75.10/32 scope global k6t-eth0
valid_lft forever preferred_lft forever
inet6 fe80::3c4c:d9ff:fe14:6f1f/64 scope link
valid_lft forever preferred_lft forever
4: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel master k6t-eth0 state UP group default qlen 1000
link/ether fa:5a:68:6c:06:50 brd ff:ff:ff:ff:ff:ff
inet6 fe80::f85a:68ff:fe6c:650/64 scope link
valid_lft forever preferred_lft forever
41: eth0-nic@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master k6t-eth0 state UP group default
link/ether 3e:4c:d9:14:6f:1f brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::3c4c:d9ff:fe14:6f1f/64 scope link
valid_lft forever preferred_lft forever
检查 虚拟机
$ virtctl console testvmi-nocloud
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UP group default qlen 1000
link/ether 00:00:00:3b:e5:7b brd ff:ff:ff:ff:ff:ff
altname enp1s0
inet 10.16.0.49/16 brd 10.16.255.255 scope global dynamic noprefixroute eth0
valid_lft 86313549sec preferred_lft 86313549sec
inet6 fe80::200:ff:fe3b:e57b/64 scope link
valid_lft forever preferred_lft forever
KubeOVN 添加第二张网卡
添加第二张网卡信息,选择上面的 network-attachment-definition 和 subnet
$ kubectl edit pod virt-launcher-testvmi-nocloud-sl55d
......
k8s.v1.cni.cncf.io/networks: '[{"interface":"net1","name":"net1","namespace":"default"}]'
net1.default.ovn.kubernetes.io/logical_switch: subnet-ipv4
......
检查,新增 net1 192.168.2.15/24
$ kubectl exec -it virt-launcher-testvmi-nocloud-sl55d ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default
link/ether 66:6d:26:df:ab:7c brd ff:ff:ff:ff:ff:ff
inet 10.16.0.49/16 brd 10.16.255.255 scope global eth0
valid_lft forever preferred_lft forever
3: k6t-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
link/ether 3e:4c:d9:14:6f:1f brd ff:ff:ff:ff:ff:ff
inet 169.254.75.10/32 scope global k6t-eth0
valid_lft forever preferred_lft forever
inet6 fe80::3c4c:d9ff:fe14:6f1f/64 scope link
valid_lft forever preferred_lft forever
4: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel master k6t-eth0 state UP group default qlen 1000
link/ether fa:5a:68:6c:06:50 brd ff:ff:ff:ff:ff:ff
inet6 fe80::f85a:68ff:fe6c:650/64 scope link
valid_lft forever preferred_lft forever
41: eth0-nic@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master k6t-eth0 state UP group default
link/ether 3e:4c:d9:14:6f:1f brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::3c4c:d9ff:fe14:6f1f/64 scope link
valid_lft forever preferred_lft forever
43: net1@if44: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
link/ether 00:00:00:d6:bb:bd brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.2.15/24 brd 192.168.2.255 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::200:ff:fed6:bbbd/64 scope link
valid_lft forever preferred_lft forever
virtctl 添加 interface
添加到 虚拟机内,命名 eth1
$ virtctl addinterface testvmi-nocloud --network-name default/net1 --iface-name eth1
检查 pod
$ kubectl exec -it virt-launcher-testvmi-nocloud-sl55d ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default
link/ether 66:6d:26:df:ab:7c brd ff:ff:ff:ff:ff:ff
inet 10.16.0.49/16 brd 10.16.255.255 scope global eth0
valid_lft forever preferred_lft forever
3: k6t-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
link/ether 3e:4c:d9:14:6f:1f brd ff:ff:ff:ff:ff:ff
inet 169.254.75.10/32 scope global k6t-eth0
valid_lft forever preferred_lft forever
inet6 fe80::3c4c:d9ff:fe14:6f1f/64 scope link
valid_lft forever preferred_lft forever
4: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel master k6t-eth0 state UP group default qlen 1000
link/ether fa:5a:68:6c:06:50 brd ff:ff:ff:ff:ff:ff
inet6 fe80::f85a:68ff:fe6c:650/64 scope link
valid_lft forever preferred_lft forever
5: net1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default
link/ether d6:d6:72:f1:12:5c brd ff:ff:ff:ff:ff:ff
inet 192.168.2.15/24 brd 192.168.2.255 scope global net1
valid_lft forever preferred_lft forever
6: k6t-net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
link/ether 06:73:79:b3:c0:6e brd ff:ff:ff:ff:ff:ff
inet 169.254.75.11/32 scope global k6t-net1
valid_lft forever preferred_lft forever
inet6 fe80::473:79ff:feb3:c06e/64 scope link
valid_lft forever preferred_lft forever
7: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel master k6t-net1 state UP group default qlen 1000
link/ether a2:7b:a6:7c:5a:81 brd ff:ff:ff:ff:ff:ff
inet6 fe80::a07b:a6ff:fe7c:5a81/64 scope link
valid_lft forever preferred_lft forever
41: eth0-nic@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master k6t-eth0 state UP group default
link/ether 3e:4c:d9:14:6f:1f brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::3c4c:d9ff:fe14:6f1f/64 scope link
valid_lft forever preferred_lft forever
43: net1-nic@if44: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master k6t-net1 state UP group default
link/ether 06:73:79:b3:c0:6e brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::473:79ff:feb3:c06e/64 scope link
valid_lft forever preferred_lft forever
检查虚拟机,新增网卡
[fedora@testvmi-nocloud ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UP group default qlen 1000
link/ether 00:00:00:3b:e5:7b brd ff:ff:ff:ff:ff:ff
altname enp1s0
inet 10.16.0.49/16 brd 10.16.255.255 scope global dynamic noprefixroute eth0
valid_lft 86312937sec preferred_lft 86312937sec
inet6 fe80::200:ff:fe3b:e57b/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UP group default qlen 1000
link/ether 00:00:00:d6:bb:bd brd ff:ff:ff:ff:ff:ff
altname enp8s0
inet 192.168.2.15/24 brd 192.168.2.255 scope global dynamic noprefixroute eth1
valid_lft 86313539sec preferred_lft 86313539sec
inet6 fe80::af9d:e259:1171:8b9/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[fedora@testvmi-nocloud ~]$ ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=254 time=2.88 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=254 time=1.34 ms
检查 libvirt xml,新增
......
<interface type='ethernet'>
<mac address='00:00:00:d6:bb:bd'/>
<target dev='tap1' managed='no'/>
<model type='virtio-non-transitional'/>
<mtu size='1400'/>
<alias name='ua-eth1'/>
<rom enabled='no'/>
<address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
</interface>
......
检查 vmi,新增
......
interfaces:
......
- bridge: {}
name: eth1
networks:
......
- multus:
networkName: default/net1
name: eth1
......
feature
热拔目前未实现,待 kubevirt 开发。