1. 需求
基于kolla部署的云平台,资源使用耗尽,需要扩容,但由于kolla部署要求网卡一致,新增节点网卡却不一致,是一个棘手的问题。
解决该棘手问题最好的方法就是采用单网卡绑定模式,线上快速绑定快速替换。不过线上替换有风险,建议还是虚拟机关机状态下变更网络。
环境:
host os : Kylin Server SP1
cloud:基于kolla部署的openstack queens
kolla也允许网卡不一致,但配置非常复杂
2. 网卡绑定
2.1 bond模式简介
bond模式如下:
mod=0 ,(balance-rr)Round-robin policy(平衡抡循环策略)
mod=1, (active-backup)Active-backup policy(主-备份策略)
mod=2, (balance-xor)XOR policy(平衡策略)
mod=3, (broadcast)(广播策略)
mod=4, (802.3ad)IEEE 802.3ad Dynamic link aggregation(IEEE802.3ad 动态链接聚合)
mod=5, (balance-tlb)Adaptive transmit load balancing(适配器传输负载均衡)
mod=6, (balance-alb)Adaptive load balancing(适配器适应性负载均衡)
mode5和mode6不需要交换机端的设置,网卡能自动聚合;mode4需要支持802.3ad。mode0,mode2和mode3理论上需要静态聚合方式;mode 1 (active-backup) 主备模式(常用),即同时只有1块网卡在工作。mode 1刚好满足我们的需求,我们后续bond模式就选择mode 1。
2.2 管理网
管理网络我们采用bond0,写成脚本,然后执行替换,脚本如下:
nmcli connection add type bond con-name bond0 ifname bond0 mode active-backup
nmcli connection delete enp13s0f0
nmcli connection add type bond-slave con-name enp13s0f0 ifname enp13s0f0 master bond0
nmcli connection up enp13s0f0
nmcli connection modify bond0 ipv4.addresses 192.168.1.11/24 ipv4.gateway 192.168.1.254 ipv4.method manual
nmcli connection up bond0
ip addr flush enp13s0f0
nmcli connection命令参数:
connection表示要对连接进行操作
add 表示添加连接
type 类型
con-name 连接的名字
ifname 网卡名
modify表示要修改连接
ipv4后边可以有以下内容:
method, dns, dns-search, dns-options, dns-priority, addresses, gateway, routes, route-metric, ignore-auto-routes, ignore-auto-dns, dhcp-hostname, dhcp-send-hostname, never-default, may-fail, dad-timeout, dhcp-timeout, dhcp-client-id, dhcp-fqdn
另附上交互式修改 IP:
# nmcli connection edit bond0
nmcli> goto ipv4.addresses
nmcli ipv4.addresses> change
Edit 'addresses' value: 192.168.1.11/24
Do you also want to set 'ipv4.method' to 'manual'? [yes]: yes
nmcli ipv4.addresses> back
nmcli ipv4> save
nmcli ipv4> activate
nmcli ipv4> quit
2.3 业务网
2.3.1 VLAN网络
brctl delif brqb630e898-03 enp13s0f2.101
nmcli connection down enp13s0f2.101
nmcli connection delete enp13s0f2.101
nmcli connection add type bond con-name bond2 ifname bond2 mode active-backup
nmcli connection add type bond-slave con-name enp13s0f2 ifname enp13s0f2 master bond2
nmcli connection up enp13s0f2
nmcli connection modify bond2 ipv4.addresses 0.0.0.0/32 ipv4.method manual
nmcli connection up bond2
nmcli connection add type vlan con-name bond2.101 ifname bond2.101 dev bond2 id 101
brctl addif brqb630e898-03 bond2.101
若有多个vlan,则需要添加多个vlan标签
2.3.2 FLAT网络
这里需要处理下桥的绑定和解绑,这里采用的linuxbridge
brctl delif brqb630e898-02 enp13s0f1
nmcli connection down enp13s0f1
nmcli connection delete enp13s0f1
nmcli connection add type bond con-name bond1 ifname bond1 mode active-backup
nmcli connection add type bond-slave con-name enp13s0f1 ifname enp13s0f1 master bond1
nmcli connection up enp13s0f1
nmcli connection modify bond1 ipv4.addresses 0.0.0.0/32 ipv4.method manual
nmcli connection up bond1
brctl addif brqb630e898-02 bond1
2.4 网络常用操作
2.4.1 nmcli device
nmcli device可以查看设备状态
2.4.2 nmcli connection
- 配置静态IP
创建ens32网卡配置信息,包含:指定永久静态IP、网关、并ifup启动
nmcli connection add con-name ens32 ifname ens32 autoconnect yes type ethernet ipv4.addresses 192.168.2.211/24 ipv4.gateway 192.168.2.254 ipv4.dns 223.5.5.5 ipv4.method manual
先添加后配置地址
nmcli connection add type ethernet con-name ens35 ifname ens35
nmcli connection modify ens35 ipv4.addresses 10.13.4.51/24 ipv4.gateway 10.13.4.254 ipv4.dns 223.5.5.5 ipv4.method manual
nmcli connection up ens35
- 配置动态 IP:
创建 ens33 网卡配置信息,指定动态获取IP,并ifup启动
nmcli connection add type ethernet con-name ens33 ifname ens33 ipv4.method auto
- VLAN 标记
nmcli connection add type vlan con-name enp13s0f1.101 ifname enp13s0f1.101 dev enp13s0f1 id 101
2.4.3 删除bond网卡
nmcli connection down bond2
nmcli connection delete bond2
nmcli connection delete enp13s0f2
或者
echo -bond2 > /sys/class/net/bonding_masters
注:echo 后面的 - 是删除设备,+ 是增加设备
3. 云平台变更及扩容
3.1 globals.yml
修改以下参数:
network_interface: "bond0"
neutron_external_interface: "bond1"
neutron_vlan_interface: "bond2"
3.2 multinode
添加需要扩容的计算节点,参数如下:
[external-compute]
...
compute3
compute4
3.3 linuxbridge_agent.ini
修改已部署的linuxbridge_agent.int,/etc/kolla/neutron-linuxbridge-agent/linuxbridge_agent.ini
[linux_bridge]
physical_interface_mappings = physnet1:bond2,external:bond1
3.4 数据库的修改
登录数据容器,导出neutron数据库,如下:
mysqldump -u root -pPASSWORD --databases neutron > neutron.sql
导出整个数据库如下:
mysqldump -u root -pPASSWORD --all-databases> all-databases.sql
导出单个表如下:
mysqldump -u root -pPASSWORD --databases neutron --tables agents > neutron-agents.sql
停掉所有neutron-linuxbridge-agent服务
通过vim neutron.sql,然后替换掉相应的网卡,然后删除neutron再导入:
drop database neutron;
mysql -u root -pPASSWORD < neutron.sql
重新启动所有neutron-linuxbridge-agent服务
3.5 计算节点扩容
kolla-ansible -i multinode deploy --limit compute3,compute4
4. 参考方案
https://access.redhat.com/documentation/zh-cn/red_hat_enterprise_linux/7/html/networking_guide/sec-configure_802_1q_vlan_tagging_using_the_command_line_tool_nmcli
https://ieevee.com/tech/2016/07/19/nmcli.html
https://www.cnblogs.com/feifeifeisir/p/12302209.html
https://cloud.tencent.com/developer/article/1833019
https://kev1nchan.vercel.app/posts/f4d9f803/
http://www.wallcopper.com/linux/3492.html
https://www.xxshell.com/2629.html
https://blog.51cto.com/u_15127637/4251233
https://developer.aliyun.com/article/909582
https://blog.csdn.net/xyj0808xyj/article/details/50558207