OpenStack Ironic 裸金属的配置及使用

环境

当前OpenStack版本为 Rocky
操作系统为 CentOS 7.6
api节点即控制节点
conductor节点即裸金属计算节点
控制节点 node1、node2、node3
计算节点 node1、node2、node3、node4
裸金属服务节点 node4
虚拟IP vip

一、安装软件包

在所有节点上都安装 yum 仓库

yum install centos-release-openstack-rocky -y

在api节点上安装

//openstack-nova-serialproxy 用来将ironic console绑定到novaconsole的组件
yum install openstack-ironic-api python-ironicclient openstack-nova-serialproxy -y
执行
systemctl enable openstack-ironic-api
systemctl start openstack-ironic-api
systemctl enable openstack-nova-serialproxy
systemctl start openstack-nova-serialproxy
//安装 dashboard 插件，并修改horizon
yum install openstack-ironic-ui -y
cp /usr/share/openstack-dashboard/openstack_dashboard/local/enabled/_2200_ironic.py /usr/share/openstack-dashboard/openstack_dashboard/enabled/
systemctl restart httpd

在 conductor 节点上安装

//shellinabox 和 socat 二者选其一，用来连接裸金属串口
yum install openstack-ironic-conductor python-ironicclient shellinabox socat -y
执行
systemctl enable openstack-ironic-conductor
systemctl start openstack-ironic-conductor
systemctl enable shellinaboxd
systemctl start shellinaboxd

二、创建数据库

在api节点上执行 创建ironic数据库

mysql -uroot -p
create database ironic CHARACTER SET utf8;
grant all privileges on ironic.* to ‘ironic’@‘localhost’ identified by ‘密码’;
grant all privileges on ironic.* to ‘ironic’@’%’ identified by ‘密码’;

很有可能 ironic 用户不能登录 ironic数据库 @vip，我们需要先查看数据库表 mysql.user 中是否有对应的权限

select User, Host from mysql.user

查看是否每个节点都对应了 ironic 用户，若未则

grant all privileges on ironic.* to ‘ironic’@‘nodexxx’ identified by ‘ironic_db_passwd_for_xxx’;

三、创建 endpoint

api节点上执行
1、创建Bare Metal service 的用户 ironic，用于认证

openstack user create --domain default --password-prompt ironic

2、授权ironic用户到service项目

openstack role add --project service --user ironic admin

3、注册 bare metal service 到service

openstack service create --name ironic --description “Ironic baremetal provisioning service” baremetal

4、创建endpoint

openstack endpoint create --region RegionOne baremetal admin http://vip:6385
openstack endpoint create --region RegionOne baremetal public http://vip:6385
openstack endpoint create --region RegionOne baremetal internal http://vip:6385
//很有可能需要将vip换为vip的IP地址

四、配置api节点（API）

vim /etc/ironic/ironic.conf

>[DEFAULT]
transport_url = rabbit://RPC_USER:RPC_PASSWORD@RPC_HOST:RPC_PORT/
# RPC_USER 即你的rabbitmq 用户
# RPC_HOST 一般使用 VIP

配置 ironic-api 使用Identity service的认证。

[DEFAULT]
auth_strategy=keystone
[keystone_authtoken]
auth_type=password
www_authenticate_uri=http://PUBLIC_IDENTITY_IP:5000
auth_url=http://PRIVATE_IDENTITY_IP:5000
username=ironic
password=IRONIC_PASSWORD
project_name=service
project_domain_name=Default
user_domain_name=Default

[database]
connection=mysql+pymysql://ironic:IRONIC_DBPASSWORD@DB_IP/ironic?charset=utf8

填充数据库表

ironic-dbsync --config-file /etc/ironic/ironic.conf create_schema

五、配置 conductor 节点（conductor）

1、配置my_ip

[DEFAULT]
my_ip = HOST_IP # 若conductor节点有多个IP，这个IP应该设置为与裸金属同一网段的IP

2、配置DB路径

[database]
connection=mysql+pymysql://ironic:IRONIC_DBPASSWORD@DB_IP/ironic?charset=utf8

3、配置rabbitmq

[DEFAULT]
transport_url = rabbit://RPC_USER:RPC_PASSWORD@RPC_HOST:RPC_PORT/

4、配置访问其他OpenStack访问的凭据

[neutron]
auth_type = password
auth_url = http://vip:5000
username=ironic
password=ironic密码
project_name=service
project_domain_id=default
user_domain_id=default
#PEM encoded Certificate Authority to use when verifying HTTPs connections. (string value)
#cafile=/opt/stack/data/ca-bundle.pem
region_name = RegionOne
valid_interfaces=public
cleaning_network = 26835017-4d61-4ebf-b0f8-05ee7b73d5ec # 此二网络皆为外部网络
provisioning_network = 26835017-4d61-4ebf-b0f8-05ee7b73d5ec

[glance]
url = http://vip:9292
auth_url = http://vip:5000
auth_type = password
project_domain_name = default
user_domain_name = default
region_name = RegionOne
project_name = service
username = ironic
password = ironic密码

[cinder]
region_name = RegionOne
project_domain_id = default
user_domain_id = default
project_name = service
password = ironic密码
username = ironic
auth_url = http://vip:5000
auth_type = password

[service_catalog]
region_name = RegionOne
project_domain_id = default
user_domain_id = default
project_name = service
password = ironic密码
username = ironic
auth_url = http://vip:5000
auth_type = password

5、启用 drivers （hardware types）

[DEFAULT]
...
# driver
enabled_hardware_types = ipmi,redfish
# boot
enabled_boot_interfaces = pxe
# console
#enabled_console_interfaces = ipmitool-socat,no-console
enabled_console_interfaces = ipmitool-shellinabox,no-console
# deploy
enabled_deploy_interfaces = direct,iscsi
# inspect
enabled_inspect_interfaces = inspector
# management
enabled_management_interfaces = ipmitool,redfish
# power
enabled_power_interfaces = ipmitool,redfish
# raid
enabled_raid_interfaces = agent
# vendor
enabled_vendor_interfaces = ipmitool, no-vendor
# storage
enabled_storage_interfaces = cinder, noop
# network
enabled_network_interfaces = flat,neutron
default_network_interface = flat

六、配置 计算服务 （计算节点和控制节点）

1、ironic-conductor 节点 配置 /etc/nova/nova.conf

[default]
compute_driver = ironic.IronicDriver # 计算节点
# firewall driver是使用 nova-network 服务才有的，而Ironic使用neutron，所以设置为 Noop
firewall_driver = nova.virt.firewall.NoopFirewallDriver
# bare metal node不需要预留内存
reserved_host_memory_mb = 0

[serial_console]
enabled=true
port_range=10000:20000
base_url=ws://192.168.3.81:6083/
proxyclient_address=192.168.3.84

[filter_scheduler]
track_instance_changes = False

[scheduler] #（控制节点）
# 多长时间尝试去发现新的hosts已经被加入到cell中，若是负数，则不发现
discover_hosts_in_cells_interval = 120
# 若不设置此选项，也可以在新节点enroll注册后 通过 命令行执行 nova-manage cell_v2 discover_hosts --by-service 来手动发现

2、配置 控制节点 裸金属控制台 /etc/nova/nova.conf

[console]
allowed_origins = 192.168.3.84

[serial_console]
enabled=true
port_range=10000:20000
serialproxy_host=0.0.0.0
serialproxy_port=6083

3、重启服务

控制节点 systemctl restart openstack-nova-scheduler
计算节点 systemctl restart openstack-nova-compute

七、配置网络服务

1、网络节点 or 控制节点 上安装 networking baremetal ML2 mechanism driver 和 L2 agent

yum install -y python2-networking-baremetal python2-ironic-neutron-agent

2、修改 /etc/neutron/plugins/ml2/ml2_conf.ini

[ml2]
type_drivers = flat
tenant_network_types = flat
mechanism_drivers = openvswitch,baremetal

[ml2_type_flat]
flat_networks = physnet1

[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver # 这里是不一样的
enable_security_group = True

[ovs]
bridge_mappings = physnet1:br-eth2
# Replace eth2 with the interface on the neutron node which you
# are using to connect to the bare metal server

3、创建 /etc/neutron/plugins/ml2/ironic_neutron_agent.ini 并添加必要配置，NOTICE: 我将此仅配置到网络节点

[ironic]
project_domain_name = Default
project_name = service
user_domain_name = Default
password = ironic密码
username = ironic
auth_url = http://identity-server.example.com/indentity  # 如http://vip:5000/v3
auth_type = password
region_name = RegionOne

然后执行
chown root:neutron ironic_neutron_agent.ini

4、重启 neutron-server 服务以加载最新的配置

5、开启 ironic-neutron-agent 服务

systemctl enable ironic-neutron-agent
systemctl start ironic-neutron-agent
#也许不需要
systemctl restart openvswitch
systemctl restart neutron-openvswitch-agent
#看一看
openstack network agent list
#别看了，现在还没有ironic-neutron

6、配置 cleaning 网络（conductor 节点）
在裸金属服务配置文件中 通过 cleaning_network 选项配置 “清理网络”，使用之前创建的外部网络ID

[neutron]
cleaning_network = FLAT_NETWORK_UUID

重启 ironic-conductor服务

systemctl restart openstack-ironic-conductor

7、配置租户网络（conductor 节点）

[DEFAULT]
...
enabled_network_interfaces = noop,flat,neutron
default_network_interface = flat

8、配置provisioning network（conductor 节点）

[neutron]
...
provisioning_network = $PROVISION_UUID_OR_NAME # 我们使用和cleaning network 同一flat网络

9、配置DHCP（conductor 节点）

[dhcp]
dhcp_provider = neutron

八、配置 PXE

在 ironic-conductor 节点安装和配置
1、确保 tftp 根目录的存在

mkdir -p /tftpboot
chown -R ironic /tftpboot

2、安装 tftp server

yum -y install tftp-server xinetd
#用于验证 tftp生效的客户端 可以安装 tftp：yum -y install tftp

3、使用 xinetd 提供的 tftp server 配置，create or edit /etc/xinetd.d/tftp

service tftp
{
  protocol        = udp
  port            = 69
  socket_type     = dgram
  wait            = yes
  user            = root
  server          = /usr/sbin/in.tftpd
  server_args     = -v -v -v -v -v --map-file /tftpboot/map-file /tftpboot
  disable         = no
  # This is a workaround for Fedora, where TFTP will listen only on
  # IPv6 endpoint, if IPv4 flag is not used.
  flags           = IPv4
}

4、在 tftp 根目录 /tftpboot 下创建一个 map-ile

echo 're ^(/tftpboot/) /tftpboot/\2' > /tftpboot/map-file
echo 're ^/tftpboot/ /tftpboot/' >> /tftpboot/map-file
echo 're ^(^/) /tftpboot/\1' >> /tftpboot/map-file
echo 're ^([^/]) /tftpboot/\1' >> /tftpboot/map-file

5、UEFI PXE - Grub setup
为了在支持 UEFI 的裸金属节点上部署带有PXE的 instances，在 ironic-conductor节点上执行以下步骤来配置 PXE UEFI 环境
安装 grub2 和 shim 包

yum -y install grub2-efi shim

复制 grub 和 shim

sudo cp /boot/efi/EFI/centos/shim.efi /tftpboot/bootx64.efi
sudo cp /boot/efi/EFI/centos/grubx64.efi /tftpboot/grubx64.efi

创建 master grub.cfg

在/tftpboot/EFI/centos下创建grub.cfg
GRUB_DIR=/tftpboot/EFI/centos
sudo mkdir -p $GRUB_DIR

vim $GRUB_DIR/grub.cfg

cat grub.cfg

set default=master
set timeout=5
set hidden_timeout_quiet=false

menuentry "master"  {
configfile /tftpboot/$net_default_mac.conf
}

# chmod 644 $GRUB_DIR/grub.cfg

6、支持 Legacy BIOS - Syslinux setup
为了支持在 baremetal 上使用 legacy BIOS和PXE来部署 instance，在 Ironic-conductor 节点上执行以下步骤
安装syslinux 包

yum -y install syslinux-tftpboot

复制 PXE 镜像到 /tftpboot

cp /usr/share/syslinux/pxelinux.0 /tftpboot

若是 部署 whole disk image

cp /usr/share/syslinux/chain.c32 /tftpboot

若是 syslinux --version > 4.0 我们需要复制所有的 lib 到 /tftpboot
Ubuntu下是：
sudo cp /usr/lib/syslinux/modules//ldlinux. /tftpboot
centos7下不用管

7、vim /etc/ironic/ironic.conf of conductor

[ipmi]
retry_timeout = 60

[pxe]
ipxe_enabled = False
pxe_append_params = nofb nomodeset vga=normal console=ttyS0 systemd.journald.forward_to_console=yes
tftp_root=/tftpboot
tftp_server=192.168.3.66
# On the ironic-conductor node, directory where images are stored on disk. (string value)
#images_path = /var/lib/ironic/images/
images_path = /nfs/pxe_images/

# On the ironic-conductor node, directory where master
# instance images are stored on disk. Setting to the empty
# string disables image caching. (string value)
#instance_master_path = /var/lib/ironic/master_images
instance_master_path = /nfs/pxe_master_images

8、启动 xinetd

systemctl enable xinetd
systemctl restart xinetd
systemctl status xinetd
重启
systemctl restart openstack-ironic-conductor

九、上传镜像和创建 flavor

我们只需要 deploy 镜像，user 镜像就用我们平时的 raw 或 qcow2镜像，添加 deploy 镜像到 镜像 仓库

glance image-create --name deploy-vmlinuz --visibility public \
  --disk-format aki --container-format aki < coreos_production_pxe.vmlinuz
# 将上述步骤获得的镜像UUID存为 DEPLOY_VMLINUZ_UUID
glance image-create --name deploy-initrd --visibility public \
  --disk-format ari --container-format ari < coreos_production_pxe_image-oem.cpio.gz
# 将上述步骤获得的镜像UUID存为 DEPLOY_INITRD_UUID

修改pxe_master_images路径

mkdir /nfs/pxe_master_images
mkdir /nfs/pxe_images
cd /nfs
chmod 777 -R pxe_master_images
chmod 777 -R pxe_images

创建flavor

openstack flavor create --ram 64147 --vcpus 40 --disk 50 my-baremetal-flavor
openstack flavor set --property resources:CUSTOM_BAREMETAL_WITH_GPU=1 my-baremetal-flavor
openstack flavor set --property resources:VCPU=0 my-baremetal-flavor
openstack flavor set --property resources:MEMORY_MB=0 my-baremetal-flavor
openstack flavor set --property resources:DISK_GB=0 my-baremetal-flavor

十、注册 Enrollment Baremetal Node

首先查看用到的 IPMI 的所有 Ironic Driver

openstack baremetal driver show ipmi

打印出 IPMI 驱动程序的属性清单

openstack baremetal driver property list ipmi

创建 baremetal node

openstack baremetal node create --driver ipmi --name BM01 \
--deploy-interface iscsi \
--raid-interface agent \
--driver-info ipmi_username=ADMIN \
--driver-info ipmi_password=ADMIN \
--driver-info ipmi_address=192.168.0.116 \
--driver-info ipmi_port=623 \
--driver-info deploy_kernel=$DEPLOY_KERNEL_UUID \
--driver-info deploy_ramdisk=$DEPLOY_RAMDISK_UUID \
--driver-info cleaning_network=$CLEANING_NETWORK \
--driver-info provisioning_network=$PRO_NETWORK \
--driver-info ipmi_terminal_port=xxxx

若是 UEFI，记得

openstack baremetal node set $BAREMETAL_NODE_UUID --property capabilities='boot_modes:uefi'

与flavor结合

openstack baremetal node set $NODE_UUID --resource-class baremetal.with-GPU

创建 baremetal port

openstack baremetal port create ac:1f:6b:7c:42:8a --node $NODE_UUID

十一、创建 server

验证裸金属：

openstack baremetal node validate $BAREMETAL_NODE_UUID

控制台

openstack baremetal node set BM01 --console-interface ipmitool-socat
or
openstack baremetal node set BM01 --console-interface ipmitool-shellinabox

纳管

openstack baremetal node manage $BAREMETAL_NODE_UUID

提供

openstack baremetal node provide $BAREMETAL_NODE_UUID

启用控制台

openstack baremetal node console enable $BAREMETAL_NODE_UUID
openstack baremetal node console show $BAREMETAL_NODE_UUID

创建 Server
openstack server create --flavor $FLAVOR_UUID --nic net-id=$NETWORK_ID --image $CIRROS_ID test_BM01