Openstack 与 Ceph集群搭建(完结)： 配置Ceph作为Openstack后端存储

参考文章

Openstack 官方对接ceph文档
Openstack 与 Ceph集群搭建(上)： 规划与准备
Openstack 与 Ceph集群搭建(中)： Ceph部署
Openstack 与 Ceph集群搭建(下)： Openstack部署
Openstack 与 Ceph集群搭建(完结)： 配置Ceph作为Openstack后端存储

1. 前置任务

1.0 修改Openstack配置文件

/etc/kolla/globals.yml，该文件定义OpenStack所需要安装的服务。

kolla_base_distro: "ubuntu"

kolla_internal_vip_address: "172.16.250.253"

network_interface: "vlan1250"

neutron_external_interface: "bond0"


enable_haproxy: "yes"
enable_cinder: "yes"
enable_cinder_backup: "yes"
enable_neutron_vpnaas: "yes"
enable_neutron_sriov: "no"
enable_neutron_dvr: "yes"
enable_neutron_fwaas: "yes"
enable_neutron_qos: "yes"
enable_neutron_agent_ha: "yes"
enable_neutron_bgp_dragent: "yes"
enable_neutron_provider_networks: "yes"
enable_neutron_trunk: "yes"
enable_neutron_port_forwarding: "yes"
enable_skyline: "yes"


# Glance
ceph_glance_user: "glance"
ceph_glance_keyring: "client.{{ ceph_glance_user }}.keyring"
ceph_glance_pool_name: "glance-images"
# Cinder
ceph_cinder_user: "cinder"
ceph_cinder_keyring: "client.{{ ceph_cinder_user }}.keyring"
ceph_cinder_pool_name: "cinder-volumes"
ceph_cinder_backup_user: "cinder-backup"
ceph_cinder_backup_keyring: "client.{{ ceph_cinder_backup_user }}.keyring"
ceph_cinder_backup_pool_name: "cinder-backups"
# Nova
ceph_nova_keyring: "{{ ceph_cinder_keyring }}"
ceph_nova_user: "{{ ceph_cinder_user }}"
ceph_nova_pool_name: "nova-vms"

glance_backend_ceph: "yes"

cinder_backend_ceph: "yes"

cinder_volume_group: "cinder-volumes"

cinder_backup_driver: "ceph"

nova_backend_ceph: "yes"

nova_compute_virt_type: "kvm"

1.1. 完成搭建的Ceph集群

请参考Openstack 与 Ceph集群搭建(中)： Ceph部署

1.2. 完成存储pool的创建

ceph osd pool create glance-images 300
ceph osd pool create cinder-backups 300
ceph osd pool create cinder-volumes 300
ceph osd pool create nova-vms 300
rbd pool init glance-images
rbd pool init cinder-backups
rbd pool init cinder-volumes
rbd pool init nova-vms

1.3. 针对OpenStack服务提供的Ceph keyring

keyring是给Openstack各服务用来访问存储使用的。生成keyring方式是在ceph管理节点执行以下操作。

注意：以下命令请保证pool名称与前面创建的名称一致

1.3.1. 生成glance的keyring

ceph auth get-or-create client.glance mon 'allow rwx' osd 'allow class-read object_prefix rbd_children, allow rwx pool=glance-images' 

1.3.2. 生成Cinder的keyring

ceph auth get-or-create client.cinder mon 'allow rwx' osd 'allow class-read object_prefix rbd_children, allow rwx pool=cinder-volumes, allow rwx pool=nova-vms ,allow rwx pool=glance-images'

1.3.3. 生成Cinder-backup的Keyring

ceph auth get-or-create client.cinder-backup mon 'allow rwx' osd 'allow class-read object_prefix rbd_children, allow rwx pool=cinder-backups' 

1.3.4. 生成Nova的keyring

ceph auth get-or-create client.nova mon 'allow rwx' osd 'allow class-read object_prefix rbd_children, allow rwx pool=nova-vms'  

2. Glance对接Ceph

2.1 编辑globals.yml文件

glance_backend_ceph: "yes"
# Glance
ceph_glance_keyring: "ceph.client.glance.keyring"
ceph_glance_user: "glance"
ceph_glance_pool_name: "glance-images"

2.2 复制ceph.conf文件

创建文件夹

mkdir -p /etc/kolla/config/glance

将ceph控制节点上的配置文件/etc/ceph/ceph.conf复制到etc/kolla/config/glance/ceph.conf

[global]
        fsid = 37bdf718-428b-11ef-90f7-ef5ebd5ff9ae
        mon_host = 10.148.250.250,10.148.250.249,10.148.250.248                                  

注意：在上述存储配置中，自动生成的配置文件mon_host可能还有其他参数，手动修改为纯IP即可。

2.3 复制keyring文件

将前置任务中针对glance生成的keyring文件复制到/etc/kolla/config/glance/ceph.client.glance.keyring

2.4 修改glance配置文件

编辑或新增/etc/kolla/config/glance.conf

[GLOBAL]
show_image_direct_url = True

警告：show_image_direct_url can present a security risk if using more than just Ceph as Glance backend(s). Please see Glance show_image_direct_url

2.5 查看

3. Cinder对接Ceph

3.1 调整multinode文件配置

当使用外部存储时Openstack无需存储节点，但是Kolla-ansible有[storage]组，为了保证部署openstack不报错，需要设置control为[storage]成员，如下：

编辑multinode文件

[storage]
node-03

3.2 编辑globals.yml文件

cinder_backend_ceph: "yes"
# Cinder
ceph_cinder_keyring: "ceph.client.cinder.keyring"
ceph_cinder_user: "cinder"
ceph_cinder_pool_name: "cinder-volumes"
ceph_cinder_backup_keyring: "ceph.client.cinder-backup.keyring"
ceph_cinder_backup_user: "cinder-backup"
ceph_cinder_backup_pool_name: "cinder-backups"

3.3 复制cepf.conf文件

与Glance保持一致：将ceph控制节点上的配置文件复制到etc/kolla/config/cinder/ceph.conf

mkdir -p /etc/kolla/config/cinder/
mkdir -p /etc/kolla/config/cinder/cinder-volume
mkdir -p /etc/kolla/config/cinder/cinder-backup

3.4 复制keyring文件

复制置任务中针对Cinder生成的keyring，因为是从存储节点复制文件到OpenStack部署节点，故自行操作，目标路径如下：

/etc/kolla/config/cinder/cinder-volume/ceph.client.cinder.keyring

/etc/kolla/config/cinder/cinder-backup/ceph.client.cinder.keyring

/etc/kolla/config/cinder/cinder-backup/ceph.client.cinder-backup.keyring

注意：cinder-backup文件夹需要复制两个keyring，分别是cinder-backup.keyring、cinder.keyring

3.5 nova关联配置

因cinder配置后端ceph存储，那么nova也必须可以访问cinder volumes，编辑globals.yml文件。

ceph_nova_keyring: "{{ ceph_cinder_keyring }}"

复制ceph.client.cinder.keyring文件到/etc/kolla/config/nova/ceph.client.cinder-backup.keyring

如果没有/etc/kolla/config/nova/文件夹，新建即可。

3.6 zun关联配置

由于zun非必要组件，此处忽略。

3.7 查看

4. Nova对接Ceph

4.1 编辑globals.yml 文件

nova_backend_ceph: "yes"

# Nova
ceph_nova_keyring: "{{ ceph_cinder_keyring }}"
ceph_nova_user: "{{ ceph_cinder_user }}"
ceph_nova_pool_name: "nova-vms"

4.2 复制ceph.conf文件

将ceph控制节点上的配置文件复制到`etc/kolla/config/nova

4.3 复制keyring

复制置任务中针对nova生成的keyring，因为是从存储节点复制文件到OpenStack部署节点，故自行操作，目标路径如下：

/etc/kolla/config/nova/ceph.client.nova.keyring

5. 配置生效

kolla-ansible -i /etc/kolla/multinode reconfigure --tags glance nova cinder