opentsack-cinder调用ceph-rbd后台存储

openstack 调用ceph-rbd块存储

环境描述
openstack 版本 QUEENS
ceph 版本 Luminous 12.2.5

openstack和ceph集群安装部署步骤，后续文章介绍；
此环境建立在已有一套ceph-rbd和openstack运行正常的环境；

块存储调用关系逻辑图如下：

openstack-cinder
openstack需要先安装一套cinder服务，可以选择安装在controller节点或者额外选择一台机器单独作为cinder服务；
在此，额外选择一台服务器作为cinder服务；
1)配置yum源，安装openstack-cinder；
yum install centos-release-openstack-queens
yum install https://rdoproject.org/repos/rdo-release.rpm
yum upgrade
yum install python-openstackclient
yum install openstack-cinder targetcli python-keystone
2)安装完成后，直接修改cinder配置文件；
vim /etc/cinder/cinder.conf
配置项目，keystone/rabbitmq/controller指向controller节点；
注意账户和口令修改，uuid修改；
将conrtoller主机地址解析加入本地hosts文件，若有本地DNS解析，则忽略此项；
###去除注释行和空行###

[DEFAULT]
transport_url = rabbit://openstack:openstack_password@controller
auth_strategy = keystone
my_ip = 192.168.17.8
enabled_backends = ceph
glance_api_servers = http://controller:9292
glance_api_version = 2
[ceph]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
volume_backend_name = ceph
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
glance_api_version = 2
rbd_user = cinder
rbd_secret_uuid = *****
backup_driver = cinder.backup.drivers.ceph
backup_ceph_conf = /etc/ceph/ceph.conf
backup_ceph_pool = backups
backup_ceph_user = cinder-backup
backup_ceph_chunk_size = 134217728
backup_ceph_stripe_unit = 0
backup_ceph_stripe_count = 0
restore_discard_excess_bytes = true
log_dir = /var/log/cinder/cinder.log
[backend]
[backend_defaults]
[barbican]
[brcd_fabric_example]
[cisco_fabric_example]
[coordination]
[cors]
[database]
connection = mysql+pymysql://cinder:cinder_pass@controller/cinder
[fc-zone-manager]
[healthcheck]
[key_manager]
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = cinder
password = cinder_password
[matchmaker_redis]
[nova]
[lvm]
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp
[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
[oslo_messaging_zmq]
[oslo_middleware]
[oslo_policy]
[oslo_reports]
[oslo_versionedobjects]
[profiler]
[service_user]
[ssl]
[vault]

3)controller节点配置
###数据库创建账户，授权###
mysql -u root -p
CREATE DATABASE cinder;

GRANT ALL PRIVILEGES ON cinder. TO 'cinder'@'localhost' IDENTIFIED BY 'stack';
GRANT ALL PRIVILEGES ON cinder. TO 'cinder'@'%' IDENTIFIED BY 'stack';

###创建openstak接口，创建账户###

.admin-openrc

openstack user create --domain default --password-prompt cinder

openstack role add --project service --user cinder admin

openstack service create --name cinderv2 --description "OpenStack Block Storage" volumev2

openstack service create --name cinderv3 --description "OpenStack Block Storage" volumev3

openstack endpoint create --region RegionOne \
volumev2 public http://controller:8776/v2/%\(project_id\)s

openstack endpoint create --region RegionOne \
volumev2 internal http://controller:8776/v2/%\(project_id\)s

openstack endpoint create --region RegionOne \
volumev2 admin http://controller:8776/v2/%\(project_id\)s

openstack endpoint create --region RegionOne \
volumev3 public http://controller:8776/v3/%\(project_id\)s

openstack endpoint create --region RegionOne \
volumev3 internal http://controller:8776/v3/%\(project_id\)s

openstack endpoint create --region RegionOne \
volumev3 admin http://controller:8776/v3/%\(project_id\)s

###安装openstack-cinder服务###
yum install openstack-cinder

###修改cinder.conf配置文件###
vim /etc/cinder/cinder.conf
[DEFAULT]
my_ip = 172.16.188.9
auth_strategy = keystone
enabled_backends = lvm
transport_url = rabbit://openstack:openstack_password@controller
[backend]
[backend_defaults]
[barbican]
[brcd_fabric_example]
[cisco_fabric_example]
[coordination]
[cors]
[database]
connection = mysql+pymysql://cinder:cinder_password@controller/cinder
[fc-zone-manager]
[healthcheck]
[key_manager]
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = cinder
password = cinder_password
[matchmaker_redis]
[nova]
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp
[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
[oslo_messaging_zmq]
[oslo_middleware]
[oslo_policy]
[oslo_reports]
[oslo_versionedobjects]
[profiler]
[service_user]
[ssl]
[vault]

4)调用ceph-rbd块存储配置
配置思路：
登陆ceph集群服务器；
安装ceph驱动；
创建存储池；
授权账户；
拷贝密钥文件；
配置libvirt调用ceph-rbd；

###安装ceph驱动###
yum install python-rbd
yum install ceph-common
###创建存储池###
ceph osd pool create volumes 128
ceph osd pool create backups 128
ceph osd pool create images 128
ceph osd pool create vms 128
###创建授权账户###
ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images'

ceph auth get-or-create client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'

ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
###拷贝密钥文件###
ceph auth get-or-create client.cinder | ssh {your-volume-server} sudo tee /etc/ceph/ceph.client.cinder.keyring
ssh {your-cinder-volume-server} chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring

ceph auth get-or-create client.cinder-backup | ssh {your-cinder-backup-server} tee /etc/ceph/ceph.client.cinder-backup.keyring
ssh {your-cinder-backup-server} chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring

ceph auth get-or-create client.glance | ssh {your-glance-api-server} sudo tee /etc/ceph/ceph.client.glance.keyring
ssh {your-glance-api-server} chown glance:glance /etc/ceph/ceph.client.glance.keyring

ceph auth get-or-create client.cinder | ssh {your-nova-compute-server} tee /etc/ceph/ceph.client.cinder.keyring
5)配置libvrit调用ceph-rbd
uuidgen > uuid-secret.txt

cat > secret.xml <<EOF
<secret ephemeral='no' private='no'>
<uuid>cat uuid-secret.txt</uuid>
<usage type='ceph'>
<name>client.cinder secret</name>
</usage>
</secret>
EOF

virsh secret-define --file secret.xml
virsh secret-set-value --secret $(cat uuid-secret.txt) --base64 $(cat client.cinder.key) && rm client.cinder.key secret.xml

6)配置完成后，重启opnestakc相关服务
systemctl restart openstack-cinder-volume
systemctl restart openstack-cinder-backup
systemctl restart openstack-glance-api
systemctl restart openstack-nova-compute

7)登陆openstack界面验证
vim /etc/openstack-dashboard/local_settings
OPENSTACK_CINDER_FEATURES = {
'enable_backup': True,
}

8)创建卷，删除卷测试，查看ceph集群存储池读写删除状态；

配置参考网址：
http://docs.ceph.com/docs/master/rbd/rbd-openstack/
https://docs.openstack.org/cinder/queens/install/

#################分割线###############

以上配置步骤较为粗略，笔记待详细整理，后续会添加截图以及相关故障排查处理过程，配置步骤整理等；

转载于:https://blog.51cto.com/driver2ice/2162759