OpenStack离线Train版安装系列—11.2存储节点-Cinder存储服务组件

本系列文章包含从OpenStack离线源制作到完成OpenStack安装的全部过程。
在本系列教程中使用的OpenStack的安装版本为第20个版本Train（简称T版本），2020年5月13日，OpenStack社区发布了第21个版本Ussuri（简称U版本）。

OpenStack部署系列文章
OpenStack Victoria版 安装部署系列教程
OpenStack Ussuri版 离线安装部署系列教程（全）
OpenStack Train版 离线安装部署系列教程（全）
欢迎留言沟通，共同进步。

---

---

11.2存储节点-Cinder存储服务组件

在存储节点服务器安装cinder存储服务

官方参考：
https://docs.openstack.org/install-guide/openstack-services.html
https://docs.openstack.org/train/install/
https://docs.openstack.org/cinder/train/install/
https://docs.openstack.org/cinder/train/install/index-rdo.html
https://docs.openstack.org/cinder/train/install/cinder-storage-install-rdo.html
（可选服务）
https://docs.openstack.org/cinder/train/install/cinder-backup-install-rdo.html
存储节点，安装和配置备份服务，在“块存储”节点上执行这些步骤。
博客：
https://yinwucheng.com/?p=491
https://www.cnblogs.com/tssc/p/9877026.html
https://www.cnblogs.com/wanlipenghtml/p/10721845.html
https://docs.openstack.org/mitaka/zh_CN/install-guide-rdo/cinder-storage-install.html

• 存储节点建议单独部署服务器（最好是物理机），测试时也可以部署在控制节点或者计算节点
• 在本文，存储节点使用LVM逻辑卷提供服务，需要提供一块空的磁盘用以创建LVM逻辑卷
• 这里在VMware虚拟机增加一块100GB的磁盘
• 在部署之前准备好存储设备。

一、存储节点准备好存储设备

0.磁盘挂载

在搭建的过程中使用的是VMware Workstation虚拟机来完成的，需要给存储节点新添加一个虚拟磁盘，并将磁盘挂载到虚拟机。
（1）.分区
增加了空间的硬盘是 /dev/sdb

[root@controller ~]# fdisk -l

Disk /dev/sda: 68.7 GB, 68719476736 bytes, 134217728 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a0797

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   134217727    66059264   8e  Linux LVM

Disk /dev/sdb: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-root: 42.7 GB, 42651877376 bytes, 83304448 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-swap: 4160 MB, 4160749568 bytes, 8126464 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/centos-home: 20.8 GB, 20824719360 bytes, 40673280 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

[root@controller ~]#

(2).分区

[root@controller ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p

Disk /dev/sdb: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xfa73ca8f

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 3
First sector (2048-41943039, default 2048): #回车
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039): #回车
Using default value 41943039
Partition 3 of type Linux and of size 20 GiB is set

Command (m for help): t
Selected partition 3
Hex code (type L to list all codes): w
Changed type of partition 'Linux' to 'unknown'

Command (m for help): q

[root@controller ~]# cd /dev/
[root@controller dev]# partprobe
[root@controller dev]# ls sd*
sda  sda1  sda2  sdb
[root@controller dev]#

(3).格式化

[root@controller dev]# mkfs.ext3 /dev/sdb
mke2fs 1.42.9 (28-Dec-2013)
/dev/sdb is entire device, not just one partition!
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
1310720 inodes, 5242880 blocks
262144 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
160 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

[root@controller dev]#

1.安装LVM相关软件包

安装LVM包

yum install lvm2 device-mapper-persistent-data -y

启动LVM的metadata服务并配置开机自启动

systemctl start lvm2-lvmetad.service
systemctl status lvm2-lvmetad.service

systemctl enable lvm2-lvmetad.service
systemctl list-unit-files |grep lvm2-lvmetad |grep enabled

2.创建LVM物理卷

创建LVM物理卷/dev/sdb

# 检查磁盘状态
[root@controller ~]# fdisk -l

# 创建LVM 物理卷 /dev/sdb
[root@controller ~]# pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created.

3.创建 LVM 卷组 cinder-volumes

(1)块存储服务会在这个卷组中创建逻辑卷

[root@controller ~]# vgcreate cinder-volumes /dev/sdb
  Volume group "cinder-volumes" successfully created
[root@controller ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               centos
  PV Size               <63.00 GiB / not usable 3.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              16127
  Free PE               1
  Allocated PE          16126
  PV UUID               aTMOV5-l175-850d-vuur-0LRK-HGiz-JBLcwZ
  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               cinder-volumes
  PV Size               20.00 GiB / not usable 4.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              5119
  Free PE               5119
  Allocated PE          0
  PV UUID               2GmNUZ-03a6-I2dH-hG0W-4ISc-5Liy-30XbrO

[root@controller ~]#

4.配置过滤器，只有实例可以访问块存储卷组,防止系统出错

(1)新增磁盘sdb为LVM卷组

• 默认只会有openstack实例访问块存储卷组，不过，底层的操作系统也会管理这些设备并尝试将逻辑卷与系统关联。
• 默认情况下LVM卷扫描工具会扫描整个/dev目录，查找所有包含lvm卷的块存储设备。如果其他项目在某个磁盘设备sda，sdc等上使用了lvm卷，那么扫描工具检测到这些卷时会尝试缓存这些lvm卷，可能导致底层操作系统或者其他服务无法正常调用他们的lvm卷组，从而产生各种问题，需要手动配置LVM，让LVM卷扫描工具只扫描包含"cinder-volume"卷组的设备/dev/sdb，我这边磁盘分区都是格式化的手工分区，目前不存在这个问题，以下是配置演示

将LVM重新配置为仅扫描包含cinder-volumes卷组的设备

vim /etc/lvm/lvm.conf
-----------------------------
devices {
filter = [ "a/sdb/", "r/.*/"]
}
-----------------------------

在devices部分中，添加一个接受/dev/sdb设备并拒绝所有其他设备的过滤器：

• 配置规则：
• a用于接受，r用于拒绝。
• 每个过滤器组中的元素都以a开头accept接受，或以 r 开头reject拒绝，后面连接设备名称的正则表达式规则。
• 过滤器组必须以"r/.*/"结束，过滤所有保留设备。
• 可以使用命令:vgs -vvvv来测试过滤器。

(2)、查看系统磁盘卷类型

[root@controller ~]# vgdisplay
  WARNING: Device for PV aTMOV5-l175-850d-vuur-0LRK-HGiz-JBLcwZ not found or rejected by a filter.
  Couldn't find device with uuid aTMOV5-l175-850d-vuur-0LRK-HGiz-JBLcwZ.
  WARNING: Couldn't find all devices for LV centos/swap while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV centos/home while checking used and assumed devices.
  WARNING: Couldn't find all devices for LV centos/root while checking used and assumed devices.
  --- Volume group ---
  VG Name               centos
  System ID
  Format                lvm2
  Metadata Areas        0
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                0
  VG Size               <63.00 GiB
  PE Size               4.00 MiB
  Total PE              16127
  Alloc PE / Size       16126 / 62.99 GiB
  Free  PE / Size       1 / 4.00 MiB
  VG UUID               Ws4wQS-K0VZ-vp9q-TB9Y-sLP9-OKyb-SZbTd8

  --- Volume group ---
  VG Name               cinder-volumes
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  18
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                4
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <20.00 GiB
  PE Size               4.00 MiB
  Total PE              5119
  Alloc PE / Size       4874 / <19.04 GiB
  Free  PE / Size       245 / 980.00 MiB
  VG UUID               AOqwgQ-yYNb-7anP-1tr4-1e3m-Bxth-b3R9rP

[root@controller ~]#

• 注意：
• 如果存储节点的操作系统磁盘/dev/sda使用的是LVM卷组，也需要将该设备添加到过滤器中，配置文件/etc/lvm/lvm.conf添加如下：

devices {
  ......
filter = [ "a/sda/", "a/sdb/", "r/.*/"]
  ......
}

• 如果计算节点的操作系统磁盘/dev/sda使用的是LVM卷组，也需要将该设备添加到过滤器中，配置文件/etc/lvm/lvm.conf添加如下：

devices {
  ......
filter = [ "a/sda/", "r/.*/"]
  ......
}

二、存储节点cinder相关软件安装与配置

1.安装相关软件

yum install openstack-cinder targetcli python-keystone -y
#用于快速配置
yum install openstack-utils -y

2.存储节点快速修改cinder配置

/etc/cinder/cinder.conf

cp /etc/cinder/cinder.conf /etc/cinder/cinder.conf.bak
grep -Ev '#|^$' /etc/cinder/cinder.conf.bak>/etc/cinder/cinder.conf

openstack-config --set  /etc/cinder/cinder.conf database connection  mysql+pymysql://cinder:CINDER_DBPASS@controller/cinder
openstack-config --set  /etc/cinder/cinder.conf DEFAULT transport_url  rabbit://openstack:RABBIT_USER_PASS@controller
openstack-config --set  /etc/cinder/cinder.conf DEFAULT auth_strategy  keystone
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken www_authenticate_uri  http://controller:5000
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken auth_url  http://controller:5000
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken memcached_servers  controller:11211
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken auth_type  password
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken project_domain_name  default
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken user_domain_name  default
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken project_name  service
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken username  cinder
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken password  CINDER_PASS
openstack-config --set  /etc/cinder/cinder.conf DEFAULT my_ip 10.0.0.41
openstack-config --set  /etc/cinder/cinder.conf DEFAULT enabled_backends  lvm
openstack-config --set  /etc/cinder/cinder.conf DEFAULT glance_api_servers  http://controller:9292
openstack-config --set  /etc/cinder/cinder.conf lvm volume_driver cinder.volume.drivers.lvm.LVMVolumeDriver
openstack-config --set  /etc/cinder/cinder.conf lvm volume_group cinder-volumes
openstack-config --set  /etc/cinder/cinder.conf lvm target_protocol  iscsi
openstack-config --set  /etc/cinder/cinder.conf lvm target_helper  lioadm
openstack-config --set  /etc/cinder/cinder.conf oslo_concurrency lock_path  /var/lib/cinder/tmp

#Replace RABBIT_PASS with the password you chose for the openstack account in RabbitMQ.
#Replace MANAGEMENT_INTERFACE_IP_ADDRESS with the IP address of the management network interface on your storage node, typically 10.0.0.41 for the first node.

#注意具体配置与之前版本有所变更
#[lvm]
# volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
# volume_group = cinder-volumes
# target_protocol = iscsi
# target_helper = lioadm

egrep -v "^#|^$" /etc/cinder/cinder.conf
grep '^[a-z]' /etc/cinder/cinder.conf

3.存储节点启动cinder服务并配置开机自启动

需要启动2个服务

systemctl start openstack-cinder-volume.service target.service
systemctl status openstack-cinder-volume.service target.service

systemctl enable openstack-cinder-volume.service target.service
systemctl list-unit-files |grep openstack-cinder |grep enabled
systemctl list-unit-files |grep target.service |grep enabled

至此存储节点的cinder服务安装完毕

至此，在存储节点安装cinder服务就完成了

三、（可选）安装和配置备份服务

#安装软件包：
yum install openstack-cinder

#编辑/etc/cinder/cinder.conf文件并完成以下操作
---------------------------------------------------------------
[DEFAULT]
# ...
backup_driver = cinder.backup.drivers.swift.SwiftBackupDriver
backup_swift_url = SWIFT_URL
---------------------------------------------------------------
#替换SWIFT_URL为对象存储服务的URL。可以通过显示对象库API端点来找到URL：

openstack catalog show object-store

#确定安装,启动块存储备份服务，并将其配置为在系统启动时启动：
systemctl start openstack-cinder-backup.service
systemctl status openstack-cinder-backup.service
systemctl enable openstack-cinder-backup.service
systemctl list-unit-files |grep openstack-cinder-backup.service |grep enabled