OpenStack Redhat Cinder对接glusterfs存储

最新推荐文章于 2024-08-05 23:01:32 发布

悾说

最新推荐文章于 2024-08-05 23:01:32 发布

阅读量331

点赞数

分类专栏： OpenStack 文章标签： openstack linux 服务器

本文链接：https://blog.csdn.net/kongshuo6/article/details/128542536

版权

OpenStack 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

一、创建虚拟机node1

自定义

默认下一步

稍后安装操作系统

选择Linux，红帽6

名称位置随意

1核就行

2G内存就行

仅主机网络

默认下一步

默认下一步

默认下一步

30G,给单个文件

添加一块50G硬盘

选择镜像

修改一下分辨率

开机，选择第二个安装

skip回车跳过

使用自定义IPV4，IPV6关掉

设置ip

默认回车ok

next下一步

默认英文

默认US

默认下一步

yes

随便取名，我写的node1

时区上海

密码随意

选择第一个使用所有

选择30G硬盘

write写入

最小化安装

安装

二、创建虚拟机node2

方法与node1一样，这里我就不演示了直接创好了

三、配置虚拟机node1

3.1 分区格式化

[root@node1 ~]# fdisk /dev/sdb 

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-6527, default 1): 1
Last cylinder, +cylinders or +size{K,M,G} (1-6527, default 6527): 
Using default value 6527

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@node1 ~]# lsblk
NAME                        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                           8:0    0   30G  0 disk 
├─sda1                        8:1    0  500M  0 part /boot
└─sda2                        8:2    0 29.5G  0 part 
  ├─vg_node1-lv_root (dm-0) 253:0    0 26.5G  0 lvm  /
  └─vg_node1-lv_swap (dm-1) 253:1    0    3G  0 lvm  [SWAP]
sdb                           8:16   0   50G  0 disk 
└─sdb1                        8:17   0   50G  0 part 
sr0                          11:0    1 1024M  0 rom  
[root@node1 ~]# mkfs.xfs /dev/sdb1 
meta-data=/dev/sdb1              isize=256    agcount=4, agsize=3276756 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=13107024, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=6399, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

3.2 开机自动挂载

[root@node1 ~]# mkdir /vol
[root@node1 ~]# blkid
/dev/sda1: UUID="f8f865a8-c79b-4881-9a51-c6c453d23ae8" TYPE="ext4" 
/dev/sda2: UUID="H8xNmN-IYdq-1WHL-VNf6-St69-MdwG-jbMtJs" TYPE="LVM2_member" 
/dev/sdb1: UUID="8f937432-1b36-49ab-9568-8fc4ec6ffa86" TYPE="xfs" 
/dev/mapper/vg_node1-lv_root: UUID="0c9f41c7-8229-440f-ad63-462044a54396" TYPE="ext4" 
/dev/mapper/vg_node1-lv_swap: UUID="fc302fb6-71d4-4b4b-bf2f-acf6664eb56b" TYPE="swap" 
[root@node1 ~]# vim /etc/fstab 
[root@node1 ~]# cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Thu Sep  1 03:38:04 2022
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_node1-lv_root /                       ext4    defaults        1 1
UUID=f8f865a8-c79b-4881-9a51-c6c453d23ae8 /boot                   ext4    defaults        1 2
/dev/mapper/vg_node1-lv_swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
UUID="8f937432-1b36-49ab-9568-8fc4ec6ffa86"	/vol	xfs  defaults	0 0
[root@node1 ~]# mount -a
[root@node1 ~]# df -hT
Filesystem           Type   Size  Used Avail Use% Mounted on
/dev/mapper/vg_node1-lv_root
                     ext4    27G  1.7G   24G   7% /
tmpfs                tmpfs  996M     0  996M   0% /dev/shm
/dev/sda1            ext4   485M   34M  426M   8% /boot
/dev/sdb1            xfs     50G   33M   50G   1% /vol

3.3 创建二级目录

因为glusterfs存储不能使用根目录下的一级目录，所以创建二级目录

[root@node1 ~]# mkdir /vol/node1

四、配置虚拟机node2

4.1 分区格式化

[root@node2 ~]# fdisk /dev/sdb 

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-6527, default 1): 1
Last cylinder, +cylinders or +size{K,M,G} (1-6527, default 6527): 
Using default value 6527

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@node2 ~]# mkfs.xfs /dev/sdb1 
meta-data=/dev/sdb1              isize=256    agcount=4, agsize=3276756 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=13107024, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=6399, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

4.2 开机自动挂载

[root@node2 ~]# mkdir /vol
[root@node2 ~]# blkid
/dev/sda1: UUID="13a7b8d9-3855-4b11-b2bc-efd0c4299511" TYPE="ext4" 
/dev/sda2: UUID="ERWFEQ-ZF9Y-T0tN-hqc9-cMeQ-ZSf0-f1NZWv" TYPE="LVM2_member" 
/dev/sdb1: UUID="e22005c2-bf50-4322-bbd2-18e59871d6c6" TYPE="xfs" 
/dev/mapper/vg_node2-lv_root: UUID="4b6f7e26-cc1f-42f6-a123-7c5a7915eafa" TYPE="ext4" 
/dev/mapper/vg_node2-lv_swap: UUID="d6a74559-d693-48dd-8d25-fb328af738ab" TYPE="swap" 
[root@node2 ~]# vim /etc/fstab 
[root@node2 ~]# cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Thu Sep  1 04:05:46 2022
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_node2-lv_root /                       ext4    defaults        1 1
UUID=13a7b8d9-3855-4b11-b2bc-efd0c4299511 /boot                   ext4    defaults        1 2
/dev/mapper/vg_node2-lv_swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
UUID="e22005c2-bf50-4322-bbd2-18e59871d6c6"	/vol	xfs	defaults	0 0
[root@node2 ~]# mount -a
[root@node2 ~]# df -hT
Filesystem           Type   Size  Used Avail Use% Mounted on
/dev/mapper/vg_node2-lv_root
                     ext4    27G  1.7G   24G   7% /
tmpfs                tmpfs  996M     0  996M   0% /dev/shm
/dev/sda1            ext4   485M   34M  426M   8% /boot
/dev/sdb1            xfs     50G   33M   50G   1% /vol

4.3 创建二级目录

因为glusterfs存储不能使用根目录下的一级目录，所以创建二级目录

[root@node2 ~]# mkdir /vol/node2

五、创建gluster卷

5.1创建池

[root@node1 ~]# gluster peer probe 192.168.100.60
peer probe: success. 
[root@node1 ~]# gluster pool list
UUID					Hostname	State
d8ef7ef9-bda6-4091-abc4-bad9c3ec69a0	192.168.100.60	Connected 
b6004bcb-a2fb-4372-a0b0-2394e11bbec7	localhost	Connected

5.2 创建卷

[root@node1 ~]# gluster volume create volmeme replica 2 192.168.100.50:/vol/node1 192.168.100.60:/vol/node2
volume create: volmeme: success: please start the volume to access data
[root@node1 ~]# gluster volume list
volmeme

5.3 启动卷

[root@node1 ~]# gluster volume start volmeme
volume start: volmeme: success
[root@node1 ~]# gluster volume status volmeme
Status of volume: volmeme
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick 192.168.100.50:/vol/node1				49152	Y	2344
Brick 192.168.100.60:/vol/node2				49152	Y	1934
NFS Server on localhost					2049	Y	2362
Self-heal Daemon on localhost				N/A	Y	2361
NFS Server on 192.168.100.60				2049	Y	1948
Self-heal Daemon on 192.168.100.60			N/A	Y	1956
 
Task Status of Volume volmeme
------------------------------------------------------------------------------
There are no active volume tasks

六、安装glusterfs驱动包

[root@controller ~]# yum list|grep gluster
glusterfs.x86_64                        3.6.0.42-1.el7          @RHEL7-Errata   
glusterfs-api.x86_64                    3.6.0.42-1.el7          @RHEL7-Errata   
glusterfs-libs.x86_64                   3.6.0.42-1.el7          @RHEL7-Errata   
glusterfs-cli.x86_64                    3.6.0.42-1.el7rhs       RHEL7-Errata    
glusterfs-fuse.x86_64                   3.6.0.42-1.el7          RHEL7-Errata    
glusterfs-rdma.x86_64                   3.6.0.29-2.el7          osp             
glusterfs-server.x86_64                 3.6.0.42-1.el7rhs       RHEL7-Errata    
[root@controller ~]# yum -y install glusterfs-fuse.x86_64
Loaded plugins: priorities, product-id, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
RH7-RHOS-6.0                                                                                                              | 1.3 kB  00:00:00     
RH7-RHOS-6.0-Installer                                                                                                    | 1.2 kB  00:00:00     
RHEL-7-RHSCL-1.2                                                                                                          | 3.3 kB  00:00:00     
RHEL7-Errata                                                                                                              | 2.9 kB  00:00:00     
rheliso                                                                                                                   | 4.1 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package glusterfs-fuse.x86_64 0:3.6.0.42-1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================================================================
 Package                              Arch                         Version                              Repository                          Size
=================================================================================================================================================
Installing:
 glusterfs-fuse                       x86_64                       3.6.0.42-1.el7                       RHEL7-Errata                        84 k

Transaction Summary
=================================================================================================================================================
Install  1 Package

Total download size: 84 k
Installed size: 245 k
Downloading packages:
glusterfs-fuse-3.6.0.42-1.el7.x86_64.rpm                                                                                  |  84 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : glusterfs-fuse-3.6.0.42-1.el7.x86_64                                                                                          1/1 
  Verifying  : glusterfs-fuse-3.6.0.42-1.el7.x86_64                                                                                          1/1 

Installed:
  glusterfs-fuse.x86_64 0:3.6.0.42-1.el7                                                                                                         

Complete!

七、测试挂载gluster卷

[root@controller ~]# mount -t glusterfs 192.168.100.50:/volmeme /mnt
[root@controller ~]# df -hT
Filesystem              Type            Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root   xfs              50G  3.7G   47G   8% /
devtmpfs                devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs                   tmpfs           3.9G  4.0K  3.9G   1% /dev/shm
tmpfs                   tmpfs           3.9G  8.5M  3.9G   1% /run
tmpfs                   tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/loop0              ext4            1.9G   22M  1.7G   2% /srv/node/swiftloopback
/dev/mapper/rhel-home   xfs              42G   33M   42G   1% /home
/dev/sda1               xfs             497M  104M  394M  21% /boot
tmpfs                   tmpfs           3.9G  8.5M  3.9G   1% /run/netns
192.168.100.50:/volmeme fuse.glusterfs   50G   33M   50G   1% /mnt
[root@controller ~]# cd /mnt/
[root@controller mnt]# ls
[root@controller mnt]# touch flag.txt

[root@node1 ~]# cd /vol/node1/
[root@node1 node1]# ls
flag.txt
[root@node2 ~]# cd /vol/node2/
[root@node2 node2]# ls
flag.txt

八、对接gluster

8.1 创建文件并配置

[root@controller ~]# vim /etc/cinder/glusterfs_shares
[root@controller ~]# cat /etc/cinder/glusterfs_shares 
192.168.100.50:/volmeme

8.2 编辑配置文件重启服务

[root@controller ~]# cd /etc/cinder/
[root@controller cinder]# vim cinder.conf 
enabled_backends=lvm,nfs,gluster
[lvm]
iscsi_helper=lioadm
volume_group=cinder-volumes
iscsi_ip_address=192.168.100.10
volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
volume_backend_name=lvm

[nfs]
nfs_shares_config=/etc/cinder/nfs_share
volume_driver=cinder.volume.drivers.nfs.NfsDriver
volume_backend_name=nfsbackend

[gluster]
volume_driver=cinder.volume.drivers.glusterfs.GlusterfsDriver
glusterfs_shares_config=/etc/cinder/glusterfs_shares
volume_backend_name=gs
[root@controller cinder]# openstack-service restart cinder

8.3 创建云硬盘类型

[root@controller ~]# source keystonerc_admin
[root@controller ~(keystone_admin)]# cinder type-create glusterfs
+--------------------------------------+-----------+
|                  ID                  |    Name   |
+--------------------------------------+-----------+
| 5a2a707d-d7f2-4c2a-b0f2-b259fa8b6e70 | glusterfs |
+--------------------------------------+-----------+
[root@controller ~(keystone_admin)]# cinder type-list 
+--------------------------------------+-----------+
|                  ID                  |    Name   |
+--------------------------------------+-----------+
| 5a2a707d-d7f2-4c2a-b0f2-b259fa8b6e70 | glusterfs |
| d172a274-12e0-4d04-ad9a-8ae6bbd0060c |   iscsi   |
| fb317c86-7941-4643-ba3b-c93491a80ada |  nfstype  |
+--------------------------------------+-----------+

设置后端名称

[root@controller ~(keystone_admin)]# cinder type-key glusterfs set volume_backend_name=gs

创建云硬盘

[root@controller ~]# source keystonerc_user 
[root@controller ~(keystone_user)]# cinder create --display-name gluster01 --volume-type glusterfs 2
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2022-09-01T13:16:13.771273      |
| display_description |                 None                 |
|     display_name    |              gluster01               |
|      encrypted      |                False                 |
|          id         | 5b7e7034-de21-4acd-b906-b0f5b7331583 |
|       metadata      |                  {}                  |
|         size        |                  2                   |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |              glusterfs               |
+---------------------+--------------------------------------+

尝试挂载给主机

[root@controller ~(keystone_user)]# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| 5b7e7034-de21-4acd-b906-b0f5b7331583 | available |  gluster01   |  2   |  glusterfs  |  false   |                                      |
| 9d31905f-6420-47e2-b75b-2b419882fa69 |   error   |    evs01     |  2   |   nfstype   |  false   | 26ac424c-7417-42a7-bbbc-016c5aab0ac4 |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
[root@controller ~(keystone_user)]# nova list
+--------------------------------------+---------+---------+------------+-------------+--------------------+
| ID                                   | Name    | Status  | Task State | Power State | Networks           |
+--------------------------------------+---------+---------+------------+-------------+--------------------+
| 26ac424c-7417-42a7-bbbc-016c5aab0ac4 | ecs01   | SHUTOFF | -          | Shutdown    | pri2=192.168.200.4 |
| 117291ac-e565-4206-ac2e-730b334f020e | gluster | ACTIVE  | -          | Running     | pri2=192.168.200.5 |
+--------------------------------------+---------+---------+------------+-------------+--------------------+
[root@controller ~(keystone_user)]# nova volume-attach gluster 5b7e7034-de21-4acd-b906-b0f5b7331583
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 5b7e7034-de21-4acd-b906-b0f5b7331583 |
| serverId | 117291ac-e565-4206-ac2e-730b334f020e |
| volumeId | 5b7e7034-de21-4acd-b906-b0f5b7331583 |
+----------+--------------------------------------+