KVM虚拟化开源高可用方案(三)glusterfs

 

 

 

说明：

设计目的：

1. 集群设计虚拟机容量70-100台,占用1个机柜，全部由1U服务器组成，其中存储服务器6台，300G*8，节点服务器10台；

2. 虚拟机可以在KVM集群宿主机之间迁移；

glusterfs集群架构：

1. 存储服务器和节点服务器组成，存储服务器通过哈希算法，可以弹性增加或者减少，并实现冗余；

2. 存储服务器每台机器至少需要4块网卡，如果机器只有板载的2块网卡，需要在加1块双口网卡，做4块网卡的绑定，这样可以提高网络带宽；

3. KVM集群每台宿主机作为glusterfs客户端，挂载glusterfs集群的文件系统，将虚拟机放置在上面；

glusterfs集群搭建：

存储节点搭建

1 安装相关组件

yum -y install wget fuse fuse-libs

2 下载安装测试

yum -y install wget fuse fuse-libs

wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-3.3.0-1.el6.x86_64.rpm

wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-debuginfo-3.3.0-1.el6.x86_64.rpm

wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-devel-3.3.0-1.el6.x86_64.rpm

wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-fuse-3.3.0-1.el6.x86_64.rpm

wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-geo-replication-3.3.0-1.el6.x86_64.rpm

wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-rdma-3.3.0-1.el6.x86_64.rpm

wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-server-3.3.0-1.el6.x86_64.rpm

yum install glusterfs-* -y

/usr/sbin/glusterfs -V

3 启动glusterd服务

默认安装完成自动启动服务，安装完成后是一个标准的redhat服务，可以启动，停止，重启

/etc/init.d/glusterd start

/etc/init.d/glusterd stop

/etc/init.d/glusterd restart

chkconfig glusterd on

4 添加服务器到存储池

在第一台服务器上执行探测操作

gluster peer probe server1

gluster peer probe server2

gluster peer probe server3

gluster peer probe server4

gluster peer probe server5

校验集群状态

[root@hp246 ~]# gluster peer status

Number of Peers: 4

Hostname: server1

Uuid: 59cd74a9-a555-4560-b98e-a7eaf2058926

State: Peer in Cluster (Connected)

Hostname: server2

Uuid: 278d94f8-cf55-42cc-a4ad-9f84295c140b

State: Peer in Cluster (Connected)

Hostname: server3

Uuid: 7fd840a2-53f5-4540-b455-3e5e7eded813

State: Peer in Cluster (Connected)

Hostname: server4

Uuid: 4bfd8649-7f74-4a9f-9f04-4267cc80a1c3

State: Peer in Cluster (Connected)

…

如果需要移出集群 执行如下命令

gluster peer detach server

5 创建集群卷

1)创建一个分布卷（只是卷连接起来，跨区卷）

gluster volume create test-volume server1:/exp1 server2:/exp2 server3:/exp3 server4:/

exp4

2)创建一个复制卷(类似raid1)

gluster volume create test-volume replica 2 transport tcp server1:/exp1 server2:/exp2

3)创建一个条带卷(类似raid0)

gluster volume create test-volume stripe 2 transport tcp server1:/exp1 server2:/exp2

4）创建一个分布条带卷(类似raid0+0)

gluster volume create test-volume stripe 4 transport tcp server1:/exp1 server2:/exp2

server3:/exp3 server4:/exp4 server5:/exp5 server6:/exp6 server7:/exp7 server8:/exp8

5）创建一个复制条带卷(类似raid1然后跨区)

gluster volume create test-volume replica 2 transport tcp server1:/exp1 server2:/exp2

server3:/exp3 server4:/exp4

6)创建一个分部复制条带卷(类似raid10)

gluster volume create test-volume stripe 2 replica 2 transport tcp server1:/exp1

server2:/exp2 server3:/exp3 server4:/exp4 server5:/exp5 server6:/exp6 server7:/exp7

server8:/exp8

7)创建条带复制卷(类似raid01)

gluster volume create test-volume stripe 2 replica 2 transport tcp server1:/exp1

server2:/exp2 server3:/exp3 server4:/exp4

查看卷信息

gluster volume info

启动卷

gluster volume start test-volume

6 客户端挂载

modprobe fuse

Verify that the FUSE module is loaded:

# dmesg | grep -i fuse

fuse init (API version 7.13)

yum -y install wget fuse fuse-libs

wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-3.3.0-1.el6.x86_64.rpm

wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-debuginfo-3.3.0-1.el6.x86_64.rpm

wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-devel-3.3.0-1.el6.x86_64.rpm

wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-fuse-3.3.0-1.el6.x86_64.rpm

wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-geo-replication-3.3.0-1.el6.x86_64.rpm

wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-rdma-3.3.0-1.el6.x86_64.rpm

wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-server-3.3.0-1.el6.x86_64.rpm

yum install glusterfs-* -y

挂载卷

mount -t glusterfs server1:/test-volume /mnt/glusterfs

mount -t glusterfs hp246:/test-volume /gfs

自动挂载

vim fstab

server1:/test-volume /mnt/glusterfs glusterfs defaults,_netdev 0 0

创建虚拟机：略

欢迎加我微信（ID：xiaolikvm）一起交流。欢迎加入云技术社区KVM/QEMU技术交流QQ，请加QQ:502207183，注明姓名，城市，及要加入KVM群。