Glusterfs 简介
GlusterFS 分布式文件系统是由 Gluster 公司的创始人兼首席技术官 Anand BabuPeriasamy 编写。一个可扩展的分布式文件系统，可用于大型的、分布式的、对大量数据进行访问的应用。它可运行于廉价的普通硬件上，并提供容错功能;也可以给大量的用户提供总体性能较高的服务。GlusterFs可以根据存储需求快速调配存储，内含丰富的自动故障转移功能，且摒弃集中元数据服务器的思想。
Glusterfs 特点
GlusterFS 体系结构，将计算、存储和 IO 资源聚合到全局名称空间中，每台服务器都被视为节点，通过添加附加节点或向每个节点添加额外存储来扩展容量。通过在更多节点之间部署存储来提高性能。
GlusterFS 支持基于文件的镜像和复制、分条、负载平衡、故障转移、调度、磁盘缓存、存储配额、卷快照等功能。
GlusterFs 各客户端之间无连接，本身依赖于弹性哈希算法，而不是使用集中式或分布式元数据模型。
GlusterFS通过各种复制选项提供数据可靠性和可用性，例如复制卷、分布卷。
1．案例环境
2．案例需求
（1）部署 GlusterFS 文件系统
（2）实现 KVM 虚拟主机不会因宿主机宕机而宕机
3．案例实现思路
（1）安装 KVM。
（2）所有节点部署 GlusterFS。
（3）客户端挂载 GlusterFS
（4）KVM 使用挂载的 GlusterFS 目录创建虚拟机。
一：部署GFS高可用分布式存储环境
1：安装部署 KVM 虚拟化平台
2：部署 GlusterFS
在所有节点上执行如下命令：
（1）关闭防所有节点的防火墙、SELiunx
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp">systemctl stop firewalld
systemctl disable firewalld
setenforce <span style="color:#c92c2c">0</span></code></span></span></span></span>（2）编写 hosts 文件
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><div style="text-align:start"><div><div><div><div><div><div><span style="color:var(--gpts-primary-text-color)"><span style="color:rgba(0, 0, 0, 0.88)"><span style="color:var(--gpts-black-text-color1)">解释</span></span></span></div></div><div></div><div></div></div></div></div></div></div><code class="language-cpp"><span style="color:#5f6364">[</span>root@node1 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># cat<span style="color:#a67f59"><<</span><span style="color:#c92c2c">EOF</span><span style="color:#a67f59">></span> <span style="color:#a67f59">/</span>etc<span style="color:#a67f59">/</span>hosts
<span style="color:#c92c2c">192.168</span><span style="color:#5f6364">.</span><span style="color:#c92c2c">10.101</span> node1
<span style="color:#c92c2c">192.168</span><span style="color:#5f6364">.</span><span style="color:#c92c2c">10.102</span> node2
<span style="color:#c92c2c">192.168</span><span style="color:#5f6364">.</span><span style="color:#c92c2c">10.103</span> node3
<span style="color:#c92c2c">192.168</span><span style="color:#5f6364">.</span><span style="color:#c92c2c">10.104</span> node4
<span style="color:#c92c2c">192.168</span><span style="color:#5f6364">.</span><span style="color:#c92c2c">10.201</span> kvm01
<span style="color:#c92c2c">192.168</span><span style="color:#5f6364">.</span><span style="color:#c92c2c">10.202</span> kvm02
<span style="color:#c92c2c">EOF</span></code></span></span></span></span>（3）安装软件
注意：先设置阿里yum仓库
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@node1 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># yum <span style="color:#a67f59">-</span>y install centos<span style="color:#a67f59">-</span>release<span style="color:#a67f59">-</span>gluster
<span style="color:#5f6364">[</span>root@node1 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># yum <span style="color:#a67f59">-</span>y install glusterfs glusterfs<span style="color:#a67f59">-</span>server glusterfs<span style="color:#a67f59">-</span>fuse glusterfs<span style="color:#a67f59">-</span>rdma</code></span></span></span></span>（4）启动 GlusterFS
在所有节点Gluster执行以下操作
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@node1 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># systemctl start glusterd<span style="color:#5f6364">.</span>service <span style="color:#a67f59">&&</span> systemctl enable glusterd<span style="color:#5f6364">.</span>service</code></span></span></span></span>（5）在 node1 上添加所有节点
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><div style="text-align:start"><div><div><div><div><div><div><span style="color:var(--gpts-primary-text-color)"><span style="color:rgba(0, 0, 0, 0.88)"><span style="color:var(--gpts-black-text-color1)">解释</span></span></span></div></div><div></div><div></div></div></div></div></div></div><code class="language-cpp"><span style="color:#5f6364">[</span>root@node1 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># gluster peer probe node2
peer probe<span style="color:#a67f59">:</span> success<span style="color:#5f6364">.</span>
<span style="color:#5f6364">[</span>root@node1 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># gluster peer probe node3
peer probe<span style="color:#a67f59">:</span> success<span style="color:#5f6364">.</span>
<span style="color:#5f6364">[</span>root@node1 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># gluster peer probe node4
peer probe<span style="color:#a67f59">:</span> success<span style="color:#5f6364">.</span></code></span></span></span></span>（6）查看集群状态
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@node1 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># gluster peer status</code></span></span></span></span>3：创建 GlusterFS 分布式复制卷
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><div style="text-align:start"><div><div><div><div><div><div><span style="color:var(--gpts-primary-text-color)"><span style="color:rgba(0, 0, 0, 0.88)"><span style="color:var(--gpts-black-text-color1)">解释</span></span></span></div></div><div></div><div></div></div></div></div></div></div><code class="language-cpp">在所有节点创建<span style="color:#a67f59">/</span>data 目录
<span style="color:#5f6364">[</span>root@node1 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># mkdir <span style="color:#a67f59">/</span>data
创建分布式复制卷。
<span style="color:#5f6364">[</span>root@node1 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># gluster volume create models replica <span style="color:#c92c2c">2</span> node1<span style="color:#a67f59">:</span><span style="color:#a67f59">/</span>data node2<span style="color:#a67f59">:</span><span style="color:#a67f59">/</span>data node3<span style="color:#a67f59">:</span><span style="color:#a67f59">/</span>data node4<span style="color:#a67f59">:</span><span style="color:#a67f59">/</span>data force
查看 models 卷
<span style="color:#5f6364">[</span>root@node1 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># gluster volume info models
启动 models 卷
<span style="color:#5f6364">[</span>root@node1 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># gluster volume start models</code></span></span></span></span>二：为KVM主机部署GFS存储
1：KVM挂载 glusterfs 卷(所有kvm主机上都配置)
（1）安装 glusterfs 客户端软件。
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># yum <span style="color:#a67f59">-</span>y install glusterfs glusterfs<span style="color:#a67f59">-</span>fuse</code></span></span></span></span>（2）创建挂载目录，并挂载 models 卷。
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><div style="text-align:start"><div><div><div><div><div><div><span style="color:var(--gpts-primary-text-color)"><span style="color:rgba(0, 0, 0, 0.88)"><span style="color:var(--gpts-black-text-color1)">解释</span></span></span></div></div><div></div><div></div></div></div></div></div></div><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># mkdir <span style="color:#a67f59">/</span>kvmdata
<span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># mount <span style="color:#a67f59">-</span>t glusterfs node1<span style="color:#a67f59">:</span>models <span style="color:#a67f59">/</span>kvmdata<span style="color:#a67f59">/</span>
<span style="color:#5f6364">[</span>root@localhost <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># vi <span style="color:#a67f59">/</span>etc<span style="color:#a67f59">/</span>fstab
node1<span style="color:#a67f59">:</span>models <span style="color:#a67f59">/</span>kvmdata glusterfs defaults<span style="color:#5f6364">,</span>_netdev <span style="color:#c92c2c">0</span> <span style="color:#c92c2c">0</span></code></span></span></span></span>（3）查看挂载卷。
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># df <span style="color:#a67f59">-</span>h</code></span></span></span></span>2：在kvm01上安装虚拟机
（1）拷贝qcow2磁盘文件
将实验用到的系统镜像上传到KVM主机的root家目录中。这里直接使用CentOS官方提供的qcow2格式的系统镜像（CentOS-7-x86_64-GenericCloud-2009.qcow2），这个镜像可以直接部署到kvm或openstack的平台中。
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># cp CentOS<span style="color:#a67f59">-</span><span style="color:#c92c2c">7</span><span style="color:#a67f59">-</span>x86_64<span style="color:#a67f59">-</span>GenericCloud<span style="color:#a67f59">-</span><span style="color:#c92c2c">2009.</span>qcow2 <span style="color:#a67f59">/</span>kvmdata<span style="color:#a67f59">/</span>test01<span style="color:#5f6364">.</span>qcow2</code></span></span></span></span>（2）部署虚拟机
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># yum <span style="color:#a67f59">-</span>y install libguestfs<span style="color:#a67f59">-</span>tools
<span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># gpasswd <span style="color:#a67f59">-</span>a qemu root</code></span></span></span></span>备注：
该工具提供了virt的高级命令，其中有一个virt-customize命令，可以为系统镜像设置密码。
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># virt<span style="color:#a67f59">-</span>customize <span style="color:#a67f59">-</span>a <span style="color:#a67f59">/</span>kvmdata<span style="color:#a67f59">/</span>test01<span style="color:#5f6364">.</span>qcow2 <span style="color:#a67f59">--</span>root<span style="color:#a67f59">-</span>password password<span style="color:#a67f59">:</span>aptech
<span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># virt<span style="color:#a67f59">-</span>install <span style="color:#a67f59">--</span>name<span style="color:#a67f59">=</span>test01 <span style="color:#a67f59">-</span>r <span style="color:#c92c2c">1024</span> <span style="color:#a67f59">--</span>vcpus<span style="color:#a67f59">=</span><span style="color:#c92c2c">1</span> <span style="color:#a67f59">--</span>disk device<span style="color:#a67f59">=</span>disk<span style="color:#5f6364">,</span>bus<span style="color:#a67f59">=</span>virtio<span style="color:#5f6364">,</span>path<span style="color:#a67f59">=</span><span style="color:#128b4e">'/kvmdata/test01.qcow2'</span><span style="color:#5f6364">,</span>size<span style="color:#a67f59">=</span><span style="color:#c92c2c">10</span> <span style="color:#a67f59">-</span>w bridge<span style="color:#a67f59">:</span>br0 <span style="color:#a67f59">--</span>virt<span style="color:#a67f59">-</span>type<span style="color:#a67f59">=</span>kvm <span style="color:#a67f59">--</span>boot hd</code></span></span></span></span>备注：
--disk：指定存储设备及其属性
device：设备类型，如cdrom、disk或floppy等，默认为disk；
bus：磁盘总线类型，其值可以为ide、scsi、usb、virtio或xen
--virt-type：使用的hypervisor（虚拟机监视器），如kvm、qemu、xen等
--boot cdrom,hd,network：指定引导次序；
（3）登录测试
三：测试实时迁移环境
1：配置kvm01和kvm02的无密码登录环境
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><div style="text-align:start"><div><div><div><div><div><div><span style="color:var(--gpts-primary-text-color)"><span style="color:rgba(0, 0, 0, 0.88)"><span style="color:var(--gpts-black-text-color1)">解释</span></span></span></div></div><div></div><div></div></div></div></div></div></div><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># ssh<span style="color:#a67f59">-</span>keygen <span style="color:#a67f59">-</span>t rsa
<span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># ssh<span style="color:#a67f59">-</span>copy<span style="color:#a67f59">-</span>id kvm02
<span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># ssh<span style="color:#a67f59">-</span>copy<span style="color:#a67f59">-</span>id kvm01
<span style="color:#5f6364">[</span>root@kvm02 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span>#ssh<span style="color:#a67f59">-</span>keygen <span style="color:#a67f59">-</span>t rsa
<span style="color:#5f6364">[</span>root@kvm02 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span>#ssh<span style="color:#a67f59">-</span>copy<span style="color:#a67f59">-</span>id kvm01
<span style="color:#5f6364">[</span>root@kvm02 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># ssh<span style="color:#a67f59">-</span>copy<span style="color:#a67f59">-</span>id kvm02</code></span></span></span></span>2：将test01虚拟机从kvm01主机迁移到kvm02主机
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># virsh migrate <span style="color:#a67f59">--</span>persistent <span style="color:#a67f59">--</span>undefinesource test01 qemu<span style="color:#a67f59">+</span>ssh<span style="color:#a67f59">:</span><span style="color:#7d8b99"><em>//kvm02/system</em></span>
<span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># virsh list <span style="color:#a67f59">--</span>all</code></span></span></span></span>备注：
迁移成功后，在kvm01主机就没有了test01的资源
3：查看kvm02主机上的虚拟机状态
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><div style="text-align:start"><div><div><div><div><div><div><span style="color:var(--gpts-primary-text-color)"><span style="color:rgba(0, 0, 0, 0.88)"><span style="color:var(--gpts-black-text-color1)">解释</span></span></span></div></div><div></div><div></div></div></div></div></div></div><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># virsh <span style="color:#a67f59">--</span>connect<span style="color:#a67f59">=</span>qemu<span style="color:#a67f59">+</span>ssh<span style="color:#a67f59">:</span><span style="color:#7d8b99"><em>//kvm02/system list</em></span>
Id 名称 状态
<span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span>
<span style="color:#c92c2c">1</span> test01 running</code></span></span></span></span>4：将test01从kvm02主机迁移到kvm01主机
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># virsh <span style="color:#a67f59">--</span>connect<span style="color:#a67f59">=</span>qemu<span style="color:#a67f59">+</span>ssh<span style="color:#a67f59">:</span><span style="color:#7d8b99"><em>//kvm02/system migrate --persistent --undefinesource test01 qemu+ssh://kvm01/system</em></span></code></span></span></span></span>四：创建虚拟机资源
1：为配置文件创建共享目录
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># mkdir <span style="color:#a67f59">/</span>kvmdata<span style="color:#a67f59">/</span>config</code></span></span></span></span>2：将test01的配置文件拷贝到共享目录
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># cp <span style="color:#a67f59">/</span>etc<span style="color:#a67f59">/</span>libvirt<span style="color:#a67f59">/</span>qemu<span style="color:#a67f59">/</span>test01<span style="color:#5f6364">.</span>xml <span style="color:#a67f59">/</span>kvmdata<span style="color:#a67f59">/</span>config<span style="color:#a67f59">/</span></code></span></span></span></span>3：取消定义test01虚拟机
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># virsh shutdown test01
<span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># virsh undefine test01
<span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># virsh list <span style="color:#a67f59">--</span>all</code></span></span></span></span>4：重新定义test01虚拟机
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><div style="text-align:start"><div><div><div><div><div><div><span style="color:var(--gpts-primary-text-color)"><span style="color:rgba(0, 0, 0, 0.88)"><span style="color:var(--gpts-black-text-color1)">解释</span></span></span></div></div><div></div><div></div></div></div></div></div></div><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm02 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># virsh define <span style="color:#a67f59">/</span>kvmdata<span style="color:#a67f59">/</span>config<span style="color:#a67f59">/</span>test01<span style="color:#5f6364">.</span>xml
<span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># virsh start test01
域 test01 已开始
<span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># virsh list <span style="color:#a67f59">--</span>all
Id 名称 状态
<span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span>
<span style="color:#c92c2c">6</span> test01 running</code></span></span></span></span>五：部署群集
1：群集组件的安装（kvm01和kvm02都安装）
设置好阿里的基础源和epel扩展源
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># yum <span style="color:#a67f59">-</span>y install pcs pacemaker fence<span style="color:#a67f59">-</span>agents<span style="color:#a67f59">-</span>all</code></span></span></span></span>备注：
Pacemaker 是一个集群管理器。它利用OpenAIS或 heartbeat提供的消息和成员能力，由辅助节点和系统进行故障检测和回收，实现性群集服务（亦称资源）的高可用性。
Fence agent是集群管理中的一个概念，用于在发生电源故障或其他故障时确保集群的稳定性。
pcs 是集群管理的一个守护进程，利用pcs提供的工具可以对集群环境进行状态的管理。
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># passwd hacluster</code></span></span></span></span>备注：
两台kvm主机的密码要一致（本案例设置的是aptech）
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># systemctl start pcsd
<span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># systemctl enable pcsd</code></span></span></span></span>2：认证组成群集的节点（只在kvm01上操作）
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span>#pcs cluster auth kvm01 kvm02 <span style="color:#a67f59">-</span>u hacluster <span style="color:#a67f59">-</span>p aptech</code></span></span></span></span>3：自动生成配置文件（只在kvm01上操作）
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># pcs cluster setup <span style="color:#a67f59">--</span>name cluster<span style="color:#a67f59">-</span>kvm kvm01 kvm02</code></span></span></span></span>4：启动群集（只在kvm01上操作）
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># pcs cluster start <span style="color:#a67f59">--</span>all
<span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># pcs cluster enable <span style="color:#a67f59">--</span>all</code></span></span></span></span>备注：
enable：将群集设置为开机自启
5：在任意一个kvm主机上查看pcs群集状态
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># pcs status</code></span></span></span></span>6：将其中一个节点关闭，查看群集状态
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span>#pcs cluster stop kvm01</code></span></span></span></span>在另一个节点查看状态
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm02 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># pcs status
Online<span style="color:#a67f59">:</span> <span style="color:#5f6364">[</span> kvm02 <span style="color:#5f6364">]</span>
OFFLINE<span style="color:#a67f59">:</span> <span style="color:#5f6364">[</span> kvm01 <span style="color:#5f6364">]</span></code></span></span></span></span>7：查看后再开启，让群集正常运行
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># pcs cluster start kvm01</code></span></span></span></span>8：关闭隔离设备的功能（每个设备都执行）
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp">pcs property set stonith<span style="color:#a67f59">-</span>enabled<span style="color:#a67f59">=</span><span style="color:#c92c2c">false</span></code></span></span></span></span>备注：
测试环境用的 VMware的环境，没有隔离设备，必须要关闭，否则无法实现高可用
9：向群集中添加资源
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># pcs resource create test01 VirtualDomain hypervisor<span style="color:#a67f59">=</span><span style="color:#128b4e">"qemu:///system"</span> config<span style="color:#a67f59">=</span><span style="color:#128b4e">"/kvmdata/config/test01.xml"</span> migration_transport<span style="color:#a67f59">=</span>ssh meta allow<span style="color:#a67f59">-</span>migrate<span style="color:#a67f59">=</span><span style="color:#128b4e">"true"</span></code></span></span></span></span>备注
meta allow-migrate="true"：允许迁移模式
10：查看当前群集状态
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><div style="text-align:start"><div><div><div><div><div><div><span style="color:var(--gpts-primary-text-color)"><span style="color:rgba(0, 0, 0, 0.88)"><span style="color:var(--gpts-black-text-color1)">解释</span></span></span></div></div><div></div><div></div></div></div></div></div></div><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># pcs status
Cluster name<span style="color:#a67f59">:</span> cluster<span style="color:#a67f59">-</span>kvm
WARNINGS<span style="color:#a67f59">:</span>
No stonith devices <span style="color:#a67f59">and</span> stonith<span style="color:#a67f59">-</span>enabled is <span style="color:#a67f59">not</span> <span style="color:#c92c2c">false</span>
Stack<span style="color:#a67f59">:</span> corosync
Current DC<span style="color:#a67f59">:</span> <span style="color:#128b4e">kvm02</span> <span style="color:#5f6364">(</span>version <span style="color:#c92c2c">1.1</span><span style="color:#5f6364">.</span><span style="color:#c92c2c">23</span><span style="color:#a67f59">-</span><span style="color:#c92c2c">1.</span>el7_9<span style="color:#5f6364">.</span><span style="color:#c92c2c">1</span><span style="color:#a67f59">-</span><span style="color:#c92c2c">9</span>acf116022<span style="color:#5f6364">)</span> <span style="color:#a67f59">-</span> partition with quorum
Last updated<span style="color:#a67f59">:</span> Wed Aug <span style="color:#c92c2c">31</span> <span style="color:#c92c2c">19</span><span style="color:#a67f59">:</span><span style="color:#c92c2c">55</span><span style="color:#a67f59">:</span><span style="color:#c92c2c">48</span> <span style="color:#c92c2c">2022</span>
Last change<span style="color:#a67f59">:</span> Wed Aug <span style="color:#c92c2c">31</span> <span style="color:#c92c2c">19</span><span style="color:#a67f59">:</span><span style="color:#c92c2c">55</span><span style="color:#a67f59">:</span><span style="color:#c92c2c">43</span> <span style="color:#c92c2c">2022</span> by root via cibadmin on kvm01
<span style="color:#c92c2c">2</span> nodes configured
<span style="color:#c92c2c">1</span> resource instance configured
Online<span style="color:#a67f59">:</span> <span style="color:#5f6364">[</span> kvm01 kvm02 <span style="color:#5f6364">]</span>
Full list of resources<span style="color:#a67f59">:</span>
<span style="color:#128b4e">test01</span> <span style="color:#5f6364">(</span>ocf<span style="color:#5f6364">::</span>heartbeat<span style="color:#a67f59">:</span>VirtualDomain<span style="color:#5f6364">)</span><span style="color:#a67f59">:</span> Started kvm01
Daemon Status<span style="color:#a67f59">:</span>
corosync<span style="color:#a67f59">:</span> active<span style="color:#a67f59">/</span>enabled
pacemaker<span style="color:#a67f59">:</span> active<span style="color:#a67f59">/</span>enabled
pcsd<span style="color:#a67f59">:</span> active<span style="color:#a67f59">/</span>enabled</code></span></span></span></span>六：KVM群集验证
1：在两台kvm主机上分别查看虚拟机状态
（1）kvm01的状态
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><div style="text-align:start"><div><div><div><div><div><div><span style="color:var(--gpts-primary-text-color)"><span style="color:rgba(0, 0, 0, 0.88)"><span style="color:var(--gpts-black-text-color1)">解释</span></span></span></div></div><div></div><div></div></div></div></div></div></div><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># virsh list <span style="color:#a67f59">--</span>all
Id 名称 状态
<span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span>
<span style="color:#c92c2c">6</span> test01 running
（<span style="color:#c92c2c">2</span>）kvm02的状态
<span style="color:#5f6364">[</span>root@kvm02 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># virsh list <span style="color:#a67f59">--</span>all
Id 名称 状态
<span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span></code></span></span></span></span>注意：此时虚拟机在kvm01
2：删除资源test01的约束
清除某个 pcs 资源只能在某个主机上运行的限制
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><div style="text-align:start"><div><div><div><div><div><div><span style="color:var(--gpts-primary-text-color)"><span style="color:rgba(0, 0, 0, 0.88)"><span style="color:var(--gpts-black-text-color1)">解释</span></span></span></div></div><div></div><div></div></div></div></div></div></div><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># pcs resource clear test01
<span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># pcs constraint ##查询限制情况，如下显示结果为无限制
Location Constraints<span style="color:#a67f59">:</span>
Ordering Constraints<span style="color:#a67f59">:</span>
Colocation Constraints<span style="color:#a67f59">:</span>
Ticket Constraints<span style="color:#a67f59">:</span></code></span></span></span></span>备注：
如果冒号后没有任何信息，表示没有约束，如果有约束会有警告提示
constraint（限制）
当把虚拟机迁移到了另一台主机，会出现约束，要想将虚拟机迁移回来，需要先清除约束
3：手动迁移
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># pcs resource move test01</code></span></span></span></span>备注：
会自动选择其他节点进行迁移
4：kvm01上查看状态
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><div style="text-align:start"><div><div><div><div><div><div><span style="color:var(--gpts-primary-text-color)"><span style="color:rgba(0, 0, 0, 0.88)"><span style="color:var(--gpts-black-text-color1)">解释</span></span></span></div></div><div></div><div></div></div></div></div></div></div><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># virsh list <span style="color:#a67f59">--</span>all
Id 名称 状态
<span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span>
<span style="color:#a67f59">-</span> test01 关闭
<span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># pcs status
Cluster name<span style="color:#a67f59">:</span> cluster<span style="color:#a67f59">-</span>kvm
Stack<span style="color:#a67f59">:</span> corosync
Current DC<span style="color:#a67f59">:</span> <span style="color:#128b4e">kvm01</span> <span style="color:#5f6364">(</span>version <span style="color:#c92c2c">1.1</span><span style="color:#5f6364">.</span><span style="color:#c92c2c">23</span><span style="color:#a67f59">-</span><span style="color:#c92c2c">1.</span>el7_9<span style="color:#5f6364">.</span><span style="color:#c92c2c">1</span><span style="color:#a67f59">-</span><span style="color:#c92c2c">9</span>acf116022<span style="color:#5f6364">)</span> <span style="color:#a67f59">-</span> partition with quorum
Last updated<span style="color:#a67f59">:</span> Wed Aug <span style="color:#c92c2c">31</span> <span style="color:#c92c2c">21</span><span style="color:#a67f59">:</span><span style="color:#c92c2c">54</span><span style="color:#a67f59">:</span><span style="color:#c92c2c">15</span> <span style="color:#c92c2c">2022</span>
Last change<span style="color:#a67f59">:</span> Wed Aug <span style="color:#c92c2c">31</span> <span style="color:#c92c2c">21</span><span style="color:#a67f59">:</span><span style="color:#c92c2c">52</span><span style="color:#a67f59">:</span><span style="color:#c92c2c">31</span> <span style="color:#c92c2c">2022</span> by root via crm_resource on kvm01
<span style="color:#c92c2c">2</span> nodes configured
<span style="color:#c92c2c">1</span> resource instance configured
Online<span style="color:#a67f59">:</span> <span style="color:#5f6364">[</span> kvm01 kvm02 <span style="color:#5f6364">]</span>
Full list of resources<span style="color:#a67f59">:</span>
<span style="color:#128b4e">test01</span> <span style="color:#5f6364">(</span>ocf<span style="color:#5f6364">::</span>heartbeat<span style="color:#a67f59">:</span>VirtualDomain<span style="color:#5f6364">)</span><span style="color:#a67f59">:</span> Started kvm02
Daemon Status<span style="color:#a67f59">:</span>
corosync<span style="color:#a67f59">:</span> active<span style="color:#a67f59">/</span>enabled
pacemaker<span style="color:#a67f59">:</span> active<span style="color:#a67f59">/</span>enabled
pcsd<span style="color:#a67f59">:</span> active<span style="color:#a67f59">/</span>enabled</code></span></span></span></span>5：kvm02上查看状态
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><div style="text-align:start"><div><div><div><div><div><div><span style="color:var(--gpts-primary-text-color)"><span style="color:rgba(0, 0, 0, 0.88)"><span style="color:var(--gpts-black-text-color1)">解释</span></span></span></div></div><div></div><div></div></div></div></div></div></div><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm02 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span>#virsh list <span style="color:#a67f59">--</span>all
Id 名称 状态
<span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span>
<span style="color:#c92c2c">2</span> test01 running</code></span></span></span></span>备注：
迁移后，test01运行在kvm02上
6：将kvm02挂起，模拟故障
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><code class="language-cpp">注意：
如果kvm01上对test01有约束，需要在kvm01上清除约束，才能将test01迁移回来
pcs resource clear test01</code></span></span></span></span>7：查看kvm01中的状态
<span style="color:#333333"><span style="background-color:#ffffff"><span style="background-color:#f7f7f7"><span style="color:#000000"><div style="text-align:start"><div><div><div><div><div><div><span style="color:var(--gpts-primary-text-color)"><span style="color:rgba(0, 0, 0, 0.88)"><span style="color:var(--gpts-black-text-color1)">解释</span></span></span></div></div><div></div><div></div></div></div></div></div></div><code class="language-cpp"><span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># pcs status
Cluster name<span style="color:#a67f59">:</span> cluster<span style="color:#a67f59">-</span>kvm
Stack<span style="color:#a67f59">:</span> corosync
Current DC<span style="color:#a67f59">:</span> <span style="color:#128b4e">kvm01</span> <span style="color:#5f6364">(</span>version <span style="color:#c92c2c">1.1</span><span style="color:#5f6364">.</span><span style="color:#c92c2c">23</span><span style="color:#a67f59">-</span><span style="color:#c92c2c">1.</span>el7_9<span style="color:#5f6364">.</span><span style="color:#c92c2c">1</span><span style="color:#a67f59">-</span><span style="color:#c92c2c">9</span>acf116022<span style="color:#5f6364">)</span> <span style="color:#a67f59">-</span> partition with quorum
Last updated<span style="color:#a67f59">:</span> Wed Aug <span style="color:#c92c2c">31</span> <span style="color:#c92c2c">21</span><span style="color:#a67f59">:</span><span style="color:#c92c2c">56</span><span style="color:#a67f59">:</span><span style="color:#c92c2c">26</span> <span style="color:#c92c2c">2022</span>
Last change<span style="color:#a67f59">:</span> Wed Aug <span style="color:#c92c2c">31</span> <span style="color:#c92c2c">21</span><span style="color:#a67f59">:</span><span style="color:#c92c2c">52</span><span style="color:#a67f59">:</span><span style="color:#c92c2c">31</span> <span style="color:#c92c2c">2022</span> by root via crm_resource on kvm01
<span style="color:#c92c2c">2</span> nodes configured
<span style="color:#c92c2c">1</span> resource instance configured
Online<span style="color:#a67f59">:</span> <span style="color:#5f6364">[</span> kvm01 <span style="color:#5f6364">]</span>
OFFLINE<span style="color:#a67f59">:</span> <span style="color:#5f6364">[</span> kvm02 <span style="color:#5f6364">]</span>
Full list of resources<span style="color:#a67f59">:</span>
<span style="color:#128b4e">test01</span> <span style="color:#5f6364">(</span>ocf<span style="color:#5f6364">::</span>heartbeat<span style="color:#a67f59">:</span>VirtualDomain<span style="color:#5f6364">)</span><span style="color:#a67f59">:</span> Started kvm01
Daemon Status<span style="color:#a67f59">:</span>
corosync<span style="color:#a67f59">:</span> active<span style="color:#a67f59">/</span>enabled
pacemaker<span style="color:#a67f59">:</span> active<span style="color:#a67f59">/</span>enabled
pcsd<span style="color:#a67f59">:</span> active<span style="color:#a67f59">/</span>enabled
<span style="color:#5f6364">[</span>root@kvm01 <span style="color:#a67f59">~</span><span style="color:#5f6364">]</span># virsh list <span style="color:#a67f59">--</span>all
Id 名称 状态
<span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span><span style="color:#a67f59">--</span>
<span style="color:#c92c2c">5</span> test01 running</code></span></span></span></span>
608
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
