glusterfs部署及应用tips

最新推荐文章于 2024-08-01 11:21:51 发布

hicode666

最新推荐文章于 2024-08-01 11:21:51 发布

阅读量241

点赞数

本文链接：https://blog.csdn.net/hicode666/article/details/104281438

版权

目标

在Linux集群上部署分布式文件系统glusterfs。构建出一个可以直接导出的NFS目录，可以在客户机上mount这个导出的目录，从而方便地提供文件存储服务。

关键词

gluster， glusterfs，分布式文件系统

测试平台

centOS 7

部署脚本

!/bin/bash

set -x
HOME=/home/hicode
cp $HOME/glusterfs.repo /etc/yum.repos.d/glusterfs.repo
yum install -y gluster glusterfs-server gluseterfs-libs
systemctl enable glusterd.service && systemctl start glusterd.service
systemctl status glusterd

centOS默认的repo里面没有gluster相关的包。需要先建立下面这个gluster repo信息

[rooeet@runrr2 ~]# cat /etc/yum.repos.d/glusterfs.repo
[myglusterfs]
name=glusterfs
baseurl=https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-5/
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql

安装与启动服务

yum install -y gluster glusterfs-server gluseterfs-libs
systemctl enable glusterd.service && systemctl start glusterd.service
systemctl status glusterd

集群内多机器的自动部署

#!/bin/bash

#Declare an array of string with type
#"192.168.129.2"

declare -a hostList=(
"192.168.129.3"
"192.168.129.5"
"192.168.129.7"
)

zz=deployFS.sh
repoCfg=/etc/yum.repos.d/glusterfs.repo
fProbe=peerProbe.sh
passwd=abcxxx


set -x

setSSH() {
for host in ${hostList[@]}; do
    echo "===========set ssh login without passwd on host $host ==========="
    echo ${passwd}  | ssh  -o StrictHostKeyChecking=no -tt hicode@$host  'mkdir -p ~/.ssh/ && touch ~/.ssh/authorized_keys && chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys'  && cat ~/.ssh/id_rsa.pub | ssh hicode@$host 'cat >> .ssh/authorized_keys'
done
}

#install glusterfs and start service
deployService() {
for host in ${hostList[@]}; do
    echo "===========set ssh login without passwd on host $host ==========="
    echo ${passwd}  | ssh -tt hicode@$host  'mkdir -p ~/.ssh/ && touch ~/.ssh/authorized_keys && chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys'  && cat ~/.ssh/id_rsa.pub | ssh hicode@$host 'cat >> .ssh/authorized_keys'
    scp ~/$zz hicode@$host:~/
    scp $repoCfg hicode@$host:~/
    echo ${passwd}  | ssh -tt hicode@$host 'sudo /bin/bash ~/deployFS.sh'
done
}

#peer probe
peerProbe() {
probeDone=0
for host in ${hostList[@]}; do
    scp $fProbe hicode@$host:~/
    if [[ probeDone == 0 ]];then
        echo ${passwd}  | ssh -tt hicode@$host 'sudo /bin/bash ~/peerProbe.sh'
        probeDone = 1
    fi
    rm -rf  /home/xqfs
    echo ${passwd}  | ssh -tt hicode@$host 'sudo rm -rf /home/xqfs/gluster/b0'
    echo ${passwd}  | ssh -tt hicode@$host 'sudo mkdir -p /home/xqfs/gluster/b0/'
    #ssh -tt hicode@$host 'mkdir -p /home/xqfs/gluster/b0'
done
}

cleanup() {
for host in ${hostList[@]}; do
        echo ${passwd}  | ssh -tt hicode@$host 'sudo rm -rf ~/peerProbe.sh'
        echo ${passwd}  | ssh -tt hicode@$host 'sudo rm -rf ~/deployFS.sh'
done
}

排错指南

Peer Rejected

Peer Rejected is a state a peer can be in. If you see a peer in this state when you run ‘gluster peer status’ that means volume configuration on that peer is out of sync with the rest of the cluster. Fixing this is pretty easy…
On the rejected peer:
Stop glusterd
In /var/lib/glusterd, delete everything except glusterd.info (the UUID file)
Start glusterd
Probe one of the good peers
Restart glusterd, check ‘gluster peer status’
You may need to restart glusterd another time or two, keep checking peer status.

Try the whole procedure a couple more times if it doesn’t work right away.

防火墙开启导致主机无法被probe

peer probe可以用对方主机的IP或者hostname去probe，如果使用hostname，那么请注意/etc/hosts文件里面一定要有相对应的表项，否则会出错。
如果被probe的机器启用了防火墙并且对应端口不允许访问，就会遇到Transport endpoint is not connected的问题，解决方法是关闭防火墙或者打开对应的端口。

(base) [root@run03 hicode]# gluster
gluster> peer probe  192.168.129.22
peer probe: success.
gluster> peer status
Number of Peers: 1

Hostname: 192.168.129.22
Uuid: 4db3992f-9f5c-439f-88d8-b5f0443a9400
State: Peer in Cluster (Connected)
gluster> volume status
No volumes present
gluster> peer probe  192.168.129.24
peer probe: failed: Probe returned with Transport endpoint is not connected

有防火墙情况下的端口操作

glusterfs的端口

Note: the brick ports have changed since version 3.4.
24007 – Gluster Daemon
24008 – Management
24009 and greater (GlusterFS versions less than 3.4) OR
49152 (GlusterFS versions 3.4 and later) – Each brick for every volume on your host requires it’s own port. For every new brick, one new port will be used starting at 24009 for GlusterFS versions below 3.4 and 49152 for version 3.4 and above. If you have one volume with two bricks, you will need to open 24009 – 24010 (or 49152 – 49153).
38465 – 38467 – this is required if you by the Gluster NFS service.

The following ports are TCP and UDP:
129 – portmapper