gpfs5.1安装

Env：centos7.6

软件包：https://pan.baidu.com/s/1qqwbfg3EuCUGKJ7nkLeb7Q 
提取码：6lhi 
 

准备工作：

/etc/hosts
节点设置无密钥访问

查看kernel版本

[root@io01 ~]# rpm -aq | grep kernel
kernel-tools-3.10.0-957.el7.x86_64
kernel-headers-3.10.0-957.el7.x86_64
kernel-tools-libs-3.10.0-957.el7.x86_64
kernel-3.10.0-957.el7.x86_64

在 Linux 节点上准备环境：

[root@io01 ~]# uname -opr
3.10.0-957.el7.x86_64 x86_64 GNU/Linux

创建一个GPFS目录：

[root@io01 ~]# mkdir GPFS.conf

将GPSF安装包放到对应目录下：

Spectrum_Scale_Advanced-5.1.1.1-x86_64-Linux-install

[root@io01 GPFS.conf]# ls
Spectrum_Scale_Advanced-5.1.1.1-x86_64-Linux-install


[root@io01 GPFS.conf]# ll Spectrum_Scale_Advanced-5.1.1.1-x86_64-Linux-install
-rwxr-xr-x. 1 root root 1411264833 Dec 29 11:04 Spectrum_Scale_Advanced-5.1.1.1-x86_64-Linux-install

如果它不可执行，您可以通过发出chmod +x 命令使其可执行

要获取自解压安装映像中所有包的列表，您可以发出以下命令（指定--manifest选项）：

[root@io01 GPFS.conf]# ./Spectrum_Scale_Advanced-5.1.1.1-x86_64-Linux-install --manifest

运行时，输出解释了如何接受协议：

[root@io01 GPFS.conf]# ./Spectrum_Scale_Advanced-5.1.1.1-x86_64-Linux-install
<...Last few lines of output...>
Press Enter to continue viewing the license agreement, or
enter "1" to accept the agreement, "2" to decline it, "3"
to print it, "4" to read non-IBM terms, or "99" to go back
to the previous screen.
1

[root@io01 GPFS.conf]# ls /usr/lpp/mmfs/5.1.1.1/license/
Chinese_TW.txt  Indonesian.txt       Polish.txt
Chinese.txt     Italian.txt          Portuguese.txt
Czech.txt       Japanese.txt         Russian.txt
English.txt     Korean.txt           Slovenian.txt
French.txt      Lithuanian.txt       Spanish.txt
German.txt      non_ibm_license.txt  Turkish.txt
Greek.txt       notices.txt

[root@io01 ~]# cd /usr/lpp/mmfs/5.1.1.1/gpfs_rpms/
[root@io01 gpfs_rpms]# rpm -ivh gpfs.base*.rpm gpfs.gpl*rpm gpfs.license.adv*.rpm gpfs.gskit*rpm gpfs.msg*rpm gpfs.adv*rpm gpfs.crypto*rpm

[root@io01 gpfs_rpms]# yum install net-tools python3 m4 ksh-20120801-139.el7.x86_64

使用 mmbuildgpl 命令在 Linux 节点上构建 GPFS 可移植层:

[root@io01 ~]# /usr/lpp/mmfs/bin/mmbuildgpl

报错提示安装依赖包：

[root@io01 GPFS.conf]# yum -y install kernel-devel gcc-c++

再次执行命令：

[root@io01 GPFS.conf]# /usr/lpp/mmfs/bin/mmbuildgpl

在 Linux 节点上手动安装IBM Spectrum Scale并部署协议:

[root@io01 gpfs_rpms]# systemctl stop firewalld
[root@io01 gpfs_rpms]# systemctl disabeld firewalld
[root@io01 gpfs_rpms]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

Dec 30 10:47:22 io01 systemd[1]: Starting firewalld - dynamic firewall daemon...
Dec 30 10:47:23 io01 systemd[1]: Started firewalld - dynamic firewall daemon.
Dec 30 12:05:49 io01 systemd[1]: Stopping firewalld - dynamic firewall daemon...
Dec 30 12:05:50 io01 systemd[1]: Stopped firewalld - dynamic firewall daemon.

允许用户执行IBM Spectrum Scale命令:

[root@io01 ~]# cat .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH
export PATH=$PATH:$HOME/bin:/usr/lpp/mmfs/bin
[root@io01 ~]# source .bash_profile

创建集群:

[root@io01 GPFS.conf]# mmcrcluster -N NodesList --ccr-enable -r /usr/bin/ssh -R /usr/bin/scp -C nju.GPFS

mmcrcluster: Performing preliminary node verification ...

io01-ib:  Warning: Permanently added 'io01-ib,10.10.10.49' (ECDSA) to the list of known hosts.

mmcrcluster: Processing quorum and other critical nodes ...

mmcrcluster: Finalizing the cluster data structures ...

mmcrcluster: Command successfully completed

mmcrcluster: Warning: Not all nodes have proper GPFS license designations.

    Use the mmchlicense command to designate licenses as needed.

mmcrcluster: mmsdrfs propagation completed.

NodesList是一个文件，其中包含要添加到集群中的节点和节点名称的列表，其内容如下：

[root@io01 GPFS.conf]# cat NodesList
io01-ib:quorum-manager

接受适用节点的服务器许可证:

[root@io01 GPFS.conf]# mmchlicense server --accept -N io01-ib

The following nodes will be designated as possessing server licenses:
io01-ib
mmchlicense: Command successfully completed
mmchlicense: mmsdrfs propagation completed.

IBM Spectrum Scale 集群现已创建 mmlscluster命令查看集群的配置信息

[root@io01 GPFS.conf]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         nju.GPFS
  GPFS cluster id:           12919536918232642001
  GPFS UID domain:           nju.GPFS
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR

 Node  Daemon node name  IP address   Admin node name  Designation
-------------------------------------------------------------------
   1   io01-ib           10.10.10.49  io01-ib          quorum-manager

启动集群：

[root@io01 GPFS.conf]# mmstartup
Thu Dec 30 12:22:53 CST 2021: mmstartup: Starting GPFS ...

查看集群状态：

[root@io01 GPFS.conf]# mmgetstate

 Node number  Node name        GPFS state  
-------------------------------------------
       1      io01-ib          active

在 Linux 系统上安装IBM Spectrum Scale 的过程中创建 NSD 和文件系统：

[root@io01 GPFS.conf]# vi NSD_filename

%nsd:
                        device=/dev/sdb
                        nsd=nsd1
                        servers=io01-ib
                        usage=dataAndMetadata
                        failureGroup=-1
                        pool=system

%nsd:
                        device=/dev/sdc
                        nsd=nsd2
                        servers=io01-ib
                        usage=dataAndMetadata
                        failureGroup=-1
                        pool=system

%nsd:
                        device=/dev/sdd
                        nsd=nsd3
                        servers=io01-ib
                        usage=dataAndMetadata
                        failureGroup=-1
                        pool=system

[root@io01 GPFS.conf]# mmcrnsd -F NSD_filename
mmcrnsd: Processing disk sdb
mmcrnsd: mmsdrfs propagation completed.

[root@storage GPFS.conf]# mmcrfs storage -F NSD_filename -A yes -B 4M -T /storage

The following disks of storage will be formatted on node storage:

    nsd1: size 15259648 MB

Formatting file system ...

Disks up to size 129.25 TB can be added to storage pool system.

Creating Inode File

  48 % complete on Wed Dec 29 18:18:44 2021

  98 % complete on Wed Dec 29 18:18:49 2021

 100 % complete on Wed Dec 29 18:18:49 2021

Creating Allocation Maps

Creating Log Files

  81 % complete on Wed Dec 29 18:18:54 2021

 100 % complete on Wed Dec 29 18:18:55 2021

Clearing Inode Allocation Map

Clearing Block Allocation Map

Formatting Allocation Map for storage pool system

Completed creation of file system /dev/storage.

mmcrfs: mmsdrfs propagation completed.

[root@storage GPFS.conf]# mmmount all

Wed Dec 29 18:20:39 CST 2021: mmmount: Mounting file systems ...

[root@storage GPFS.conf]# df -h

Filesystem               Size  Used Avail Use% Mounted on

devtmpfs                  63G     0   63G   0% /dev

tmpfs                     63G     0   63G   0% /dev/shm

tmpfs                     63G   12M   63G   1% /run

tmpfs                     63G     0   63G   0% /sys/fs/cgroup

/dev/mapper/centos-root  542G   15G  527G   3% /

/dev/sdp2               1014M  151M  864M  15% /boot

/dev/sdp1                200M   12M  189M   6% /boot/efi

tmpfs                     13G     0   13G   0% /run/user/0

/dev/loop0               9.5G  9.5G     0 100% /iso

storage                   15T  6.4T  8.3T  44% /storage

报错：

2022-08-05_14:48:27.509+0800: [X] The daemon memory configuration hard floor of 1536 MiB is above the system memory configuration of 972 MiB. GPFS is shutting down. Ensure the system has enough memory for the daemon to run.

2022-08-05_14:48:27.814+0800: mmremote: Initiating GPFS shutdown ...

2022-08-05_14:48:30.828+0800: mmremote: Starting the mmsdrserv daemon ...

2022-08-05_14:48:30.847+0800: mmremote: Unloading GPFS kernel modules ...

2022-08-05_14:48:30.992+0800: mmremote: Completing GPFS shutdown ...

内存配置太低，最少2G