及配置准备
安装
中放在/soft_ins下,当然也可以建一个 NFS 服务器共享安装。
[NSD1][root][/home/scripts/gpfs]>rcp_file.sh /home/scripts/gpfs/install_gpfs.sh
[NSD1][root][/home/scripts/gpfs]> run_cmd.sh /home/scripts/gpfs/install_gpfs.shITPUB个人空间r]+zC|3Q
确认每台服务器安装显示 ok
[NSD1][root][/home/scripts/gpfs]>run_cmd.sh /home/scripts/gpfs/chang_profile.shITPUB个人空间 F1@ @ R:N
[NSD1][root][/home/scripts/gpfs]>run_cmd.sh mkdir /share
[NSD1][root][/home/scripts/gpfs]>run_cmd.sh ln -s /share /tmp/mmfs
[NSD1][root][/home/scripts/gpfs]>run_cmd.sh mkdir /tmp/gpfsITPUB个人空间_$D_6bd
ITPUB个人空间0L$ke\8b(I&zmH
ITPUB个人空间:`6j\-Lk7B7Y i.v
r6mzF"G[0
)ARe ve'y#S+B/[ {0
Y8]t6\BS)zX'q8G0ITPUB个人空间4ll"m^+L6Wz
[NSD1][root][/]>vi /tmp/gpfs/nodefile
NSD1:quorum
NSD2:quorum
App1:client
App2:client
App3:client
App4:client
2q$I8h c Q:SU b0
先确保清理干净原先 GPFS 集群
[NSD1][root][/tmp/gpfs]>mmdelnode -f
2o6sGvH]F&r0
建立集群
[NSD1][root][/tmp/gpfs]>mmcrcluster -C bgbcrun -U bgbc
-N /tmp/gpfs/nodefile -p NSD1 -s NSD2
Thu Jun 28 15:42:57 BEIST 2007: 6027-1664 mmcrcluster: Processing node NSD1
Thu Jun 28 15:42:57 BEIST 2007: 6027-1664 mmcrcluster: Processing node NSD2
…..
mmcrcluster: Command successfully completed
mmcrcluster: 6027-1371 Propagating the cluster configuration to all
affected nodes. This is an asynchronous process.
其中参数含义
-C bgbcrun 设定集群名称
-U bgbc 定义域名
-N /tmp/gpfs/nodefile 指定节点文件名
-p NSD1 指定主NSD服务器为 NSD1
-s NSD1 指定备NSD服务器为 NSD1
Cn-^ e0y dk0
[NSD1][root][/tmp/gpfs]>mmlscluster
GPFS cluster information
========================
GPFS cluster name: bgbcrun.NSD1
GPFS cluster id: 739157013761844865
GPFS UID domain: bgbc
Remote shell command: /usr/bin/rsh
Remote file copy command: /usr/bin/rcp
GPFS cluster configuration servers:
-----------------------------------
Primary : NSD1
Secondary server: NSD2
Node Daemon node name IP address Admin node name Designation|-------10--------20--------30--------40--------50--------60--------70--------80--------9||-------- XML error: The previous line is longer than the max of 90 characters ---------|-----------------------------------------------------------------------------------------
1 NSD1 10.66.3.98 NSD1 quorum
2 NSD2 10.66.3.99 NSD2 quorum
3 App1 10.66.5.51 App1
4 App2 10.66.5.52 App2
5 App3 10.66.5.53 App3
6 App4 10.66.5.54 App4ITPUB个人空间7?N0R$@IDH
W'\i]lz s)I0
)Ks?7K `XWK0
ITPUB个人空间Q-h\}/|7LG8X
ITPUB个人空间S7E;N#Jou9Oq
ITPUB个人空间/P'P? u[(o!HITPUB个人空间hMh+sx\
[NSD1][root][/tmp/gpfs]>vi /tmp/gpfs/nsdfile
添加
hdisk2:NSD1:NSD2: dataAndMetadata:4ITPUB个人空间A'vuOu
[NSD1][root][/tmp/gpfs]>mmcrnsd -F /tmp/gpfs/nsdfile
mmcrnsd: Processing disk hdisk2
mmcrnsd: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.ITPUB个人空间K'jA0t*k#}A
此时,该文件作了转换
[NSD1][root][/tmp/gpfs]>cat nsdfile
# hdisk2:NSD2:NSD1: dataAndMetadata:4
gpfs1nsd:::dataAndMetadata:4:
ZZ,w4B$kYbVd0
[NSD1][root][/tmp/gpfs]>lspv
hdisk3 00003e846ffa7a6e gpfs1nsd
tU3V(H'MsOU} M4r0
[NSD2][root][/tmp/gpfs]>mmstartup -a
Thu Jun 28 15:52:12 BEIST 2007: 6027-1642 mmstartup: Starting GPFS ...
NSD2: 6027-2114 The GPFS subsystem is already active.
…
App4: 6027-2114 The GPFS subsystem is already active.
j8Auy5Yo#f0ITPUB个人空间X:Hc7\a/DXbR^W2Z
JQ7jz n wgg`0
ITPUB个人空间{/b7E[c;Qfk@hN(Jl
ITPUB个人空间#C!h'O|0f2z1g_ S
$oK)Dg.fM0ITPUB个人空间 ^MST*kn
[NSD2][root][/]>mmcrfs /share sharelv -F /tmp/gpfs/nsdfile -A yes -B 64K -n 30 -v no
GPFS: 6027-531 The following disks of sharelv will be formatted on node NSD1:
gpfs1nsd: size 67108864 KB
GPFS: 6027-540 Formatting file system ...
GPFS: 6027-535 Disks up to size 140 GB can be added to storage pool 'system'.
Creating Inode File
Creating Allocation Maps
Clearing Inode Allocation Map
Clearing Block Allocation Map
GPFS: 6027-572 Completed creation of file system /dev/sharelv.
mmcrfs: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
参数含义如下:
/share 文件系统 mount 点名
sharelv 指定文件系统 lv 名
-F 指定 NSD 的文件名
-A 自动 mount 选项为 yes
-B 块大小为64K
-n 挂载文件系统的节点估计数30个
-v 校验建立磁盘是否已有文件系统 为否ITPUB个人空间|jp*O"S?1P6`@
[NSD2][root][/home/scripts/gpfs]>run_cmd.sh mount /shareITPUB个人空间&FMO"r(gI%i
[App1][root][/]>mkdir /share/user1work
[App1][root][/]>chown user1:bea /share/user1work
[App1][root][/]>chmod 700 /share/user1work
[App1][root][/]>mkdir /share/user1temp
[App1][root][/]>chown user1:bea /share/user1temp
[App1][root][/]> chmod 750 /share/user1tempITPUB个人空间iTe c h i
同样在另3台机器建立其他目录并修改相应所有者即权限,即 work 目录只有所有者可操作,temp 目录可读可进入。ITPUB个人空间 P\#F*d9Uy+g:G
ITPUB个人空间0jMSGY,sA5M'`
ITPUB个人空间]]ZgU@4P9j
ITPUB个人空间vg{S1nH0g9kb7t_J
5h3a6zrTRc0
在节点开机后自动启动 GPFS
[NSD1][root][/]>mmchconfig autoload=yes
mmchconfig: Command successfully completed
mmchconfig: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
;X,}G9[#F,ha.Sy?3Y0
采用多个 Quorum 盘
[NSD1][root][/]> mmchconfig singleNodeQuorum=no
mmchconfig: 6027-1119 Obsolete option: singleNodeQuorum
"^e u!b/Jz7JR0
[NSD1][root][/]>mmlsconfig
Configuration data for cluster bgbcrun.NSD2:
-----------------------------------------------
clusterName bgbcrun.NSD2
clusterId 739157013761844865
clusterType lc
autoload no
useDiskLease yes
uidDomain bgbc
maxFeatureLevelAllowed 912
/W"Iz_4X0
至此 GPFS 的安装配置完成
[NSD1][root][/]>mmgetstate -a
Node number Node name GPFS state
------------------------------------------
1 NSD1 active
2 NSD2 active
3 App1 active
4 App2 active
5 App3 active
10 App4 activeITPUB个人空间MG].u'oN.E\)AQ
Sd]Y9LRA.g(s1w3{0
] n\#M0mYg![0
ITPUB个人空间({&C8~y(G)of"Y
C`Y d:_Z0
/fmO8J.n-v0
最终,通过 userid 的不同,在 App1 的共享目录 /share下,user1 只能读写自己文件夹的文件,对其它用户的 temp 目录可读,对其它用户的 work 目录不能读写。
[App1][/share]>ls -latr
drwx------ 2 user1 bea 2048 Jun 28 17:21 user1work
drwxr--x-- 2 user1 bea 2048 Jun 28 17:21 user1temp
drwx------ 2 502 bea 2048 Jun 28 17:22 user2work
drwxr--x--- 2 502 bea 2048 Jun 28 17:22 user2temp
drwx------ 2 503 bea 2048 Jun 28 17:22 user3work
drwxr-x-- 2 503 bea 2048 Jun 28 17:22 user3temp
drwxr---- 2 504 bea 2048 Jun 28 17:22 user4work
drwxr-x--- 2 504 bea 2048 Jun 28 17:23 user4temp
[App1][user1][/share]>cd user2work
ksh: user2temp : Permission denied.
[App1][user1][/share]>cd user2temp
[App1][user1][/share]>>a
The file access permissions do not allow the specified action.
ksh[2]: a: 0403-005 Cannot create the specified file.
[App1][user1][/share]>cd /user1temp
[App1][user1][/share/user1temp]>>a
[App1][user1][/share/user1temp]>ls -l
-rw-r----- 1 user1 bea 0 Aug 13 18:47 a
[App1][user1][/share/xhtemp]>rm a
[App1][user1][/share/xhtemp]>ls -l
[App1][/share]>ls -latr
drwxr------ 2 user1 bea 2048 Jun 28 17:21 user1work
drwxr------ 2 user1 bea 2048 Jun 28 17:21 user1temp
drwxr------ 2 502 bea 2048 Jun 28 17:22 user2work
drwxr------ 2 502 bea 2048 Jun 28 17:22 user2temp
drwxr------ 2 503 bea 2048 Jun 28 17:22 user3work
drwxr------ 2 503 bea 2048 Jun 28 17:22 user3temp
drwxr------ 2 504 bea 2048 Jun 28 17:22 user4work
drwxr------ 2 504 bea 2048 Jun 28 17:23 user4temp
-r}l#b-W5b)m0
同样,在其它3台机器也有类似我们希望的结果。
!P(RD6T*A+O g:c5[+^ }0
[ y)n.{zX2h1f0
5v {N ` V q0
(H!V*v mB:O;\K,N7w0
1.新增一个节点
除所有准备和安装工作外,还需完成以下操作:
[NSD1][root][/home/scripts/gpfs]>mmaddnode -N bgbcw14:client
Thu Jun 28 16:28:21 BEIST 2007: 6027-1664 mmaddnode: Processing node App3
mmaddnode: Command successfully completed
mmaddnode: 6027-1371 Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.ITPUB个人空间i&W]1p}M tQ
2.停掉节点
停掉所有节点
[NSD2][root][/home/scripts/gpfs]>mmshutdown -a
Mon Jul 30 09:56:02 BEIST 2007: 6027-1341 mmshutdown: Starting force unmount of
GPFS file systems
NSD1: forced unmount of /share
…
App4: forced unmount of /share
Mon Jul 30 09:56:07 BEIST 2007: 6027-1344 mmshutdown: Shutting down GPFS daemons
NSD1: Shutting down!
…
App3: Shutting down!
NSD1: 'shutdown' command about to kill process 368890
….
App4: 'shutdown' command about to kill process 474040
Mon Jul 30 09:56:13 BEIST 2007: 6027-1345 mmshutdown: Finished
4h1l \B6L P0v"yk g#J4G0
也可以用 mmshutdown -N 只停某个节点
清除 GPFS
1.fuser –kcu /share
2.unmount /share #在所有节点
3.mmdelfs sharelv,
4.mmlsfs sharelv #检查结果
5.mmdelnsd –F /tmp/gpfs/nsdfile
6.mmshutdown –a
7.mmdelnode –n /tmp/gpfs/nodefile
8.mdelnode –f #最后清除集群