lustre02: 安装

============================================
1. Required Tools and Utilities
a) e2fsprogs 
见后面server和client安装
b) which perl
c) ssh /pdsh (recommended, optional)
./configure --prefix=/usr/local  --with-ssh
make
sudo make install 
pdsh -R ssh -w $w7,$w8,$w9,$w10,$w11 hostname
pdsh -R ssh -w root@node[001-002] hostname
pdsh -w ssh:root@node[001-002] hostname
pdsh -R ssh -w $w7,$w8,$w9,$w10,$w11 date
d)NTS(Network Time Service) ）（recommended, optional）
------------------------------------
2. server:  MDS/OSS 
a) kernel
sudo rpm -ivh --aid --force kernel-2.6.18-194.17.1.el5_lustre.1.8.7.x86_64-ext4.rpm
sudo rpm -ivh --aid --force kernel-devel-2.6.18-194.17.1.el5_lustre.1.8.7.x86_64-ext4.rpm
sudo rpm -ivh --aid --force kernel-headers-2.6.18-194.17.1.el5_lustre.1.8.7.x86_64-ext4.rpm
b) modules
sudo rpm -ivh --aid --force lustre-modules-1.8.7-2.6.18_194.17.1.el5_lustre.1.8.7.x86_64-ext4.rpm
c) ldiskfs
sudo rpm -ivh --aid --force lustre-ldiskfs-3.1.6-2.6.18_194.17.1.el5_lustre.1.8.7.x86_64-ext4.rpm
d) utilities (lustre)
sudo rpm -ivh --aid --force lustre-1.8.7-2.6.18_194.17.1.el5_lustre.1.8.7.x86_64-ext4.rpm
e) e2fsprogs
rpm -qa | grep e2fsprogs  //查看e2fsprogs版本
sudo rpm -Uvh --aid --force e2fsprogs-1.41.12.2.ora1-0redhat.rhel5.x86_64.rpm
sudo rpm -Uvh e2fsprogs-devel-1.41.12.2.ora1-0redhat.rhel5.x86_64.rpm
------------------------------------
3. client: 
a) kernel  (patched client)
sudo rpm -ivh --aid --force kernel-2.6.18-194.17.1.el5_lustre.1.8.7.x86_64-ext4.rpm
b) client modules
sudo rpm -ivh lustre-client-modules-1.8.7-2.6.18_194.17.1.el5_lustre.1.8.7.x86_64.rpm 
sudo rpm -ivh lustre-client-1.8.7-2.6.18_194.17.1.el5_lustre.1.8.7.x86_64.rpm 
c) e2fsprogs
rpm -qa | grep e2fsprogs  //查看e2fsprogs版本
sudo rpm -Uvh --aid --force e2fsprogs-1.41.12.2.ora1-0redhat.rhel5.x86_64.rpm
sudo rpm -Uvh e2fsprogs-devel-1.41.12.2.ora1-0redhat.rhel5.x86_64.rpm
------------------------------------
4. Verify that the boot loader (grub.conf or lilo.conf) 
cat /etc/grub.conf
title Red Hat Enterprise Linux Server (2.6.18-194.17.1.el5_lustre.1.8.7)
root (hd0,0)
kernel /vmlinuz-2.6.18-194.17.1.el5_lustre.1.8.7 ro root=LABEL=/ elevator=deadline  quiet   console=tty1 console=ttyS1,115200 
initrd /initrd-2.6.18-194.17.1.el5_lustre.1.8.7.img
------------------
然后reboot， 新的lustre内核路径在
/boot/config-2.6.18-194.17.1.el5_lustre.1.8.7
/boot/initrd-2.6.18-194.17.1.el5_lustre.1.8.7.img
-------------------------------------------
在OSS/MDS节点上准备磁盘
注意：先查看可以用作格式化的磁盘/dev/sdd
fdisk -l
df -h
cat /etc/mtab
cat /etc/fstab
sudo umount /data2
在/etc/fstab中注释掉 #LABEL=/data2 /data2  xfs  noatime,inode64,allocsize=16m 0 0
------------------------------------
5. Configuring the Lustre File System
a)Lustre networking (LNET) by adding this line to the /etc/modprobe.conf
options lnet networks=tcp
--------------------------------
b) 加载内核模块
--------------------------------
modprobe lustre
modprobe ldiskfs
modprobe lnet 'network="tcp0(eth0)"'
--------------------------------
内核模块位于/lib/modules/$(uname -r)/kernel
查看某个内核模块信息：modinfo ****
modprobe会分析/lib/modules/$(uname -r)/modules.dep依赖性，并加载
modprobe AA
lsmod |grep AA //目前内核加载了那些模块？
modprobe -r AA
--------------------------------
c)MDS  ($w9)
Create a combined MGS/MDT file system on the block device. On the MDS node, run:
sudo mkfs.lustre --mgs --mdt --reformat --fsname=temp /dev/sdd
-------------------------------------------
Mount the combined MGS/MDT file system on the block device. On the MDS node, run:
sudo mkdir /mnt/mdt
sudo mount -t lustre /dev/sdd /mnt/mdt
--------------- 挂载成功------------
可以在cat /etc/mtab中看到 /dev/sdd /mnt/mdt lustre rw 0 0
-------------------------------------------
d)OSS 
格式化的时候会告诉mgsnode，因此，说明可以动态增加OSS节点而无需重启其他节点，比PVFS的扩展性好，牛！
($w10)
Create the OST. On the OSS node, run:
sudo mkfs.lustre --ost --reformat --fsname=temp --mgsnode=172.28.4.9@tcp0 /dev/sdd
Mount the OST. On the OSS node, run:
sudo mkdir /mnt/ost1
mount -t lustre /dev/sdd /mnt/ost1
($w11)
Create the OST. On the OSS node, run:
sudo mkfs.lustre --ost --reformat --fsname=temp --mgsnode=172.28.4.9@tcp0 /dev/sdd
Mount the OST. On the OSS node, run:
sudo mkdir /mnt/ost2
mount -t lustre /dev/sdd /mnt/ost2
-------------------------------------------
e)client
Create the client (mount the file system on the client)
sudo mkdir /lustre
sudo mount -t lustre 172.28.4.9@tcp0:/temp /lustre
------------------------------------
6. verify the file system from client( df, dd and ls)
a) df -h /lustre  or df -h
b) cd /lustre
c) sudo dd if=/dev/zero of=/lustre/zero.dat bs=4M count=2
e) ls -lsah
f) 测速
向lustre写
sudo time dd if=/dev/zero of=/lustre/zero1.dat bs=4M count=256
[sudo] password for caoj7: 
256+0 records in
256+0 records out
1073741824 bytes (1.1 GB) copied, 8.88542 seconds, 121 MB/s
0.00user 2.20system 0:08.88elapsed 24%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+213minor)pagefaults 0swaps
本地写
sudo time dd if=/dev/zero of=/data1/zero1.dat bs=4M count=256
256+0 records in
256+0 records out
1073741824 bytes (1.1 GB) copied, 0.704258 seconds, 1.5 GB/s
0.00user 0.70system 0:00.73elapsed 95%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+213minor)pagefaults 0swaps
问题：向lustre写数据的效率比向本地磁盘写要慢？为什么？
------------------------------------
7. 停止lustre
按照先client(umount /luster)、OSS、再MDS的顺序。
------------------------------------
8. 配置与管理
sudo lfs --help
---------------------
Try interactive use without arguments or use one of:
"setstripe"
"getstripe"
"pool_list"
"find"
"check"
"catinfo"
"join"
"osts"
"df"
"quotachown"
"quotacheck"
"quotaon"
"quotaoff"
"setquota"  //用户配额
"quota"
"quotainv"
"path2fid"
"help"
"exit"
"quit"
----------------------------------
lfs setquota -u 用户名 -g 组名 -b 总容量（软） -B 总容量（硬） /lustre
lfs df -h