主机系统:rhel6.0 (数量不限)
系统环境:selinux and iptables disabled
服务节点: desktop26.example.com 192.168.0.26
计算节点: desktop80.example.com 192.168.0.80
desktop83.example.com 192.168.0.83
1. 使用SSH密钥让各个节点做到无需密码连接
先要在每个节点都要新建一个普通用户用作以后的集群用户
- useradd yejk
- passwd yejk (密码设为westos)
注意每台节点上的这个普通用户的密码和ID必须相同
这里要注意,无论有多少个节点,这个普通用户家目录里的数据都应该完全相同,为了达到这一点可以用一些网络存储实现,这里使用NFS实现,因为实现方法简单且工作稳定
在任意一个节点上部署NFS将yejk的家目录发布出来,其他节点直接挂载此家目录
- yum install nfs-utils rpcbind -y
- vim /etc/exports
- /home/yejk 192.168.0.0/24(rw,no_root_squash)
- exportfs -r #使配置生效
- exportfs -v #查看
- /etc/init.d/rpcbind start #需要rpcbind支持
- /etc/init.d/nfs start
- mount 192.168.0.80:/home/yejk /home/yejk
- vim /etc/fstab
- 192.168.0.80:/home/yejk /home/yejk nfs defaults 0 0
此时在任意节点上的操作就可以在每个节点上生效
- su - yejk
- ssh-keygen 一路回车即可
- ssh-copy-id -i ~/.ssh/id_rsa.pub desktop80.example.com(与自己信任连接)
- ssh-copy-id -i ~/.ssh/id_rsa.pub desktop83.example.com(与其他节点信任连接)
- sh-copy-id -i ~/.ssh/id_rsa.pub desktop26.example.com(与其他节点信任连接)
2.在每个节点上安装mpich2
- yum install mpich2 -y
- su - yejk
- cd /home/yejk
创建.mpd.conf文件 (文件前加".")
- vim .mpd.conf
- secretword=westos #节点的密码,每台节点都得相同
- chmod 600 .mpd.conf
创建集群节点集合文件mpd.hosts
- vim mpd.hosts
- desktop80.example.com
- desktop83.example.com
注:如果以root用户启动mpd服务, 把mpd.conf文件创建在/etc目录中(不加"."),mpd.hosts放在root主目录中
3.1测试mpich2:
启动mpich2
- mpd
观看启动机器:
- mpdtrace
- desktop80.example.com
退出:
- mpdallexit
3.2运行集群系统
- mpdboot -n 2 -f mpd.hosts 参数-n 2指定了要起动的机器个数,-f mpd.hosts指定了运行mpd.hosts文件中指定的节点。
- mpdtrace
- desktop80.example.com
- desktop83.example.com
- mpdallexit
3.3单机测试
- ./icpi-64
- Enter the number of intervals: (0 quits) 1000000000
- pi is approximately 3.1415926535921401, Error is 0.0000000000023470
- wall clock time = 46.571311
- Enter the number of intervals: (0 quits) 10000
- pi is approximately 3.1415926544231341, Error is 0.0000000008333410
- wall clock time = 0.000542
- Enter the number of intervals: (0 quits) 0
3.4 集群测试
- mpdboot -n 2 -f mpd.hosts
- mpiexec -n 2 /root/icpi-64
- Enter the number of intervals: (0 quits) 1000000000
- pi is approximately 3.1415926535899761, Error is 0.0000000000001830
- wall clock time = 15.530082
- Enter the number of intervals: (0 quits) 10000
- pi is approximately 3.1415926544231323, Error is 0.0000000008333392
- wall clock time = 0.006318
- Enter the number of intervals: (0 quits) 0
- mpdallexit
由上述测试的结果可以看出使用单机单进程运行,intervals设为1000000000,耗时46.571311秒,而用两台机器双进程则只有15.530082秒,明显快很多,并行运算还是
很有效果的。不过,如果我们把intervals改为10000,单机运行只用了0.000542秒,而两台机器却花了0.006318秒,这是因为并行运算过程中,参与运算的机器需要通过网
络传递一些消息,如果计算量不大的话,花在了这上面的时间影响会比较明显,因而反不如单机版的来得快
4. 排错
- mpdcheck获得帮助信息
- mpdcheck -pc
- mpdcheck -l
- 通过mpd.hosts文件查错
- mpdcheck -f mpd.hosts
- mpdcheck -f mpd.hosts -ssh
- 对任意两台机器进行查错
- Station11上:
- mpdcheck -s
- 返回主机名host和端口port
- server listening at INADDR_ANY on: station11 52576
- station12上:
- mpdcheck -c 192.168.0.1 40782
- client successfully recvd ack from server: ack_from_server _to_client
- station11 上返回消息传递结果
- server has conn on from ('192.168.0.12', 54438)
- server successfully recvd msg from client: hello_from_client_to_server
- station1上:
- mpd -e &
- 返回使用的端口
- [1]12703
- mpd_port=42498
- station12上:
- mpd -h station11 -p 41563 &
- [1]5122
torque作业调度系统
- yum install gcc gcc-c++ make -y
- tar zxf torque-3.0.0.tar.gz
- cd torque-3.0.0
- ./configure --with-rcp=scp --with-default-server=desktop26.example.com (--with-rcp=scp用于ssh key方式)
- make
- make install (torque的配置目录: /var/spool/torque)
- make packages (生成计算节点安装包,即在station11和station12上安装的包,确保所有计算节点和服务节点的架构是相同的)
- cp contrib/init.d/pbs_server /etc/init.d/
- cp contrib/init.d/pbs_sched /etc/init.d/
- scp contrib/init.d/pbs_mom desktop80.example.com:/etc/init.d
- scp contrib/init.d/pbs_mom desktop83.example.com:/etc/init.d
- vi /var/spool/torque/server_priv/nodes (设定计算节点,服务节点也可做计算)
- desktop80.example.com
- desktop83.example.com
- ./torque.setup root (设置torque的管理帐户)
- qterm -t quick (停止torque)
- service pbs_server start (启动torque)
- service pbs_sched start (启动调度程序)
- scp torque-package-clients-linux-x86_64.sh torque-package-mom-linux-x86_64.sh desktop80.example.com:~
- scp torque-package-clients-linux-x86_64.sh torque-package-mom-linux-x86_64.sh desktop83.example.com:~
在所有计算节点安装torque,在所有计算节点执行如下命令:
- ./torque-package-clients-linux-x86_64.sh --install
- ./torque-package-mom-linux-x86_64.sh --install
注:如果计算节点的架构和服务节点不同,安装方法如下:
- tar zxf torque-3.0.0.tar.gz
- ./configure --with-rcp=scp --with-default-server=server1.example.com
- make
- make install_mom install_clients
- vi /var/spool/torque/mom_priv/config (所有计算节点执行此配置)
- $pbsserver server1.example.com
- $logevent 255
- service pbs_mom start (所有计算节点执行此命令,启动计算节点守护进程)
测试
torque的调度需要使用非root用户
- su - yejk
- cd /home/yejk
- vim job1.pbs (串行作业)
- #!/bin/bash
- #PBS -N job1
- #PBS -o job1.log
- #PBS -e job1.err
- #PBS -q batch
- cd /home/yejk
- echo Running on hosts `hostname`
- echo Time is `date`
- echo Directory is $PWD
- echo This job runs on the following nodes:
- cat $PBS_NODEFILE
- echo This job has allocated 1 node
- ./prog
- vi job2.pbs (并行作业)
- #!/bin/bash
- #PBS -N job2
- #PBS -o job2.log
- #PBS -e job2.err
- #PBS -q batch
- #PBS -l nodes=2
- cd /home/yejk
- echo Time is `date`
- echo Directory is $PWD
- echo This job runs on the following nodes:
- cat $PBS_NODEFILE
- NPROCS=`wc -l < $PBS_NODEFILE`
- echo This job has allocated $NPROCS nodes
- mpiexec -machinefile $PBS_NODEFILE -np $NPROCS ./prog
- vi prog
- #!/bin/bash
- echo 1000000000 | ./icpi-64 (icpi程序是mpi自带的,拷贝过来即可)
- chmod +x prog
- qsub jobx.pbs(提交作业)
- qstat (查看作业)
- pbsnodes (查看节点)
使用单节点计算,串行计算截图:
查看结果
- [yejk@desktop26 ~]$ cat job1.log
- Running on hosts desktop80.example.com
- Time is Sat Jun 2 05:33:32 CST 2012
- Directory is /home/yejk
- This job runs on the following nodes:
- desktop80.example.com
- This job has allocated 1 node
- Enter the number of intervals: (0 quits) pi is approximately 3.1415926535899708, Error is 0.0000000000001776
- wall clock time = 43.059767
- Enter the number of intervals: (0 quits) No number entered; quitting
使用集群并行计算截图:
- [yejk@desktop26 ~]$ cat job2.log
- Time is Sat Jun 2 05:49:29 CST 2012
- Directory is /home/yejk
- This job runs on the following nodes:
- desktop83.example.com
- desktop80.example.com
- This job has allocated 2 nodes
- Enter the number of intervals: (0 quits) pi is approximately 3.1415926535900072, Error is 0.0000000000002141
- wall clock time = 23.318623
- Enter the number of intervals: (0 quits) No number entered; quitting
- You have mail in /var/spool/mail/yejk
- [yejk@desktop26 ~]$
可以看到同样的任务并行的时间用的短
注:1. 以上测试我是以yejk用户测试,确保所有节点上有此用户,并且在计算节点上的yejk用户目录中有 prog 和 icpi 两个可执行程序
2. 启动mpd程序时要使用yejk用户 因为在调度时会连接计算节点上/tmp/
mpd2.console_yejk
转载于:https://blog.51cto.com/yejiankang/885484