db2 purecale是ibm推出的基于共享磁盘的，可扩展的数据库集群软件，类似于oracle的rac。具体功能这里就不过多介绍了，直接介绍在rehdat上安装purescale的实战经历
1 安装手册
软硬件版本
IB交换机：Mellanox
PC服务器：X3850 X5
存储版本：DS5100
OS版本：RHEL6.1
DB2版本：db2 purescale10.1.0.2
建议使用purescale认证的软硬件版本，否则可能会碰到各种莫名其妙的问题
1.1 规划
| 主机名 | IP地址 | 管理 | IB地址 | CF | member |
|
| Wtydb21 | 192.168.2.101 | 192.168.1.101 | 192.168.0.101 | primary | YES |
|
| Wtydb22 | 192.168.2.102 | 192.168.1.102 | 192.168.0.102 | second | YES |
|
| Wtydb23 | 192.168.2.103 | 192.168.1.103 | 192.168.0.103 |
| YES |
|
IB地址规划
|
| 管理IP | 浮动IP | IB |
|
| Switch-1 | 192.168.1.104 | 192.168.1.106 | 192.168.0.201 |
|
| Switch-2 | 192.168.1.105 |
| 192.168.0.202 |
|
1.2 安装前环境准备
1.2.1 Nfs+yum源配置
1.2.1.1 配置NFS
Wtydb21为NFS server，其它主机为client，为了在各服务器安装软件包方便，建议配置yum
1，手工安装nfs软件包
rpm –qa nfs
修改为nfs默认打开,所有服务器设置
| [root@stydb1 ~]# chkconfig --list nfs nfs 0:off 1:off 2:off 3:off 4:off 5:off 6:off [root@stydb1 ~]# chkconfig nfs on [root@stydb1 ~]# chkconfig --list nfs nfs 0:off 1:off 2:on 3:on 4:on 5:on 6:off service portmap start service nfs start service rpcbind start |
2, 确保nfs server端防火墙处于关闭状态
service iptables stop
3，nfs server端配置
| A，Mount NFS磁盘mount nfs disks，本例共享/data目录 mount /dev/mapper/vg_1-lv_data /data B，vi /etc/fstab，添加如下一行，修改为自动挂载 /dev/mapper/vg_1-lv_data /data ext4 defaults 1 2 C，nfs共享/data目录 vi /etc/exports /data *(rw) /data wtydb22(rw) wtydb23(rw) /usr/sbin/exportfs -r |
4，客户端设置
| A，查看nfs服务器端的设置 [root@wtydb22 ~]# showmount -e wtydb21 Export list for wtydb21: /data wtydb23,wtydb22
B，vi /etc/fstab wtydb21:/data /data nfs rw rpm -qa|grep nfs C，手工mount mkdir /data mount wtydb21:/data/ /data/ |
1.2.1.2 配置yum源
1，新建目录mkidr/data/rhel6.1
cp 光盘内容到/data/rhel6.1目录，也可以直接使用光盘或者iso文件的介质文件
2，配置yum（三台服务器上配置）
vi/etc/yum.repos.d/rhel6.repo
[Server]
name=rhel6server
baseurl=file:///data/rhel6.1/Server/
enable=1
gpcheck=1
gpgkey=file:///data/rhel6.1/RPM-GPG-KEY-redhat-release
3，测试
yum list
也可通过iso文件实现
| mount -o loop /software/rhel-server-6.1-x86_64-dvd.iso /mnt/cdrom
vi /etc/yum.repos.d/rhel6.repo [Server] name=rhel6server baseurl=file:///mnt/cdrom/Server/ enable=1 gpcheck=1 gpgkey=file:///mnt/cdrom/RPM-GPG-KEY-redhat-release |
1.2.2 ssh互通
1， 生成ssh key，在~/.ssh/目录下会生成两个文件
| ssh-keygen -t dsa，一路默认回车 在.ssh目录下 id_dsa (the private key) and id_dsa.pub (the public key) for DSA encryption |
2，把三台主机的id_dsa.pub文件的内容复制到authorized_keys文件中，把authorized_keys放于三台主机的.ssh目录下
1.2.3 GPFS安装配置（可选）
正常情况下，GPFS不需要单独安装，随着安装db2 purescale软件会自动完成gpfs安装。.
1.2.3.1 手工安装
| [root@wtydb22 data]# cd /data/esefp2/server/db2/linuxamd64/gpfs [root@wtydb22 gpfs]# ls base db2ckgpfs db2gutil errMsg fp installGPFS uninstallGPFS [root@wtydb22 gpfs]# ./installGPFS -i -g "GPFS" is not installed.
DBI1070I Program installGPFS completed successfully.
|
# vi /tmp/gpfsprofile
wtydb21:quorum
wtydb22:quorum
wtydb23:quorum
[root@pure1, /]
mmcrcluster -n /tmp/gpfsprofile -p wtydb21-s wtydb22 -C gpfs_cluster -r /usr/bin/ssh -R /usr/bin/scp
# mmcrcluster -n /home/gpfs/gpfs.allnodes-p node1 -s node2 -C gpfs_cluster -r /usr/bin/ssh -R /usr/bin/scp
# cd /opt/ibm/db2/V10.1/bin
# ./db2cluster -cfs -add -license
The license for the shared file system
./db2cluster -cfs -start -all
./db2cluster -cfs -create -filesystem db2fs-disk /dev/dm-3 -mount /db2fs
1.2.3.2 安装配置GPFS
| 1， 安装gpfs软件（三台主机上运行） 本次从db2 purescale安装介质中直接安装，其它安装方法见参考文献 rpm –qa gpfs验证软件是否已经正确安装 2， 3， 修改PATH路径（三台主机上运行）
在3 台主机上修改HOME目录下的。bash_profile文件，在文件末尾增加 export PATH=$PATH:/usr/lpp/mmfs/bin 3， 新建目录,用做GPFS文件系统（三台主机上运行） mkdir /db2fs 4， 创建gpfs 群集配置文件 [root@wtydb21 tmp]# vi /tmp/gpfsprofile wtydb21:quorum-manager wtydb22:quorum-manager wtydb23:quorum-manager 5，创建集群，注意指定ssh方式 [root@wtydb21 pam.d]# [root@wtydb21 gpfs]# mmcrcluster -N /tmp/gpfsprofile -p wtydb21 -s wtydb22 -C gpfs_cluster -r /usr/bin/ssh -R /usr/bin/scp Sat Apr 6 12:17:35 CST 2013: mmcrcluster: Processing node wtydb21 Sat Apr 6 12:17:35 CST 2013: mmcrcluster: Processing node wtydb22 Sat Apr 6 12:17:38 CST 2013: mmcrcluster: Processing node wtydb23 mmcrcluster: Command successfully completed mmcrcluster: Warning: Not all nodes have proper GPFS license designations. Use the mmchlicense command to designate licenses as needed. mmcrcluster: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
mmcrcluster 命令其中参数含义 -C bgbcrun 设定集群名称 -U bgbc 定义域名 -N /tmp/gpfs/nodefile 指定节点文件名 -p NSD1 指定主NSD 服务器为 NSD1 -s NSD1 指定备NSD 服务器为 NSD1 6，接受许可协议 [root@wtydb21 pam.d]# mmchlicense server --accept -N wtydb21,wtydb22,wtydb23 7，确认创建集群状况 [root@wtydb21 ~]# mmlscluster [root@wtydb21 gpfs]# mmlscluster
GPFS cluster information ======================== GPFS cluster name: gpfs_cluster.wtydb21 GPFS cluster id: 12146727015547904479 GPFS UID domain: gpfs_cluster.wtydb21 Remote shell command: /usr/bin/ssh Remote file copy command: /usr/bin/scp
GPFS cluster configuration servers: ----------------------------------- Primary server: wtydb21 Secondary server: wtydb22
Node Daemon node name IP address Admin node name Designation ------------------------------------------------------------------- 1 wtydb21 192.168.2.101 wtydb21 quorum-manager 2 wtydb22 192.168.2.102 wtydb22 quorum-manager 3 wtydb23 192.168.2.103 wtydb23 quorum-manager 8，生成nsd 盘，使用/dev/dm-3
[root@wtydb21 etc]# vi /tmp/nsdprofile dm-3:::dataAndMetadata:: [root@wtydb21 gpfs]# mmcrnsd -F /tmp/nsdprofile mmcrnsd: Processing disk dm-3 mmcrnsd: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process.
此时系统自动修改/tmp/gpfsnsdprofile 文件内容如下： [root@wtydb21 ~]# cat /tmp/nsdprofile # dm-4:::dataAndMetadata:: gpfs1nsd:::dataAndMetadata:-1::system
9,启动集群 [root@wtydb21 /]# mmstartup -a . [root@wtydb21 src]# mmgetstate -a -L
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks ------------------------------------------------------------------------------------ 1 wtydb21 2 2 3 active quorum node 2 wtydb22 2 2 3 active quorum node 3 wtydb23 2 2 3 active 10,创建gpfs 文件系统
参数含义如下： /datafs 文件系统 mount 点名 gpfs_lv 指定文件系统 lv 名 -F 指定 NSD 的文件名 -A 自动 mount 选项为 yes -B 块大小为64K -n 挂载文件系统的节点估计数30 个 -v 校验建立磁盘是否已有文件系统 为否 11，挂载文件系统 [root@wtydb21 /]# mount /db2fs [root@wtydb21 /]# df
12，设置gpfs 开机自启 [root@wtydb21 /]# mmchconfig autoload=yes
13，查询GPFS 配置信息 [root@wtydb21 share]# mmlsconfig
[root@wtydb21 share]# mmgetstate -a |
1.2.3.3 修改参数配置
db2 purescale要求必须符合以下参数配置
| [root@wtydb21 src]# /opt/ibm/db2/V10.1/bin/db2cluster -cfs -verify -configuration The shared file system cluster option 'adminMode' has not been set to the optimal value of 'allToAll'. See the DB2 Information Center for more details. The shared file system cluster option 'maxFilesToCache' has been set too low. The value should be at least '10000'. See the DB2 Information Center for more details. The shared file system cluster option 'usePersistentReserve' has not been set to the optimal value of 'yes'. See the DB2 Information Center for more details. The shared file system cluster option 'verifyGpfsReady' has not been set to the optimal value of 'yes'. See the DB2 Information Center for more details. The shared file system cluster option 'failureDetectionTime' could not be verified. Re-issue this command after the cluster manager peer domain has been created and is online. A diagnostic log has been saved to '/tmp/ibm.db2.cluster.toBL0B'.修改过程如下 mmchconfig adminMode allToAll mmchconfig maxFilesToCache=20000 mmchconfig pagepool=1024M mmchconfig usePersistentReserve=yes mmchconfig verifyGpfsReady=yes
mmshutdown -a db2cluster -cfs -stop -all mmchconfig failureDetectionTime=35 db2cluster -cfs -start -all
mmstartup -a |
1.2.3.4 验证GPFS
db2cluster -cfs -verify -configuration
db2cluster -cfs -verify -filesystem mygpfs1
实例的所有者db2sdin1对文件系统具有访问权限
1.2.3.5 删除GPFS集群（未验证）
| 1.fuser –kcu /ods/gpfs 2. unmount /ods/gpfs #在所有节点上执行 3. mmdelfs gpfslv, 4. mmlsfs gpfslv #检查结果 5. mmdelnsd –F /ods/gpfsnsdprofile 6. mmshutdown –a 7. mmdelnode -a 8. mmdelnode –f #最后清除集群 |
Optional. For DB2 managed GPFSinstallations, verify the remote shell and
remote file copy settings default todb2locssh and db2scp. For example:
/usr/lpp/mmfs/bin/mmlscluster
Remote shell command:/var/db2/db2ssh/db2locssh
Remote file copy command:/var/db2/db2ssh/db2scp
1，卸载所有gpfs文件系统，mmumount all -a
2，删除文件系统，mmdelfs gpfs_lv
3，删除nsd节点，mmdelnsd -F /tmp/nsdprofile
4，停止gpfs系统，mmshutdown-a
5，卸载gpfs软件，先查看已经安装的软件rpm -qa|grep gpfs
rpm -e gpfs.gpl
rpm -e gpfs.msg.en_us
rpm -e gpfs.base
rpm -e gpfs.docs
6，删除目录 /var/mmfs和/usr/lpp/mmfs
7，删除/var/adm/ras目录下以mm开头的文件
8，删除/tmp/mmfs（如果存在）目录和其内容
gpfs log file path /var/adm/ras
1.2.3.6 Explanation
The verifyGpfsReady=yesconfiguration attribute is set, but the /var/mmfs/etc/gpfsreadyscript could not be executed.
1.2.3.7 User response
Make sure /var/mmfs/etc/gpfsreadyexists and is executable, or disable the verifyGpfsReadyoption via mmchconfig verifyGpfsReady=no.
ERROR: DBI20093E The DB2 installer failed to set up thedb2sshid because the
GPFS file system is a user-managed filesystem.
参考文献
1.2.4 网络环境
1.2.4.1 1，修改vi /etc/hosts文件
| [root@wtydb21 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.0.101 wtydb21-ib0 192.168.3.101 wtydb21-ib3 192.168.0.102 wtydb22-ib0 192.168.3.102 wtydb22-ib3 192.168.0.103 wtydb23-ib0 192.168.3.103 wtydb23-ib3
192.168.2.101 wtydb21 192.168.2.102 wtydb22 192.168.2.103 wtydb23 |
备注：IP和主机名之间用空格，一定不要用tab键，否则报错。lookback不要匹配主机名
1.2.4.2 2，修改主机名
第一步：
#hostnamewtydb21
第二步：
修改/etc/sysconfig/network中的hostname
第三步：
修改/etc/hosts文件
1.2.4.3 3，设置网卡绑定
eth0和eth4做网卡绑定，主备模式
使用ethtool确认网络连接状态
1，cd/etc/sysconfig/network-scripts/ 修改相关配置文件
| 新建 ifcfg-bond0 DEVICE=bond0 BOOTPROTO=static IPADDR=192.168.2.101 NETMASK=255.255.255.0 GATEWAY=192.168.2.254 ONBOOT=YES TYPE=Ethernet USERCTL=no BONDING_OPTS="miimon=100 mode=1" 修改ifcfg-eth0内容：
DEVICE=eth0 BOOTPROTO=none ONBOOT=yes USERCTL=no MASTER=bond0 SLAVE=yes
修改ifcfg-eth4内容
DEVICE=eth4 BOOTPROTO=none ONBOOT=yes USERCTL=no MASTER=bond0 SLAVE=yes |
2，vi/etc/modprobe.conf /etc/modprobe.d/bonding.conf
aliasbond0 bonding
3，重启网络配置
servicenetwork
4， 关闭networkmanger(rhel6.3版本需要，之前rhel5配置时无此步骤)
| chkconfig NetworkManager off; /etc/init.d/NetworkManager stop; chkconfig network on; /etc/init.d/network restart ifconfig bond1 up |
1.2.4.4 4，关闭相关服务：防火墙和selinux
1，chkconfig iptablesoff
service iptables stop
2，selinux
vi /etc/selinux/config
#SELINUX=enforcing
SELINUX=disabled
将文件中的SELINUX="" 为 disabled ，然后重启。
如果不想重启系统，使用命令setenforce 0
1.2.5 IB环境准备
1.2.5.1 交换机配置
With the Mellanox SM HA software, thesystem manager can enter and modify
all IB subnet configurations for the subnetmanagers from a single location.
Assign a virtual IP address (VIP) to themanagement port of the switch to
manage the high availability domain. Thesystem manager must configure all
the switches in a Mellanox SM HAenvironment to join the same IB subnet, and
assign the subnet a name. After joining thesubnet, the subnet managers are
synchronized and you must select one as themaster subnet manager, and the
others become standby subnet managers.
1.2.5.2 安装infiniband驱动软件
infiniband软件
| Install the packages required and reboot: rhel 6.3版本 [root@serv]# yum groupinstall "Infiniband Support" [root@serv]# yum install infiniband-diags perftest qperf opensm [root@serv]# chkconfig rdma on [root@serv]# chkconfig opensm on [root@serv]# shutdown -r now
|
1.2.5.3 配置IP地址
同普通以太网卡配置，例外db2数据库使用RDMA协议，而不是TCP|IP协议
| [root@wtydb22 network-scripts]# cat ifcfg-ib0 DEVICE="ib0" BOOTPROTO="static" IPADDR=192.168.0.102 #NETMASK=255.255.255.0 PREFIX=24 NM_CONTROLLED="yes" ONBOOT="yes" TYPE="InfiniBand" |
1.2.5.4 参数配置
1，log_mtts_per_seg参数
如果 CF 或成员所在的主机有超过 64 GB 的内存，那么 Mellanox HCA 驱动程序（mlx4_core）模块的参数 log_mtts_per_seg 必须从 3（默认）增加到 7，以便有更大的内存注册。
要增加大小，以根用户身份发出以下命令：
在 SUSE 上：
echo "optionsmlx4_core log_mtts_per_seg=7" >>
/etc/modprobe.conf.local
在 RHEL 上：
echo "optionsmlx4_core log_mtts_per_seg=7" >> /etc/modprobe.conf
RHEL 6 上需要运行如下命令
echo "options mlx4_corelog_mtts_per_seg=7" >> /etc/modprobe.d/modprobe.conf
要让该更改生效，您必须重启服务器。要检查您的更改在模块上是否有效，输入：
<host-name>/sys/module/mlx4_core/parameters# cat /sys/module/mlx4_core/parameters/log_mtts_per_seg
2，/etc/rdma/dat.conf配置文件
On RHEL 6.1, the DAT configuration file islocated in /etc/rdma/dat.conf
and it is updated by the group installationof the "InfiniBand Support"
package
系统默认配置好了
[root@wtydb21 Packages]# cat/etc/rdma/dat.conf
ofa-v2-mlx4_0-1 u2.0 nonthreadsafe defaultlibdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
ofa-v2-mlx4_0-2 u2.0 nonthreadsafe defaultlibdaploscm.so.2 dapl.2.0 "mlx4_0 2" ""
ofa-v2-ib0 u2.0 nonthreadsafe defaultlibdaplofa.so.2 dapl.2.0 "ib0 0" ""
……
1.2.5.5 最佳实践
网络配置
2）检查infiniband交换机的配置
a,管理软件需要最低版本
image-PPC_M405EXEFM_1.1.2500.img
b,启动子网管理SM，管理口配置VIP，互联口配置IP地址
c，配置故障转移
d,两个交换机之间需要2根以上的互连线，建议CF和member连接至交换机的总port数除以2
1.2.6 依赖软件安装
使用yum install命令在三台主机同时安装即可
yum install libstdc++
yum install pam-1.1.1
yum install pam_krb5-2.3.11
yum install pam-devel
yum install pam_pkcs11
所需软件包，详见官方安装文档说明。
| libstdc++-4.4.5-6.el6.i686, pam-1.1.1-8.el6.i686, pam_krb5-2.3.11-6.el6.i686, pam-devel-1.1.1-8.el6.i686, pam_pkcs11-0.6.2-11.1.el6.i686, pam_ldap-185-8.el6.i686 下列包rhel 6.1 均已经默认安装 For InfiniBand network type (both 32- bit and 64-bit libraries unless specified) : libibcm dapl (64-bit libraries only) ibsim (64-bit libraries only) ibutils (64-bit libraries only) libibverbs librdmacm libcxgb3 libibmad libibumad libipathverbs (64-bit libraries only) libmlx4 libmthca libnes (64-bit libraries only) libmlx4 rdma (no architecture)
ntp-4.2.4p8-2.el6.x86_64/ntpdate-4.2.4p8-2.el6.x86_64 libstdc++-4.4.5-6.el6.x86_64 libstdc++-4.4.5-6.el6.i686 glibc-2.12-1.25.el6.x86_64 glibc-2.12-1.25.el6.i686 gcc-c++-4.4.5-6.el6.x86_64 gcc-4.4.5-6.el6.x86_64 kernel-2.6.32-131.0.15.el6.x86_64 kernel-devel-2.6.32-131.0.15.el6.x86_64 kernel-headers-2.6.32-131.0.15.el6.x86_64 kernel-firmware-2.6.32-131.0.15.el6.noarch ntp-4.2.4p8-2.el6.x86_64 ntpdate-4.2.4p8-2.el6.x86_64 sg3_utils-1.28-3.el6.x86_64
sg3_utils-libs-1.28-3.el6.x86_64 binutils-2.20.51.0.2-5.20.el6.x86_64 binutils-devel-2.20.51.0.2-5.20.el6.x86_64 openssh-5.3p1-52.el6.x86_64 cpp-4.4.5-6.el6.x86_64 ksh-20100621-16.el6.x86_64
|
安装示例说明
|
[root@wtydb23 ~]# yum install compat-libstdc++-33-3.2.3-69.el6.x86_64.rpm
|
1.2.6.1 GPFS依赖软件包
另外，GPFS软件还需要I686等软件包需要安装
v For multiple communication adapter portson InfiniBand network and single or multiple communication adapter port at CFson 10GE network,
the minimum support level is RHEL 6.1.
i686 which is 32-bit packages might not getinstalled by default when installing x86_64 server. Make sure that all the32-bit dependencies are
explicitly installed. For example:
libstdc++-4.4.5-6.el6.i686,
pam-1.1.1-8.el6.i686,
pam_krb5-2.3.11-6.el6.i686,
pam-devel-1.1.1-8.el6.i686,
pam_pkcs11-0.6.2-11.1.el6.i686,
pam_ldap-185-8.el6.i686
Alternatively, run the yum command aftercreating a source from local DVD or after registering to RHN:
yum install *.i686
Ensure the following 32-bit RSCT packagesare installed:
– libibcm.i686
– libibverbs-rocee.i686
– librdmacm.i686
– libcxgb3.i686
– libibmad.i686
– libibumad.i686
– libmlx4-rocee.i686
– libmthca.i686·
DB2 pureScale Feature requireslibstdc++.so.6. Verify that the files exist with
the following commands:
ls /usr/lib/libstdc++.so.6*
ls /usr/lib64/libstdc++.so.6*
1.2.6.2 install SAM
| [root@wtydb21 tsamp]# cat /tmp/prereqSAM.1.log prereqSAM: >>> Prerequisite on wtydb21 check - log started : Fri Apr 12 14:22:16 CST 2013 prereqSAM: OPTIONS = '' prereqSAM: OPT_SILENT = 0 prereqSAM: OPT_NOLICCHECK = 0 prereqSAM: Detected operating system Linux prereqSAM: Detected architecture i386x prereqSAM: Detected distribution RH prereqSAM: Supported operating system versions RH Linux i386x - 5.0 6.0 prereqSAM: Detected operating system version Red Hat Enterprise Linux Server release 6.1 (Santiago)
prereqSAM: rpm package and version installed 'ksh'
prereqSAM: Using default prerequisite checking on the following rpm package 'perl' prereqSAM: rpm package and version installed 'perl'
prereqSAM: Using default prerequisite checking on the following rpm package 'libstdc++' 'i686' prereqSAM: rpm package and version installed 'libstdc++' 'i686'
prereqSAM: Using default prerequisite checking on the following rpm package 'libstdc++' 'x86_64' prereqSAM: rpm package and version installed 'libstdc++' 'x86_64'
prereqSAM: Using default prerequisite checking on the following rpm package 'compat-libstdc++-33' 'x86_64' prereqSAM: Error: The following rpm package is not installed 'compat-libstdc++-33' 'x86_64'
prereqSAM: Using default prerequisite checking on the following rpm package 'pam' 'i686' prereqSAM: rpm package and version installed 'pam' 'i686'
1 missing package: compat-libstdc++-33 (x86_64)
prereqSAM: Error: Prerequisite checking for the ITSAMP installation failed Linux i386x RH Red Hat Enterprise Linux Server release 6.1 (Santiago) prereqSAM: Most severe error code returned 21 prereqSAM: One or more prerequisite packages were not found installed prereqSAM: <<< Prerequisite on wtydb21 check - log ended : Fri Apr 12 14:22:19 CST 2013
|
1.2.7 内核参数调整
db2 10.1 可以自动调整内核参数为最优值，但是个人设置一下还是没啥坏处吧
able 1. Enforced minimum settings for Linuxinterprocess communication kernel parametersIPC kernel parameter Enforced minimum setting
kernel.shmmni (SHMMNI) 256 * <size of RAM in GB>
kernel.shmmax (SHMMAX) <size of RAM in bytes>
kernel.shmall (SHMALL) 2 * <size of RAM in the defaultsystem page size>
共享内存段的设置建议
Beginning with the first section on SharedMemory Limits, the SHMMAX limit is the maximum size of a shared memory segmenton a Linux system. The SHMALL limit is the maximum allocation of shared memorypages on a system.
It is recommended to set the SHMMAX value to be equal to the amount ofphysical memory on your system. However, the minimum required on x86 systems is268435456 (256 MB) and for 64-bit systems, it is 1073741824 (1 GB).
kernel.sem (SEMMNI) 256 * <size of RAM in GB>
kernel.sem (SEMMSL) 250
kernel.sem (SEMMNS) 256 000
kernel.sem (SEMOPM) 32
信号量相关的内核设置
kernel.msgmni (MSGMNI) 1 024 * <size of RAM in GB>
kernel.msgmax (MSGMAX) 65 536
kernel.msgmnb (MSGMNB) 65 536
消息相关的内核设置
设置的小例子
#Example for a computer with 128GB of RAM，100GB分配给数据库使用
kernel.shmmni=4096
kernel.shmmax=107374182400
//单个共享内存段的最大大小，建议设置为最大物理内存大小。本例设置为100GB；大小为100*1024*1024*1024=107374182400
kernel.shmall=8388608
//系统中能够分配的共享内存页的大小，页的大小是4KB，总大小设置为100GB，应该是100*1024*1024*1024/4/1024=26214400
#kernel.sem=<SEMMSL> <SEMMNS><SEMOPM> <SEMMNI>
kernel.sem=250 1024000 32 4096
kernel.msgmni=16384
kernel.msgmax=65536
kernel.msgmnb=65536
参数设置说明
具体设计的例子
vi /etc/sysctl.conf
2，网络相关内核参数
注释掉rp_filter=1的行，
# net.ipv4.conf.all.rp_filter = 1
# changes for DB2 pureScale
然后添加
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.all.arp_ignore = 1
我的服务器设置内存128GB 100GB分配给数据库使用
1.2.8 blacklist.conf
在所有的机器(pcf、scf、mb1、mb2)上，修改文件：
/etc/modprobe.d/blacklist.conf
添加如下内容：
# RSCT hatsd, add for pureScale
blacklist iTCO_wdt
blacklist iTCO_vendor_support
1.2.9 共享存储准备
1.2.9.1 需求
一般至少需要四块物理盘，
tirebreker disk 25M
实例共享文件系统 10G
数据文件系统
日志文件系统
1.2.9.2 安装方法
1，安装device-mapper-multipath rpm。
当在 DM-多路径中添加新设备时，这些新设备会位于 /dev 目录的三个不同位置：dev/mapper/mpathn、/dev/mpath/mpathn 和 /dev/dm-n。 /dev/mapper 中的设备是在引导过程中生成的。可使用这些设备访问多路径设备，例如在生成逻辑卷时。提供 /dev/mpath 中的设备是为了方便，这样可在一个目录中看到所有多路径设备。这些设备是由 udev 设备管理器生成的，且在系统需要访问它们时不一定能启动。请不要使用这些设备生成逻辑卷或者文件系统。
所有 /dev/dm-n 格式的设备都只能是作为内部使用，且应该永远不要使用。
（备注 db2 purescale不能使用/dev/mapper/mpath设备，只能使用dm-设备名称）
yuminstall device-mapper-multipath*
2，打开multipathd
| [root@wtydb21 ~]# chkconfig --list multipathd multipathd 0:off 1:off 2:off 3:off 4:off 5:off 6:off [root@wtydb21 ~]# chkconfig multipathd on [root@wtydb21 ~]# chkconfig --list multipathd multipathd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
|
3，修改配置文件，保持多路径设备名称保持一致（安装db2 purescale 本步骤可省略）
一般可以修改/etc/multipath.conf文件，但为了保持集群（oraclerac），也可以修改/var/lib/multipath/bindings，并把文件复制到集群中的其它节点
4，查看配置情况
| modprobe dm-multipath service multipathd start multipath -v2 multipath -v2 命令输出多路径设备的路径来显示哪些设备是多路径设备。如果没有输出任何结果，请确定正确调试了所有 SAN 连接并使系统多路径。
执行以下命令确定在引导时启动了多路径守护进程
mpathconf --enable --with_multipathd y
chkconfig multipathd on
chkconfig --list multipathd
$ service multipathd stop $ service multipathd start $ multipathd –F $ multipathd –v2
|
参考文献
1.2.10 用户和组
可手工创建，也可在安装db purescale软件时配置。
两个用户分别属于不同的组，集群中不同的主机上需要相同的UID和GID，相同的home目录
You should use two different users with twodifferent groups. Each of the two users should have the same UID, GID, groupname, and the home directory on all the hosts.
Required user User name Groupname
Instance owner db2sdin1 db2iadm1
Fenced user db2sdfe1 db2fadm1
| groupadd -g 999 db2iadm1 groupadd -g 998 db2fadm1 useradd -g db2iadm1 -u 1004-m db2sdin1 useradd -g db2fadm1 -u 1003 -m db2sdfe1
passwd db2sdin1 passwd db2fenc1 |
1.2.11 NTP设置
| vi /etc/ntp.conf server 192.168.0.81 # local clock fudge 192.168.0.81 stratum 10 restrict 192.168.0.81 mask 255.255.255.255 nomodify notrap noquery
service ntp start chkconfig ntp on chkconfig --list ntpd |
1.2.12 WatchDog设置
DB2使用自己的watch dog，因此需要关闭机器带的intelwatch dog
In some installations, if Intel TCOWatchDog Timer Driver modules are loaded by default, they should beblacklisted, so that they do not start automatically or conflict with RSCT. Toblacklist the modules, edit the following files:
To verify if the modules are loaded
lsmod | grep -i iTCO_wdt; lsmod | grep -iiTCO_vendor_support
Edit the configuration files:
On RHEL 5.x and RHEL 6.1, edit file /etc/modprobe.d/blacklist.conf:
# RSCT hatsd
blacklist iTCO_wdt
blacklist iTCO_vendor_support
1.2.13 安装前检查
使用db2prereqcheck命令进行安装前的检查，一定确保检查完成通过。其实即使了在安装tivoli，gpfs等还需要一些依赖包，也可能出现安装报错
$ ./db2prereqcheck -i
检查是否满足具体安装版本需求，使用-v命令
$ ./db2prereqcheck -v 10.1.0.0
To generate a report file containingvalidation information and output from the db2prereqcheck command (including asummary section listing all failing tests), use the -o <filename>parameter. Specify the full report filename path to generate the report inanother directory. The prerequisites for installing the DB2 product will onlybe displayed on the command screen without the -o <filename> parameter.
$ ./db2prereqcheck -i -o report.rpt
检查IB网络是否满足要求
$ ./db2prereqcheck -i -p -t <networkconfiguration type>
db2prereqcheck –I –p –t MULT_IB_PORT_CLUSTER
1.2.14 接管GPFS（可选）
Purescale有个bug，不在认证中的阵列会出现不能使用/dev/dm-硬盘安装的情况，可以先手工安装gpfs，使用dm-盘，在使用db2cluster接管gpfs
解决不能直接使用/dev/dm- 进行安装的问题
[root@wtydb21 db2sdin1]#/opt/ibm/db2/V10.1/bin/db2cluster -cfs -verify -configuration
/opt/ibm/db2/V10.1/instance/db2cluster_prepare-cfs_takeover
| take over a user-managed GPFS cluster: 1. Log on as root on any machine in your cluster. 2. Run the db2cluster_prepare command with the following parameters: db2cluster_prepare -cfs_takeover 3. Check the error return code using the echo $? command. If there are errors, resolve the errors and rerun the command as specified in Step 2. 4. To verify that you've properly added the record, run the following command: [root@wtydb21 tmp]# db2greg -dump S,GPFS,3.5.0.4,/usr/lpp/mmfs,-,-,0,0,-,1365173996,0 S,DB2,10.1.0.2,/opt/ibm/db2/V10.1,,,2,0,,1365174117,0 V,DB2GPRF,DB2SYSTEM,wtydb21,/opt/ibm/db2/V10.1, V,GPFS_CLUSTER,NAME,gpfs_cluster.wtydb21,-,DB2_MANAGED
|
1.2.15 nable SCSI-3 PR
[root@wtydb21 src]#/usr/lpp/mmfs/bin/tsprinquiry dm-3
IBM :2810XIV :0000
vi /var/mmfs/etc/prcapdevices
IBM:2810XIV:0000
/usr/lpp/mmfs/bin/mmchconfigusePersistentReserve=yes
1.2.16 IB内核参数
echo "options mlx4_corelog_mtts_per_seg=7" >>/etc/modprobe.conf
1.3 安装purescale Feature
仅安装purescale Feature，不安装db2 实例
1.3.1 db2setup图形安装
本文通过db2_install手工实现安装
1.3.2 db2_install手工安装
1.3.2.1 安装软件
需要在两台主机分别进行安装
| [root@wtydb21 server]# ./db2_install -t /tmp/db2_install.trc -l db2_install.log DBI1324W Support of the db2_install command is deprecated. For more information, see the DB2 Information Center.
Default directory for installation of products - /opt/ibm/db2/V10.1
*********************************************************** Install into default directory (/opt/ibm/db2/V10.1) ? [yes/no] yes
Specify one of the following keywords to install DB2 products.
AESE ESE CONSV WSE EXP CLIENT RTCL
Enter "help" to redisplay product names.
Enter "quit" to exit.
*********************************************************** ESE *********************************************************** Do you want to install the DB2 pureScale Feature? [yes/no] yes DB2 installation is being initialized.
Total number of tasks to be performed: 50 Total estimated time for all tasks to be performed: 1910 second(s)
Task #1 start Description: Checking license agreement acceptance Estimated time 1 second(s) Task #1 end
Task #2 start Description: Base Client Support for installation with root privileges Estimated time 3 second(s) Task #2 end
Task #54 start Description: Updating global profile registry Estimated time 3 second(s) Task #54 end
The execution completed successfully.
For more information see the DB2 installation log at "/data/esefp2/server/db2_install.log" .……. |
1.3.2.2 新建实例
安装前检查，没有输出即正常
[root@wtydb21 ~]# preprpnode wtydb21 wtydb22wtydb23
开始安装
| /opt/ibm/db2/V10.1/instance/db2icrt -d -cf wtydb21 -cfnet wtydb21-ib0 -m wtydb22 -mnet wtydb22-ib0 -instance_shared_dev /dev/dm-15 -tbdev /dev/dm-2 -u db2sdfe1 db2sdin1
|
1.3.3 添加成员和CF
| --添加第二个CF /opt/ibm/db2/V10.1/instance/db2iupdt -d -add -cf wtydb22 -cfnet wtydb22-ib0 db2sdin1 --添加member /opt/ibm/db2/V10.1/instance/db2iupdt -d -add -m wtydb22 -mnet wtydb22-ib0 db2sdin1 /opt/ibm/db2/V10.1/instance/db2iupdt -d -add -m wtydb23 -mnet wtydb23-ib0 db2sdin1 |
1.3.4 安装补丁包
本次直接使用fp2最新补丁集安装。
1.3.5 添加IB网卡(待实施)
更新前检查db2nodes.cfg文件
| [root@wtydb23 msdata]# cat /home/db2sdin1/sqllib/db2nodes.cfg 0 wtydb22 0 wtydb22-ib0 - MEMBER 1 wtydb23 0 wtydb23-ib0 - MEMBER 128 wtydb23 0 wtydb23-ib0 - CF 129 wtydb22 0 wtydb22-ib0 - CF |
更新IB网络配置
| db2iupdt -update -cf wtydb23 -cfnet wtydb23-ib0,wtydb23-ib3 db2sdin1 db2iupdt -update -m wtydb23 -mnet wtydb23-ib0, wtydb23-ib3 db2sdin1 |
1.4 删除软件（可选）
安装过程中如果出现问题，可以使用如下方法删除相关软件
1, 备份数据库
2，使用db2stop 命令停止所有实例
3，使用db2idrop -g db2sdin1删除所有主机上的数据库实例.另外，db2idrop -g命令会保留最后主机上的gpfs数据库，需要手工清除gpfs。
4，使用db2_deinstall -a命令删除db2 purescale软件
2 安装中常见问题
1. /dev/mapper/mapth*设备不能使用
/dev/mapper/mapth*设备本质上是链接文件，GPFS不能使用，应该使用多路径软件映射后的/dev/mapth设备
2. 直接/dev/dm-*设备报错
该问题是软件的bug，主要是由于XIV存储不在软件支持列表导致，可以通过手工创建GPFS，然后db2cluster接管的方式或者改由
3 高可用测试
初始状态，系统正常
[db2sdin1@wtydb22 ~]$ db2instance -list
ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME
-- ---- ----- --------- ------------ ----- ---------------- ------------ -------
0 MEMBER STARTED wtydb22 wtydb22 NO 0 0 wtydb22-ib0
1 MEMBER STARTED wtydb21 wtydb21 NO 0 0 wtydb21-ib0
2 MEMBER STARTED wtydb23 wtydb23 NO 0 0 wtydb23-ib0
128 CF PRIMARY wtydb21 wtydb21 NO - 0 wtydb21-ib0
129 CF PEER wtydb22 wtydb22 NO - 0 wtydb22-ib0
HOSTNAME STATE INSTANCE_STOPPED ALERT
-------- ----- ---------------- -----
wtydb23 ACTIVE NO NO
wtydb21 ACTIVE NO NO
wtydb22 ACTIVE NO NO
数据库能正常连接
| [db2sdin1@wtydb22 ~]$ db2 connect to msdb
Database Connection Information
Database server = DB2/LINUXX8664 10.1.2 SQL authorization ID = DB2SDIN1 Local database alias = MSDB |
3.1 一台服务器问题
3.1.1 一台服务器公网断
1，断掉wtydb21主机的网卡
| [root@wtydb21 ~]# ifdown bond0 |
2，此时，从其它主机观察状态，正常
| [db2sdin1@wtydb22 ~]$ db2instance -list ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME -- ---- ----- --------- ------------ ----- ---------------- ------------ ------- 0 MEMBER STARTED wtydb22 wtydb22 YES 0 0 wtydb22-ib0 1 MEMBER WAITING_FOR_FAILBACK wtydb21 wtydb23 YES 0 1 wtydb23-ib0 2 MEMBER STARTED wtydb23 wtydb23 YES 0 0 wtydb23-ib0 128 CF ERROR wtydb21 wtydb21 YES - 0 wtydb21-ib0 129 CF PEER wtydb22 wtydb22 NO - 0 wtydb22-ib0
HOSTNAME STATE INSTANCE_STOPPED ALERT -------- ----- ---------------- ----- wtydb23 ACTIVE NO NO wtydb21 ACTIVE YES YES wtydb22 ACTIVE NO NO There is currently an alert for a member, CF, or host in the data-sharing instance. For more information on the alert, its impact, and how to clear it, run the following command: 'db2cluster -cm -list -alert'.
[db2sdin1@wtydb22 ~]$ db2 connect to msdb
Database Connection Information
Database server = DB2/LINUXX8664 10.1.2 SQL authorization ID = DB2SDIN1 Local database alias = MSDB |
3，网卡恢复后，系统能自动回复
一台服务器IB网络断
A,主机1 停止bond0网卡
| [root@wtydb21 ~]# ifdown bond0 |
B, 主机2 仍能成功连接数据库
| [root@wtydb22 ~]# db2 connect to msdb
Database Connection Information
Database server = DB2/LINUXX8664 10.1.2 SQL authorization ID = ROOT Local database alias = MSDB |
C, 此时集群状态，wtydb21处于waiting_for_fallback状态
| [db2sdin1@wtydb23 ~]$ db2instance -list ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME -- ---- ----- --------- ------------ ----- ---------------- ------------ ------- 0 MEMBER STARTED wtydb22 wtydb22 NO 0 0 wtydb22-ib0 1 MEMBER STARTED wtydb23 wtydb23 NO 0 0 wtydb23-ib0 2 MEMBER WAITING_FOR_FAILBACK wtydb21 wtydb22 NO 0 1 wtydb22-ib0 128 CF PRIMARY wtydb23 wtydb23 NO - 0 wtydb23-ib0 129 CF PEER wtydb22 wtydb22 NO - 0 wtydb22-ib0
HOSTNAME STATE INSTANCE_STOPPED ALERT -------- ----- ---------------- ----- wtydb21 ACTIVE NO YES wtydb23 ACTIVE NO NO wtydb22 ACTIVE NO NO There is currently an alert for a member, CF, or host in the data-sharing instance. For more information on the alert, its impact, and how to clear it, run the following command: 'db2cluster -cm -list -alert'. |
3.2 重启一台服务器（wtydb23）
wtydb22主机上显示，CF正常切换，数据库仍能提供服务，测试通过。重新起来后，恢复正常
| [db2sdin1@wtydb22 ~]$ db2instance -list ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME -- ---- ----- --------- ------------ ----- ---------------- ------------ ------- 0 MEMBER STARTED wtydb22 wtydb22 NO 0 0 wtydb22-ib0 1 MEMBER WAITING_FOR_FAILBACK wtydb23 wtydb21 NO 0 1 wtydb21-ib0 2 MEMBER STARTED wtydb21 wtydb21 NO 0 0 wtydb21-ib0 128 CF ERROR wtydb23 wtydb23 YES - 0 wtydb23-ib0 129 CF PRIMARY wtydb22 wtydb22 NO - 0 wtydb22-ib0
HOSTNAME STATE INSTANCE_STOPPED ALERT -------- ----- ---------------- ----- wtydb21 ACTIVE NO NO wtydb23 INACTIVE NO YES wtydb22 ACTIVE NO NO There is currently an alert for a member, CF, or host in the data-sharing instance. For more information on the alert, its impact, and how to clear it, run the following command: 'db2cluster -cm -list -alert' |
3.3 重启IB交换机switch-1
wtydb22成为提供服务的主机
| [db2sdin1@wtydb22 ~]$ db2instance -list
ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME -- ---- ----- --------- ------------ ----- ---------------- ------------ ------- 0 MEMBER STARTED wtydb22 wtydb22 NO 0 0 wtydb22-ib0 1 MEMBER WAITING_FOR_FAILBACK wtydb23 wtydb22 YES 0 2 wtydb22-ib0 2 MEMBER WAITING_FOR_FAILBACK wtydb21 wtydb22 YES 0 1 wtydb22-ib0 128 CF ERROR wtydb23 wtydb23 YES - 0 wtydb23-ib0 129 CF PRIMARY wtydb22 wtydb22 NO - 0 wtydb22-ib0
HOSTNAME STATE INSTANCE_STOPPED ALERT -------- ----- ---------------- ----- wtydb21 ACTIVE NO YES wtydb23 ACTIVE NO YES wtydb22 ACTIVE NO NO |
switch-1 起来以后，数据库自动恢复
| [db2sdin1@wtydb23 ~]$ db2instance -list ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME -- ---- ----- --------- ------------ ----- ---------------- ------------ ------- 0 MEMBER STARTED wtydb22 wtydb22 NO 0 0 wtydb22-ib0 1 MEMBER STARTED wtydb23 wtydb23 NO 0 0 wtydb23-ib0 128 CF PEER wtydb23 wtydb23 NO - 0 wtydb23-ib0 129 CF PRIMARY wtydb22 wtydb22 NO - 0 wtydb22-ib0
HOSTNAME STATE INSTANCE_STOPPED ALERT -------- ----- ---------------- ----- wtydb23 ACTIVE NO NO wtydb22 ACTIVE NO NO |
4 新增加一台服务器
前期环境准备，同前面
目标通过检验：/software/ese10.1.2/server/db2prereqcheck -p -tMULTI_IB_PORT_CLUSTER -v 10.1.0.2
| /opt/ibm/db2/V10.1.01/instance/db2iupdt -d -add -m wtydb23 -mnet wtydb23-ib0 db2sdin1 |
1552
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
