Exadata虚拟环境搭建

最新推荐文章于 2024-05-19 17:03:37 发布

cileimian3035

最新推荐文章于 2024-05-19 17:03:37 发布

阅读量686

点赞数

文章标签：数据库运维 shell

(一) 配置要求

1. CPU Intel Core i3以上(或者AMD AthlonⅡ X4以上)，推荐Core i5以上(AMD PhenomⅡ X4以上)；

2. 内存至少4GB以上，推荐配置8GB；

3. 磁盘空余至少在40GB以上；

4. 安装好虚拟机，推荐使用Oracle Virtualbox；

5. 有Oracle Linux 5安装介质。可前往https://edelivery.oracle.com/下载，介质名为V27570-01.ZIP。也可以到http://mirrors.dotsrc.org/oracle-linux/ 上下载。这里用的是OEL5.7；

6. 有Exadata 11.2.3.2 Cell的安装介质。下载地址同上，Cell介质名为V33693-01.ZIP；

7. 准备Oracle Clusterware以及Oracle database的安装介质，这里用的11.2.0.4版本；

8. 准备补丁工具Opatch；

9. 准备最新的Exadata RDBMS Bundle Patch。

(二) 安装步骤

1. 搭建Exadata存储服务器虚拟机

1) 安装OEL 5.7

安装过程省略，主要是内存和磁盘分配，分别给1GB内存和100GB的虚拟硬盘空间。注意的是需要选上软件开发包，例如gcc/aio之类的。这里安装完成之后可以先拷贝一份虚拟机镜像，后续用作数据库服务器。

2) 修改静态IP地址及hosts文件

点击(此处)折叠或打开

[root@cell ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
# Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
DEVICE=eth0
BOOTPROTO=static
IPADDR=192.168.56.101
NETMASK=255.255.255.0
NETWORK=192.168.56.0
BROADCAST=192.168.56.255
ONBOOT=yes
HWADDR=00:0c:29:69:08:4a
TYPE=Ethernet
[root@cell ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
192.168.56.101 cell.com

3) 修改内核

Oracle Linux默认使用UEK，如果使用UEK，则在后面的步骤中无法正常启动cellsrv服务。使用grub修改配置，将默认启动内核修改为redhat兼容内核：

[root@cell ~]# vi /etc/grub.conf

将default=0修改为default=1，然后重启。

4) 上传解压

解压V33693-01.ZIP得到cellImageMaker_11.2.3.2.0_LINUX.X64_120713-1.x86_64.tar，继续解压得到一个名为dl180的文件夹。

在dl180/boot/cellbits下找到cell.bin文件。这个bin文件实际上是一个zip压缩包。可以使用unzip对它进行解压。

点击(此处)折叠或打开

[root@cell ~]# unzip V33693-01.zip
[root@cell ~]# tar –pxvf cellImageMaker_11.2.3.2.0_LINUX.X64_120713-1.x86_64.tar
[root@cell cellbits]# unzip cell.bin
Archive: cell.bin
warning [cell.bin]: 6408 extra bytes at beginning or within zipfile
(attempting to process anyway)
inflating: cell-11.2.3.2.0_LINUX.X64_120713-1.x86_64.rpm
inflating: jdk-1_5_0_15-linux-amd64.rpm

解压后得到cell-11.2.3.2.0_LINUX.X64_120713-1.x86_64.rpm和jdk-1_5_0_15-linux-amd64.rpm两个rpm包。

5) 安装jdk和cell

安装jdk：

点击(此处)折叠或打开

[root@cell cellbits]# rpm -ivh jdk-1_5_0_15-linux-amd64.rpm
Preparing... ########################################### [100%]
1:jdk ########################################### [100%]

安装cell可能会报错，提示有LWP依赖包，这是因为默认没有安装perl-libwww-perl，安装这个包建议使用yum配置。

点击(此处)折叠或打开

[root@cell cellbits]# yum install perl-libwww-perl

再次安装cell，仍然出现报错：

点击(此处)折叠或打开

[root@cell cellbits]# rpm -ivh cell-11.2.3.2.0_LINUX.X64_120713-1.x86_64.rpm
Preparing... ########################################### [100%]
Pre Installation steps in progress ...
error: %pre(cell-11.2.3.2.0_LINUX.X64_120713-1.x86_64) scriptlet failed, exit status 1
error: install: %pre scriptlet failed (2), skipping cell-11.2.3.2.0_LINUX.X64_120713-1

通过以下命令生成具体的检查条件的脚本，在进行分析:

点击(此处)折叠或打开

[root@cell cellbits]# rpm --scripts -qp cell-11.2.3.2.0_LINUX.X64_120713-1.x86_64.rpm >>diag.log

查看diag.log文件，发现所有的的判断信息都会记录到/opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/.install_log.txt文件里，查看这个文件，发现里面只有一条记录，/var/log/oracle这个目录不存在。

点击(此处)折叠或打开

[root@cell cellbits]# cat diag.log
if [ $? == 0 ]; then
echo 'sage rpm is installed. Please remove it first using: "rpm -e sage". Then try installing cell rpm again.' >> /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/.install_log.txt 2>&1
exit 1
fi
if [ ! -d /ar/log/oracle ]; then
echo '/var/log/oracle does not exist or not found. Check it exists and is writable and retry.' >> /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/.install_log.txt 2>&1
exit 1
fi………….
[root@cell cellbits]# cat /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/.install_log.txt
/var/log/oracle does not exist or not found. Check it exists and is writable and retry.
手工建立/var/log/oracle目录，并修改权限后再次安装cell：
[root@cell cellbits]# rpm -ivh cell-11.2.3.2.0_LINUX.X64_120713-1.x86_64.rpm
Preparing... ########################################### [100%]
Pre Installation steps in progress ...
1:cell ########################################### [100%]
Post Installation steps in progress ...
Set cellusers group for /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/deploy/log directory
Set 775 permissions for /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/deploy/log directory
/
Installation SUCCESSFUL.
Starting RS and MS... as user celladmin
Done. Please Login as user celladmin and create cell to startup CELLSRV to complete cell configuration.
WARNING: Using the current shell as root to restart cell services.
Restart the cell services using a new shell.

6) 建立虚拟磁盘和闪盘

执行dd.sh创建对应的磁盘和闪盘：其中磁盘12块，每块大小为1GB，闪盘4块，每块大小1GB，并修改权限为660：

点击(此处)折叠或打开

[root@cell cell]# mkdir -p /opt/oracle/cell/disks/raw
[root@cell cell]# cd /opt/oracle/cell/disks/raw
[root@cell raw]# vi dd.sh
[root@cell raw]# cat dd.sh
dd if=/dev/zero of=disk01 bs=1M count=1000
dd if=/dev/zero of=disk02 bs=1M count=1000
dd if=/dev/zero of=disk03 bs=1M count=1000
dd if=/dev/zero of=disk04 bs=1M count=1000
dd if=/dev/zero of=disk05 bs=1M count=1000
dd if=/dev/zero of=disk06 bs=1M count=1000
dd if=/dev/zero of=disk07 bs=1M count=1000
dd if=/dev/zero of=disk08 bs=1M count=1000
dd if=/dev/zero of=disk09 bs=1M count=1000
dd if=/dev/zero of=disk10 bs=1M count=1000
dd if=/dev/zero of=disk11 bs=1M count=1000
dd if=/dev/zero of=disk12 bs=1M count=1000
dd if=/dev/zero of=FLASH01 bs=1M count=1000
dd if=/dev/zero of=FLASH02 bs=1M count=1000
dd if=/dev/zero of=FLASH03 bs=1M count=1000
dd if=/dev/zero of=FLASH04 bs=1M count=1000
[root@cell raw]# sh dd.sh
[root@cell raw]# chmod 660 *

7) 启动cell服务

切换到celladmin用户（该用户在安装完cell之后会建立），然后重启celld服务：

点击(此处)折叠或打开

[root@cell raw]# su - celladmin
[celladmin@cell ~]$ cellcli -e alter cell restart services all
Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services... running
Starting CELLSRV services...
The STARTUP of CELLSRV services was not successful.
CELL-01547: CELLSRV startup failed due to unknown reasons.
Starting MS services...
The STARTUP of MS services was successful.

启动cellrv失败，可以去查看/opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/log/diag/asm/cell/cell/trace/alert.log，该日志是cellcli工具相关日志，可以看到报错说明cellinit.ora文件里缺少ip的参数。

点击(此处)折叠或打开

Tue Jul 19 15:10:55 2016
[RS] Started monitoring process /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/cellsrv/bin/cellrssmt with pid 6503
Errors in file /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/log/diag/asm/cell/cell/trace/rstrc_6485_4.trc (incident=25):
RS-7445 [Required IP parameters missing] [Check cellinit.ora] [] [] [] [] [] [] [] [] [] []
Incident details in: /opt/oracle/cell11.2.3.2.0_LINUX.X64_120713/log/diag/asm/cell/cell/incident/incdir_25/rstrc_6485_4_i25.trc
Sweep [inc][25]: completed
[RS] Required IP parameters not configured in cellinit.ora. Err: 36
OS Hugepage status:
Total/free hugepages available=12/12; hugepage size=2048KB
[RS] Start service CELLSRV failed with error: -74.

在cellinit.ora中添加网卡的信息:

点击(此处)折叠或打开

[celladmin@cell ~]$ cellcli -e create cell cell1 interconnect1=eth0

执行成功可以看到文件里添加了一行ipaddress1等的信息:

点击(此处)折叠或打开

[celladmin@cell ~]$ cat /opt/oracle/cell/cellsrv/deploy/config/cellinit.ora
#CELL Initialization Parameters
version=0.0
HTTP_PORT=8888
bbuChargeThreshold=800
SSL_PORT=23943
RMI_PORT=23791
ipaddress1=192.168.56.101/24
bbuTempThreshold=60
DEPLOYED=TRUE
JMS_PORT=9127
BMC_SNMP_PORT=162

然后再次重启celld服务，可以看到cellsrv、ms、rs都正常启动了：

点击(此处)折叠或打开

[celladmin@cell ~]$ cellcli -e alter cell restart services all
Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services... running
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Starting MS services...
The STARTUP of MS services was successful.

8) 创建celldisk、griddisk、flashcache和flashlog

登陆cellcli工具界面，创建celldisk，flashcache和flashlog也会自动创建，先创建完celldisk后再建立griddisk。

点击(此处)折叠或打开

[celladmin@cell ~]$ cellcli
CellCLI> create celldisk all
CellDisk CD_disk01_cell1 successfully created
CellDisk CD_disk02_cell1 successfully created
CellDisk CD_disk03_cell1 successfully created
CellDisk CD_disk04_cell1 successfully created
CellDisk CD_disk05_cell1 successfully created
CellDisk CD_disk06_cell1 successfully created
CellDisk CD_disk07_cell1 successfully created
CellDisk CD_disk08_cell1 successfully created
CellDisk CD_disk09_cell1 successfully created
CellDisk CD_disk10_cell1 successfully created
CellDisk CD_disk11_cell1 successfully created
CellDisk CD_disk12_cell1 successfully created
CellCLI> create griddisk data_CD_disk01_cell1 celldisk=CD_disk01_cell1
GridDisk data_CD_disk01_cell1 successfully created
CellCLI> create griddisk data_CD_disk02_cell1 celldisk=CD_disk02_cell1
GridDisk data_CD_disk02_cell1 successfully created
CellCLI> create griddisk data_CD_disk03_cell1 celldisk=CD_disk03_cell1
GridDisk data_CD_disk03_cell1 successfully created
CellCLI> create griddisk data_CD_disk04_cell1 celldisk=CD_disk04_cell1
GridDisk data_CD_disk04_cell1 successfully created
CellCLI> create griddisk data_CD_disk05_cell1 celldisk=CD_disk05_cell1
GridDisk data_CD_disk05_cell1 successfully created
CellCLI> create griddisk data_CD_disk06_cell1 celldisk=CD_disk06_cell1
GridDisk data_CD_disk06_cell1 successfully created
CellCLI> create griddisk data_CD_disk07_cell1 celldisk=CD_disk07_cell1
GridDisk data_CD_disk07_cell1 successfully created
CellCLI> create griddisk data_CD_disk08_cell1 celldisk=CD_disk08_cell1
GridDisk data_CD_disk08_cell1 successfully created
CellCLI> create griddisk data_CD_disk09_cell1 celldisk=CD_disk09_cell1
GridDisk data_CD_disk09_cell1 successfully created
CellCLI> create griddisk data_CD_disk10_cell1 celldisk=CD_disk10_cell1
GridDisk data_CD_disk10_cell1 successfully created
CellCLI> create griddisk data_CD_disk11_cell1 celldisk=CD_disk11_cell1
GridDisk data_CD_disk11_cell1 successfully created
CellCLI> create griddisk data_CD_disk12_cell1 celldisk=CD_disk12_cell1
GridDisk data_CD_disk12_cell1 successfully created

可以使用下面的命令查看信息，加detail可以查看详细信息：

点击(此处)折叠或打开

CellCLI> list flashcache detail
name: cell1_FLASHCACHE
cellDisk: FD_02_cell1,FD_01_cell1,FD_00_cell1,FD_03_cell1
creationTime: 2016-07-19T15:13:20+08:00
degradedCelldisks:
effectiveCacheSize: 3.1875G
id: 895f521d-d20e-4c7c-b73c-45162654e498
size: 3.1875G
status: normal
CellCLI> list flashlog detail
name: cell1_FLASHLOG
cellDisk: FD_02_cell1,FD_03_cell1,FD_01_cell1,FD_00_cell1
creationTime: 2016-07-19T15:13:09+08:00
degradedCelldisks:
effectiveSize: 512M
efficiency: 100.0
id: dec427f6-78a2-421e-8f2a-8d4a82e653e3
size: 512M
status: normal
CellCLI> list cell
cell1 online
CellCLI> list griddisk
data_CD_disk01_cell1 active
data_CD_disk02_cell1 active
data_CD_disk03_cell1 active
data_CD_disk04_cell1 active
data_CD_disk05_cell1 active
data_CD_disk06_cell1 active
data_CD_disk07_cell1 active
data_CD_disk08_cell1 active
data_CD_disk09_cell1 active
data_CD_disk10_cell1 active
data_CD_disk11_cell1 active
data_CD_disk12_cell1 active
CellCLI> list griddisk data_CD_disk05_cell1 detail
name: data_CD_disk05_cell1
availableTo:
cachingPolicy: default
cellDisk: CD_disk05_cell1
comment:
creationTime: 2016-07-19T15:57:22+08:00
diskType: HardDisk
errorCount: 0
id: aab593f4-a694-48d2-8d73-84496697571c
offset: 48M
size: 944M
status: active

到这一步，我们的存储节点虚拟机就基本创建完成了。

1. 搭建Exadata数据库服务器虚拟机

1) 安装OEL 5.7

这里我们直接使用之前克隆的一台虚拟机做为数据库服务器的虚拟机，修改相应配置，这里也是需要选择开发的安装包，如果没有也可以直接通过oracle-validated来配置，虚拟机的内存最好也相应调大一点。

2) 修改静态IP地址及安装oracle-validated

点击(此处)折叠或打开

[root@db ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
# Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
DEVICE=eth0
BOOTPROTO=none
HWADDR=00:0C:29:2B:82:02
ONBOOT=yes
NETMASK=255.255.255.0
IPADDR=192.168.56.102
GATEWAY=192.168.56.1
TYPE=Ethernet
USERCTL=no
IPV6INIT=no
PEERDNS=yes
[root@db yum.repos.d]# yum -y install oracle-validated

安装完成后就会把所有Oracle需要的rpm包并且默认创建好所有用户组和用户，但无GRID用户，需要另外创建。

同时建议关闭SELinux，在/etc/selinux/config中将SELINUX=disabled，重启操作系统就能关闭SELinux，或者使用命令echo 0 >/selinux/enforce关闭SELinux。

点击(此处)折叠或打开

[root@db yum.repos.d]# echo 0 >/selinux/enforce
[root@db yum.repos.d]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - SELinux is fully disabled.
SELINUX=disabled
# SELINUXTYPE= type of policy in use. Possible values are:
# targeted - Only targeted network daemons are protected.
# strict - Full SELinux protection.
SELINUXTYPE=targeted

1) 创建Exadata所需的配置文件

Cellinit.ora，此文件中填入数据库节点的IP地址和子网掩码：

Cellip.ora，此文件填入存储节点的IP地址:

点击(此处)折叠或打开

[root@db u01]# mkdir -p /etc/oracle/cell/network-config
[root@db u01]# chown -R grid:oinstall /etc/oracle/cell/network-config/
[root@db u01]# vi /etc/oracle/cell/network-config/cellinit.ora
[root@db u01]# cat /etc/oracle/cell/network-config/cellinit.ora
ipaddress1=192.168.56.102/24
[root@db u01]# vi /etc/oracle/cell/network-config/cellip.ora
[root@db u01]# cat /etc/oracle/cell/network-config/cellip.ora
cell="192.168.56.101"

2) 开始安装

这里由于内存不充足，所以只是搭建了Standalone Server的服务器，Exadata并不是强制要求安装RAC，如果内存充足可以搭建RAC，步骤都是和非Exadata的差不多，下面我只把有区别的部分截出来。

到ASM这一步可以看到Cell的Griddisk了，这些以o/开头的ASM磁盘就是Exadata

Cell端创建的griddisk。

如果没有找到griddisk，也可以手工修改ASM_DISKSTRING到“o/cell的ip地址/*”。如：“o/192.168.56.101/data*”。

最后就按照标准的流程来安装Oracle Database，这样模拟的Exadata虚拟环境就搭建完成了。

(三) 报错问题

1. ORA-56865

l 问题：

安装GI到最后一步ASMCA启动asm资源时失败，报错ORA-56865错误地址的CLELLINIT.ORA文件

l 日志：

点击(此处)折叠或打开

INFO: Read: Configuring ASM failed with the following message:
INFO: Read: Configuring HA resource failed. The following error occured:
INFO: Read: PRCR-1079 : Failed to start resource ora.asm
INFO: Read: CRS-5017: The resource action "ora.asm start" encountered the following error:
INFO: Read: ORA-56865: Invalid IP address in CELLINIT.ORA
INFO: Read: . For details refer to "(:CLSN00107:)" in "/u01/app/grid/product/11.2.0/grid/log/db/agent/ohasd/oraagent_grid/oraagent_grid.log".
INFO: Read:
INFO: Read: CRS-2674: Start of 'ora.asm' on 'db' failed
ERROR: diskgroup DATA1 was not created
ORA-15018: diskgroup cannot be created
ORA-15072: command requires at least 2 regular failure groups, discovered only 1