When setting up your LUNs, it is important to create the NSD such
that they map one to one with a LUN that is a single RAID device.
For file systems holding large Oracle databases, set the GPFS file
system block size through the mmcrfs command using
the -B option, to a large value:
512 KB is generally suggested.
256 KB is suggested if there is activity other than Oracle using
the file system and many small files exist which are not in the database.
1 MB is suggested for file systems 100 TB or larger.The
large block size makes the allocation of space for the databases manageable and
has no affect on performance when Oracle is using the Asynchronous I/O (AIO) and
Direct I/O (DIO) features of AIX®.
Set the GPFS worker threads through the mmchconfig prefetchThreads command to allow the maximum
parallelism of the Oracle AIO threads:
This setting can be as large as 548.
The GPFS prefetch threads must be adjusted accordingly through the
mmchconfig
prefetchThreads command as the sum of those two classes of threads
must be 550 or less.
When requiring GPFS sequential I/O, set the prefetch threads
between 50 and 100 (the default is 64), and set the worker threads to have the
remainder.
Note: These changes through
the mmchconfig command take effect upon restart of
the GPFS daemon.
The number of AIX AIO kprocs to create should
be approximately the same as the GPFS worker1Threads
setting.
The AIX AIO maxservers setting is the number
of kprocs PER CPU. It is suggested to set is slightly
larger than worker1Threads divided by the number of CPUs.
For example if worker1Threads is set to 500 on a 32-way
SMP, set maxservers to 20.
Set the Oracle database block size equal to the LUN segment size or
a multiple of the LUN pdisk segment size.
Set the Oracle read-ahead value to prefetch one or two full GPFS
blocks. For example, if your GPFS block size is 512 KB, set the Oracle blocks to
either 32 or 64 16 KB blocks.
Do not use the dio option on the mount command as this forces DIO when accessing allfiles. Oracle automatically uses DIO to open database
files on GPFS.
When running Oracle RAC 10g, it is suggested you increase the value
for OPROCD_DEFAULT_MARGIN to at least 500 to avoid
possible random reboots of nodes.
In the control script for the Oracle CSS daemon, located in /etc/init.cssd the
value for OPROCD_DEFAULT_MARGIN is set to 500
(milliseconds) on all UNIX® derivatives except for AIX. For AIX this value is
set to 100. From a GPFS perspective, even 500 milliseconds maybe too low in
situations where node failover may take up to a minute or two to resolve.
However, if during node failure the surviving node is already doing direct IO to
the oprocd
control file, it should have the necessary tokens and indirect block cached and
should therefore not have to wait during failover.