|
Technical Articles & Tips
Contents:
A1. README A2. cache_oraus.c A3. test.c A4. test.sh A5. test1.sh A6. test_v.c A7. test_v.sh A8. c.sh A9. c64.sh A10. Cache_open_calls.c 1. IntroductionIf you work with Oracle clients that make use of OCI (Oracle Call Interface), you may have noticed that the OCI driver in 8.1.7.x makes thousands of calls to translate messages from the The problem can be overcome by caching the The performance benefits can be seen in applications like sqlplus and Oracle clients (C and C++) making use of the OCI driver. Java technology-based applications using JDBC (native driver) should also see similar benefits. 2. Identifying the ProblemAn Oracle client application can be trussed on the Solaris Operating System (OS) to see the calls being made to this file -- "truss" is a system utility available on the Solaris platform, and it can be used to trace system calls made by an application. A running Oracle client application can be trussed as follows: truss -p [oracle client pid] The open("/oracle/app/oracle/product/8.1.7/rdbms/mesg/oraus.msb", O_RDONLY) = 9 fcntl(9, F_SETFD, 0x00000001) = 0 lseek(9, 0, SEEK_SET) = 0 read(9, "1513 "011303/t/t/0/0/0/0".., 256) = 256 lseek(9, 512, SEEK_SET) = 512 read(9, "1C88 Y r ~ M/0/0/0/0/0/0".., 512) = 512 lseek(9, 1024, SEEK_SET) = 1024 read(9, "/018/0 $/0 7/0 @/0 J/0 V".., 512) = 512 lseek(9, 39936, SEEK_SET) = 39936 read(9, "/0/t0519/0/0/0 >051A/0/0".., 512) = 512 close(9) = 0 These system calls can be expensive. The number of times these system calls are executed depends on the client application and the duration of the application run. 3. The Workaround: Cache |
|
Table 1: open()
Call
6.2 Interposition of the fcntl()
Function Call
A fcntl()
call is made to change the file control parameters. The first time fcntl()
is executed to change oraus.msb control parameters, the control is first transferred to the fcntl()
in libc.so. The return value is cached, as well as returned back to the Oracle client (STEP 2). The next time fcntl()
is executed, if the file descriptor matches the oraus.msb
file descriptor, the cached return value is returned (STEP 3). The control is not transferred to fcntl()
in libc.so.
int fcntl(int fildes, int cmd, int arg) { static int ret; static int(*fptr)() = 0; char* path; STEP 1 if (fptr == 0) { fptr = (int (*)())dlsym(RTLD_NEXT, "fcntl"); if (fptr == NULL) { fprintf(stderr, "dlopen: %s /n", dlerror()); return 0; } } STEP 2 if (k_fcntl_bm_fd == -1) { if (fildes == k_bm_fd) { ret = ((*fptr)(fildes, cmd, arg)); k_fcntl_bm_fd = k_bm_fd; return ret;; } STEP 3 } else if (k_fcntl_bm_fd == fildes) { return ret; } STEP 4 return ((*fptr)(fildes, cmd, arg)); }
|
Table 2: fcntl()
Call
6.3 Interposition of the lseek()
, read()
, and close()
Function Calls
For the lseek()
call, if the file descriptor matches the cached oraus.msb
file descriptor, the file offset is stored instead of calling the lseek()
in libc.so (STEP L2). On a read()
call, if the file descriptor matches the cached oraus.msb
file descriptor, the file offset stored from the lseek()
is used to index into the memory mapped oraus.msb
data. A memcpy()
is then executed (STEP R2). So an I/O call is now transformed to a simple memcpy()
call. A close()
on the cached file descriptor is ignored so that the cached file descriptor can be reused.
off_t lseek(int fildes, off_t offset, int whence) { static off_t (*fptr)() = 0;STEP L1
if (fptr == 0) { fptr = (int (*)())dlsym(RTLD_NEXT, "lseek"); if (fptr == NULL) { fprintf(stderr, "dlopen: %s /n", dlerror()); return 0; } }STEP L2
if (fildes == k_bm_fd) { k_bm_offset = offset; return offset; }STEP L3
return ((*fptr)(fildes, offset, whence)); }
|
Table 3: lseek()
Call
ssize_t read(int fildes, void *buf, size_t nbyte) { static ssize_t (*fptr)() = 0;STEP R1
if (fptr == 0) { fptr = (ssize_t (*)())dlsym(RTLD_NEXT, "read"); if (fptr == NULL) { fprintf(stderr, "dlopen: %s/n", dlerror()); return (0); } }STEP R2
if (fildes == k_bm_fd) { memcpy(buf, k_bm_buf+k_bm_offset, nbyte); return nbyte; }STEP R3
return ((*fptr)(fildes, buf, nbyte)); }
|
Table 4: read()
Call
int close(int fildes) { static int(*fptr)() = 0;STEP C1
if (fptr == 0) { fptr = (int (*)())dlsym(RTLD_NEXT, "close"); if (fptr == NULL) { fprintf(stderr, "dlopen: %s /n", dlerror()); return 0; } }STEP C2
if (fildes == k_bm_fd) { return 0; }STEP C3
return ((*fptr)(fildes)); }
|
Table 5: close()
Call
7. Performance Improvement
The performance improvement comes from not executing the system calls. The multiple instances of open()
,fcntl()
,lseek()
, read()
, and close()
to read the oraus.msb
messages are transformed to user-level calls. Also, the I/O operation becomes a simple memory-to-memory transfer.
To measure the performance gains, a test program mimicking an Oracle client's behavior was developed. The test program performs open()
,lseek()
, read()
, and close()
calls on the oraus.msb
file 50,000 times.
The test program showed a gain of about 1000%. However, in a real Oracle client application, this might translate to about a 2 percent-to-15 percent gain in performance.
7.1 Without LD_PRELOAD
and cache_oraus.so
|
Table 6: ptime test.sh
7.2 With LD_PRELOAD and cache_oraus.so
|
Table 7: ptime test.sh
8. Caveat
The preceding code is not multithreaded. For Oracle clients that are multithreaded, the write access to the cache variables needs to be mutex protected or synchronized.
9. Conclusion
Caching the file oraus.msb
in memory improves the performance of Oracle clients. Even though the performance gain observed in the test program is about 1000 percent, the gain when running a really big Oracle client application might not be more than 2 to 15 percent, due to other I/O contentions. This gain can be achieved without any code changes to the client application.
10. Acknowledgments
I would like to thank Teresa Riege (Sun) and Thomas Yen (Kenan/BP billing platform, CSG Systems) for their contributions and help in testing this utility during the 2002 Kenan/BP study (see related article listed in References). I would also like to thank my colleagues at MDE (IPE) for supporting me during this project.
I would like also to thank Ben Humphreys, technical support specialist, Sun, who made the 64-bit modifications to the code.
11. References
- Nagendra Nagarajayya, S.R. Venkatramanan, "Fast Sockets, An Interprocess Communication Library to Boost Application Performance"
- Sun Product Documentation site
- Oracle MetaLink (for news, problems, and bugs)
- CSG Kenan/BP Exceeds Billing Needs of Largest Telecom Service Providers in Scalability Test
Appendix: Test Programs and Wrappers
The Sample Code is being made available by Sun merely as a convenience to you. The Sample Code below is provided "AS IS." Sun makes no warranties of any kind whatsoever with respect to the Sample Code.
ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY WARRANTY OF NON-INFRINGEMENT OR IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, ARE HEREBY DISCLAIMED AND EXCLUDED TO THE EXTENT ALLOWED BY APPLICABLE LAW. IN NO EVENT WILL SUN BE LIABLE FOR ANY LOST REVENUE, PROFIT OR DATA, OR FOR DIRECT, SPECIAL, INDIRECT, CONSEQUENTIAL, INCIDENTAL, OR PUNITIVE DAMAGES HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY ARISING OUT OF THE USE OF OR INABILITY TO USE THE SAMPLE CODE, EVEN IF SUN HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
A1. README
* * Oct 02, 03 * Nagendra Nagarajayya * MDE/IPE(CSI) * Sun Microsystems, Inc * set LD_PRELOAD within a wrapper to cache_oraus.so, and specify the path to the oraus.msb file using the ENV variable, "oraus_msb_file". If the env is not specified, the following path is used, /oracle/app/oracle/product/8.1.7/rdbms/mesg/oraus.msb. WARNING: The solution presented does not support multi-threaded clients, but should be very easy to do so. BTW, it could work as it is with MT threaded clients, but has not been tested. Example: export LD_PRELOAD=./cache_oraus.so export oraus_msb_file=/ora/app/oracle/product/8.1.7/rdbms/mesg/oraus.msb Note: cache_open_calls.c is similar to cache_oraus.c but caches only the open calls. It does not memory map the oraus.msb file. Compilation instructions: ------------------------- Use the wrapper c.sh to compile cache_oraus.c. c.sh also compiles cache_open_calls.c, and the test programs. Use c64.sh to compile cache_oraus.c for 64 bit support. To test the library: -------------------- Use, ptime test.sh, and ptime test1.sh. To verify if the library is reading the contents properly, use test_v.sh, and test_v.c
A2. Cache_oraus.c
/* Date: April 18, 2002 Author: Nagendra Nagarajayya MDE/IPE/CSI Sun Microsystems, Inc. Date: October 02, 2003 Ben Humphreys Technical Support Specialist Sun Microsystems, Inc. Descr: made changes to source for 64 bit support Description: This caches oraus.msb file in memory and transforms I/O calls to the file to a read access. */ /* The Sample Code is being made available by Sun merely as a convenience to you. The Sample Code below is provided "AS IS." Sun makes no warranties of any kind whatsoever with respect to the Sample Code. ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY WARRANTY OF NON-INFRINGEMENT OR IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, ARE HEREBY DISCLAIMED AND EXCLUDED TO THE EXTENT ALLOWED BY APPLICABLE LAW. IN NO EVENT WILL SUN BE LIABLE FOR ANY LOST REVENUE, PROFIT OR DATA, OR FOR DIRECT, SPECIAL, INDIRECT, CONSEQUENTIAL, INCIDENTAL, OR PUNITIVE DAMAGES HOWEVER CAUSED AND REGARDLESS OF THE THEORY OF LIABILITY ARISING OUT OF THE USE OF OR INABILITY TO USE THE SAMPLE CODE, EVEN IF SUN HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. */ #include <stdio.h> #include <stdlib.h> #include <string.h> #include <unistd.h> #include <dlfcn.h> #include <sys/types.h> #include <sys/stat.h> #include <assert.h> #include <sys/mman.h> #include <sys/int_types.h> static int k_fcntl_bm_fd = -1; static int k_bm_fd = -1; static int k_bm_offset = 0; static char* k_bm_buf ; static struct stat k_bm_stat_buf ; int fcntl(int fildes, int cmd, intptr_t arg) { static int ret; static ssize_t(*fptr)() = 0; char* path; if (fptr == 0) { fptr = (ssize_t (*)())dlsym(RTLD_NXT, "fcntl"); if (fptr == NULL) { fprintf(stderr, "dlopen: %s /n", dlerror()); return 0; } } if (k_fcntl_bm_fd == -1) { if (fildes == k_bm_fd) { ret = ((*fptr)(fildes, cmd, arg)); k_fcntl_bm_fd = k_bm_fd; return ret;; } } else if (k_fcntl_bm_fd == fildes) { return ret; } return ((*fptr)(fildes, cmd, arg)); } off_t lseek(int fildes, off_t offset, int whence) { static off_t(*fptr)() = 0; if (fptr == 0) { fptr = (off_t (*)())dlsym(RTLD_NEXT, "lseek"); if (fptr == NULL) { fprintf(stderr, "dlopen: %s /n", dlerror()); return 0; } } if (fildes == k_bm_fd) { k_bm_offset = offset; return offset; } /* fprintf(stderr, "offset=%d k_bm_offset=%d/n", offset, k_bm_offset); */ return ((*fptr)(fildes, offset, whence)); } ssize_t read(int fildes, void *buf, size_t nbyte) { static ssize_t(*fptr)() = 0; if (fptr == 0) { fptr = (ssize_t (*)())dlsym(RTLD_NEXT, "read"); if (fptr == NULL) { fprintf(stderr, "dlopen: %s/n", dlerror()); return (0); } } /* fprintf(stderr, "fildes = %d k_bm_fd = %d /n", fildes, k_bm_fd); */ if (fildes == k_bm_fd) { memcpy(buf, k_bm_buf+k_bm_offset, nbyte); return nbyte; } return ((*fptr)(fildes, buf, nbyte)); } int open(const char *path, int oflag, mode_t mode) { static char* msb_path; static int(*fptr)() = 0; if (fptr == 0) { fptr = (int (*)())dlsym(RTLD_NEXT, "open"); if (fptr == NULL) { fprintf(stderr, "dlopen: %s /n", dlerror()); return 0; } msb_path = (char*)getenv("oraus_msb_file"); } if (!msb_path) { msb_path = "/oracle/app/oracle/product/8.1.7/rdbms/mesg/oraus.msb"; } if (strcmp(path, msb_path) == 0) { if (k_bm_fd == -1) { k_bm_fd = ((*fptr)(path, oflag, mode)); if (k_bm_fd <= 0) { perror(path); exit(1); } fstat(k_bm_fd, &k_bm_stat_buf); /* fprintf(stderr, "open the file %d/n",k_bm_fd); */ k_bm_buf = (char*)mmap((caddr_t) 0, k_bm_stat_buf.st_size, (PROT_READ), MAP_SHARED, k_bm_fd, 0); assert(k_bm_buf != MAP_FAILED); return k_bm_fd; } else { /* fprintf(stderr, "re-open the file %d/n",k_bm_fd); */ return k_bm_fd; } } return ((*fptr)(path, oflag, mode)); } int close(int fildes) { static int(*fptr)() = 0; if (fptr == 0) { fptr = (int (*)())dlsym(RTLD_NEXT, "close"); if (fptr == NULL) { fprintf(stderr, "dlopen: %s /n", dlerror()); return 0; } } if (fildes == k_bm_fd) { /* fprintf(stderr, "close the file %d/n",k_bm_fd); */ return 0; } return ((*fptr)(fildes)); } See README file for details on how to compile this program, and the environment needed to execute the Oracle client program. Appendixes A3-A6 explain a little more about the test programs.
A3. test.c
This test program reads the oraus.msb
file 50,000 times to mimic Oracle client application behavior.
#include <stdio.h> #include <unistd.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #include <string.h> int fd = 0; int fd1 = 0; char buf[1024]; char buf1[1024]; int x1 = 0; int x2 = 0; int x3 = 0; int x4 = 0; int x5 = 0; int main() { int i; int r=-1; for (i=0; i<50000; i++) { fd = open("/ora/app/oracle/product/8.1.7/rdbms/mesg/oraus.msb", O_RDONLY); /* fd1 = open("/oracle/app/oracle/product/8.1.7/rdbms/mesg/oraus.msb_1", O_RDONLY); */ /* fprintf(stderr, "fd = %d fd1 = %d /n", fd, fd1); */ r = fcntl(fd, F_SETFD); lseek(fd, 0, SEEK_SET); read(fd, &buf, 256); /* r = fcntl(fd1, F_SETFD); lseek(fd1, 0, SEEK_SET); read(fd1, &buf1, 256); x1 = memcmp(&buf1, &buf, 256); if (x1 != 0 ) { fprintf(stderr, "x1 did not mach %d/n", x1); } */ lseek(fd, 512, SEEK_SET); read(fd, &buf, 512); /* lseek(fd1, 512, SEEK_SET); read(fd1, &buf1, 512); x2 = memcmp(&buf1, &buf, 512); if (x2 != 0 ) { fprintf(stderr, "x2 did not mach %d /n", x2); } */ lseek(fd, 1024, SEEK_SET); read(fd, &buf, 512); /* lseek(fd1, 1024, SEEK_SET); read(fd1, &buf1, 512); x3 = memcmp(&buf1, &buf, 512); if (x3 != 0 ) { fprintf(stderr, "x3 did not mach /n"); } */ lseek(fd, 39936, SEEK_SET); read(fd, &buf, 512); /* lseek(fd1, 39936, SEEK_SET); read(fd1, &buf1, 512); x4 = memcmp(&buf1, &buf, 512); if (x4 != 0 ) { fprintf(stderr, "x4 did not mach /n"); } */ close(fd); /* close(fd1); */ } }
A4. test.sh
This is a wrapper to execute the test program to show the performance gains of caching the contents of oraus.msb
in memory. The LD_PRELOAD
and oraus_msb_file
environment variables point to cache_oraus.so
and the oraus.msb
file.
#!/bin/bash export oraus_msb_file=/ora/app/oracle/product/8.1.7/rdbms/mesg/oraus.msb export LD_PRELOAD=/usr/lib/cache_oraus.so export LD_PRELOAD=./cache_oraus.so time ./test
A5. test1.sh
This is a wrapper to execute the test program without the caching mechanism.
#!/bin/bash time ./test
A6. test_v.c
This program was used to verify that the interposed calls work properly.
#include <stdio.h> #include <unistd.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #include <string.h> int fd = 0; int fd1 = 0; char buf[1024]; char buf1[1024]; int x1 = 0; int x2 = 0; int x3 = 0; int x4 = 0; int x5 = 0; int main() { int i; int r=-1; for (i=0; i<50000; i++) { fd = open("/ora/app/oracle/product/8.1.7/rdbms/mesg/oraus.msb", O_RDONLY); fd1 = open("/ora/app/oracle/product/8.1.7/rdbms/mesg/oraus.msb_1", O_RDONLY); /* fprintf(stderr, "fd = %d fd1 = %d /n", fd, fd1); */ r = fcntl(fd, F_SETFD); lseek(fd, 0, SEEK_SET); read(fd, &buf, 256); r = fcntl(fd1, F_SETFD); lseek(fd1, 0, SEEK_SET); read(fd1, &buf1, 256); x1 = memcmp(&buf1, &buf, 256); if (x1 != 0 ) { fprintf(stderr, "x1 did not mach %d/n", x1); } lseek(fd, 512, SEEK_SET); read(fd, &buf, 512); lseek(fd1, 512, SEEK_SET); read(fd1, &buf1, 512); x2 = memcmp(&buf1, &buf, 512); if (x2 != 0 ) { fprintf(stderr, "x2 did not mach %d /n", x2); } lseek(fd, 1024, SEEK_SET); read(fd, &buf, 512); lseek(fd1, 1024, SEEK_SET); read(fd1, &buf1, 512); x3 = memcmp(&buf1, &buf, 512); if (x3 != 0 ) { fprintf(stderr, "x3 did not mach /n"); } lseek(fd, 39936, SEEK_SET); read(fd, &buf, 512); lseek(fd1, 39936, SEEK_SET); read(fd1, &buf1, 512); x4 = memcmp(&buf1, &buf, 512); if (x4 != 0 ) { fprintf(stderr, "x4 did not mach /n"); } close(fd); close(fd1); } }
A7. test_v.sh
This is a wrapper for testing test_v.c
.
#!/bin/bash export LD_PRELOAD=/usr/lib/cache_oraus.so export LD_PRELOAD=./cache_oraus.so time ./test_v
A8. c.sh
#!/bin/bash PATH=/opt/SUNWspro/bin/:$PATH export PATH; cc -fast -xarch=v8plusa -xdepend -xvector -xstrconst -xprefetch -G -o cache_open_calls.so cache_open_calls.c -ldl cc -fast -xdepend -xvector -xstrconst -xarch=v8plusa -xprefetch -G -o cache_oraus.so cache_oraus.c -ldl cc -fast -xstrconst -xvector -xdepend -xarch=v8plusa -xprefetch -o test test.c -ldl cc -fast -xstrconst -xvector -xdepend -xarch=v8plusa -xprefetch -o test_v test_v.c -ldl
A9. c64.sh
#!/bin/bash # Changes to source cache_oraus.c for 64 bit support was made # by Ben Humphreys PATH=/opt/SUNWspro/bin/:$PATH export PATH; cc -fast -xarch=v9a -xcode=abs64 -xdepend -xvector -xstrconst -xprefetch -G -o cache_open_calls.so cache_open_calls.c -ldl cc -fast -xarch=v9a -xcode=abs64 -xdepend -xvector -xstrconst -xprefetch -G -o cache_oraus.so cache_oraus.c -ldl cc -fast -xarch=v9a -xcode=abs64 -xstrconst -xvector -xdepend -xprefetch -o test test.c -ldl cc -fast -xarch=v9a -xcode=abs64 -xstrconst -xvector -xdepend -xprefetch -o test_v test_v.c -ldl< name=A10>
A10. Cache_open_calls.c
/* Date: April 18, 02 Author: Nagendra Nagarajayya Description: Caches open, and close calls to oraus.msb file */ #include <stdio.h> #include <unistd.h> #include <dlfcn.h> #include <string.h> #include <stdlib.h> static int k_bm_fd = -1; int open(const char *path, int oflag, mode_t mode) { static int (*fptr)() = 0; if (fptr == 0) { fptr = (int (*)())dlsym(RTLD_NEXT, "open"); if (fptr == NULL) { fprintf(stderr, "dlopen: %s /n", dlerror()); return 0; } } if (strcmp(path, "/oracle/app/oracle/product/8.1.7/rdbms/mesg/oraus.msb") == 0) { if (k_bm_fd == -1) { k_bm_fd = ((*fptr)(path, oflag, mode)); if (k_bm_fd <= 0) { perror(path); exit(1); } /* fstat(k_bm_fd, &k_bm_stat_buf); fprintf(stderr, "open the file %d/n",k_bm_fd); */ return k_bm_fd; } else { /* fprintf(stderr, "re-open the file %d/n",k_bm_fd); */ return k_bm_fd; } } return ((*fptr)(path, oflag, mode)); } int close(int fildes) { static int (*fptr)() = 0; if (fptr == 0) { fptr = (int (*)())dlsym(RTLD_NEXT, "close"); if (fptr == NULL) { fprintf(stderr, "dlopen: %s /n", dlerror()); return 0; } } if (fildes == k_bm_fd) { /* fprintf(stderr, "close the file %d/n",k_bm_fd); */ return 0; } return ((*fptr)(fildes)); }
About the Author
Nagendra Nagarajayya has been at Sun for more than 9 years. A Staff Engineer in Market Development Engineering (MDE/IPE), he works with telco ISVs on architecture, performance tuning, sizing and scaling, benchmarking, porting, and so on. His interests include multithreading, concurrency and parallelism, HA, distributed computing, networking, and the Java and Solaris platforms.