wps1:/oracle/app/oracle/admin/mss/bdump$sqlplus '/as sysdba'
SQL*Plus: Release 10.2.0.4.0 - Production on Sat Nov 19 10:03:49 2011
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
NFS server 130.34.3.102 not responding still trying
为了排查问题,使用truss跟踪一下sqlplus的过程:
wps1:/oracle$truss -D sqlplus '/as sysdba'
0.0000: execve("/oracle/niyl/bin/sqlplus", 0x2FF22964, 0x2FF22970) argc: 4
0.0226: execve("/oracle/app/oracle/product/10.2.0/db/bin/sqlplus", 0x2FF22964, 0x2FF22970) argc: 2
0.0611: kusla(2, 0x09FFFFFFF000C490) = -1
0.0041: thread_init(0x0900000000739020, 0x09001000A0860350) =
0.0004: sbrk(0x0000000000000000) = 0x00000001100ED448
0.0002: vmgetinfo(0x0FFFFFFFFFFFF570, 7, 16) = 0
0.0003: sbrk(0x0000000000000000) = 0x00000001100ED448
......
0.0002: kioctl(7, 22528, 0x0000000000000000, 0x0000000000000000) = -1
kread(7, "\0 DA '\014\0\001\002\0".., 4096) = 164
0.0002: close(7) = 0
0.0002: __libc_sbrk(0x0000000000010020) = 0x000000011041D9A0
0.0002: kioctl(1, 22528, 0x0000000000000000, 0x0000000000000000) = 0
wps1:/oracle$truss -D sqlplus '/as sysdba'
0.0000: execve("/oracle/niyl/bin/sqlplus", 0x2FF22964, 0x2FF22970) argc: 4
0.0226: execve("/oracle/app/oracle/product/10.2.0/db/bin/sqlplus", 0x2FF22964, 0x2FF22970) argc: 2
0.0611: kusla(2, 0x09FFFFFFF000C490) = -1
0.0041: thread_init(0x0900000000739020, 0x09001000A0860350) =
0.0004: sbrk(0x0000000000000000) = 0x00000001100ED448
0.0002: vmgetinfo(0x0FFFFFFFFFFFF570, 7, 16) = 0
0.0003: sbrk(0x0000000000000000) = 0x00000001100ED448
......
0.0002: kioctl(7, 22528, 0x0000000000000000, 0x0000000000000000) = -1
kread(7, "\0 DA '\014\0\001\002\0".., 4096) = 164
0.0002: close(7) = 0
0.0002: __libc_sbrk(0x0000000000010020) = 0x000000011041D9A0
0.0002: kioctl(1, 22528, 0x0000000000000000, 0x0000000000000000) = 0
kwrite(1, "\n", 1) = 1
SQL*Plus: Release 10.2.0.4.0 - Production on Sat Nov 19 10:59:07 2011
kwrite(1, " S Q L * P l u s : R e".., 70) = 70
SQL*Plus: Release 10.2.0.4.0 - Production on Sat Nov 19 10:59:07 2011
kwrite(1, " S Q L * P l u s : R e".., 70) = 70
kwrite(1, "\n", 1) = 1
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
kwrite(1, " C o p y r i g h t ( c".., 56) = 56
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
kwrite(1, " C o p y r i g h t ( c".., 56) = 56
kwrite(1, "\n", 1) = 1
0.0002: kfcntl(1, F_GETFL, 0x0000000000000008) = 2
0.0002: lseek(4, 512, 0) = 512
kread(4, "17A5\0\0\0\0\0\0\0\0\0\0".., 512) = 512
0.0002: lseek(4, 1024, 0) = 1024
kread(4, "\016\0 *\0 R\0 h\081\09E".., 512) = 512
0.0002: lseek(4, 4608, 0) = 4608
kread(4, "\00F\0A0\0\0\0 b\0A1\0\0".., 512) = 512
0.0003: __libc_sbrk(0x0000000000010020) = 0x000000011042D9C0
0.0003: statx(".", 0x0FFFFFFFFFFF7170, 176, 010) = 0
0.0002: kopen(".", O_RDONLY) = 7
0.0002: getdirent64(7, 0x0000000110434810, 4096) = 680
0.0002: klseek(7, 0, 0, 0x0FFFFFFFFFFF7070) = 0
0.0002: kfcntl(7, F_GETFD, 0x00000001100F34B8) = 0
0.0002: kfcntl(7, F_SETFD, 0x0000000000000001) = 0
0.0002: close(7) = 0
0.0002: statx("/", 0x0FFFFFFFFFFF7390, 176, 020) = 0
0.0002: statx("./", 0x0FFFFFFFFFFF7390, 176, 020) = 0
0.0002: statx("./../", 0x0FFFFFFFFFFF7170, 176, 010) = 0
0.0002: kopen("./../", O_RDONLY) = 7
0.0002: getdirent64(7, 0x0000000110434810, 4096) = 1776
0.0002: klseek(7, 0, 0, 0x0FFFFFFFFFFF7070) = 0
0.0002: kfcntl(7, F_GETFD, 0x00000001100F34B8) = 0
0.0002: kfcntl(7, F_SETFD, 0x0000000000000001) = 0
0.0002: fstatx(7, 0x0FFFFFFFFFFF7390, 176, 020) = 0
0.0002: getdirent64(7, 0x0000000110434810, 4096) = 1776
0.0002: statx("./../.", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../..", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.TTauthority", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.Xauthority", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.bash_history", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.dt", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.dtprofile", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.java", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.mozilla", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.profile", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.rhosts", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.sh_history", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.topasrecrc", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.vi_history", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.vnc", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.wmrc", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../IBM", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../TT_DB", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../admin", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../aixfix", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../audit", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../bin", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../bpmdata", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../dbscripts", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../dev", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../esa", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../etc", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../filedata", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0512: statx("./../ha_script", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../home", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../ihsGSKitUpgradeLog.txt", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../isoxlc", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0003: statx("./../lib", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../lost+found", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../lpp", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../ls.sh", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../mnt", 0x0FFFFFFFFFFF7440, 176, 021) = 0
2.0001: statx("./../nbu_nfs_102", 0x0FFFFFFFFFFF7440, 176, 021) (sleeping...)
NFS server 130.34.3.102 not responding still trying
0.0002: kfcntl(1, F_GETFL, 0x0000000000000008) = 2
0.0002: lseek(4, 512, 0) = 512
kread(4, "17A5\0\0\0\0\0\0\0\0\0\0".., 512) = 512
0.0002: lseek(4, 1024, 0) = 1024
kread(4, "\016\0 *\0 R\0 h\081\09E".., 512) = 512
0.0002: lseek(4, 4608, 0) = 4608
kread(4, "\00F\0A0\0\0\0 b\0A1\0\0".., 512) = 512
0.0003: __libc_sbrk(0x0000000000010020) = 0x000000011042D9C0
0.0003: statx(".", 0x0FFFFFFFFFFF7170, 176, 010) = 0
0.0002: kopen(".", O_RDONLY) = 7
0.0002: getdirent64(7, 0x0000000110434810, 4096) = 680
0.0002: klseek(7, 0, 0, 0x0FFFFFFFFFFF7070) = 0
0.0002: kfcntl(7, F_GETFD, 0x00000001100F34B8) = 0
0.0002: kfcntl(7, F_SETFD, 0x0000000000000001) = 0
0.0002: close(7) = 0
0.0002: statx("/", 0x0FFFFFFFFFFF7390, 176, 020) = 0
0.0002: statx("./", 0x0FFFFFFFFFFF7390, 176, 020) = 0
0.0002: statx("./../", 0x0FFFFFFFFFFF7170, 176, 010) = 0
0.0002: kopen("./../", O_RDONLY) = 7
0.0002: getdirent64(7, 0x0000000110434810, 4096) = 1776
0.0002: klseek(7, 0, 0, 0x0FFFFFFFFFFF7070) = 0
0.0002: kfcntl(7, F_GETFD, 0x00000001100F34B8) = 0
0.0002: kfcntl(7, F_SETFD, 0x0000000000000001) = 0
0.0002: fstatx(7, 0x0FFFFFFFFFFF7390, 176, 020) = 0
0.0002: getdirent64(7, 0x0000000110434810, 4096) = 1776
0.0002: statx("./../.", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../..", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.TTauthority", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.Xauthority", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.bash_history", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.dt", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.dtprofile", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.java", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.mozilla", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.profile", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.rhosts", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.sh_history", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.topasrecrc", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.vi_history", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.vnc", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../.wmrc", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../IBM", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../TT_DB", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../admin", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../aixfix", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../audit", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../bin", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../bpmdata", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../dbscripts", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../dev", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../esa", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../etc", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../filedata", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0512: statx("./../ha_script", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../home", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../ihsGSKitUpgradeLog.txt", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../isoxlc", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0003: statx("./../lib", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../lost+found", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../lpp", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../ls.sh", 0x0FFFFFFFFFFF7440, 176, 021) = 0
0.0002: statx("./../mnt", 0x0FFFFFFFFFFF7440, 176, 021) = 0
2.0001: statx("./../nbu_nfs_102", 0x0FFFFFFFFFFF7440, 176, 021) (sleeping...)
NFS server 130.34.3.102 not responding still trying
Oracle的进程派生的时候,会受操作系统的影响,不同的操作系统行为不同,而这个问题就是发生在
AIX下的。在其他的平台上,则无此现象。
根据文档1316251.1的描述:
Oracle code calls a Unix system call, 'getcwd' to get the current working directory. Then, after that, all the control reverts over to the operating system. From what we can see, the function 'getcwd' calls 'getwd' which in turn calls 'stat'. Once 'stat' is entered it starts processing directory entries in the order shown below by performing a 'statx' call for each entry.
Once the root directory is reached then 'lstat' calls 'statx' for each entry in the directory. Oracle has no control over this processing and there is nothing we can do to prevent it (it is all at the OS level at this point).
根据文档1316251.1的描述:
Oracle code calls a Unix system call, 'getcwd' to get the current working directory. Then, after that, all the control reverts over to the operating system. From what we can see, the function 'getcwd' calls 'getwd' which in turn calls 'stat'. Once 'stat' is entered it starts processing directory entries in the order shown below by performing a 'statx' call for each entry.
Once the root directory is reached then 'lstat' calls 'statx' for each entry in the directory. Oracle has no control over this processing and there is nothing we can do to prevent it (it is all at the OS level at this point).
statx的行为我们可以通过/usr/include/sys/stat.h定义得知,在这一步中,是去获取对应的文件或者目录的相关信息,包括:
device id
file serial number
user id
group id
Time of last access
Time of last data modification
Time of last file status change
Type of fs
device id
file serial number
user id
group id
Time of last access
Time of last data modification
Time of last file status change
Type of fs
......
等等信息
等等信息
在这个hang住的进程中,就是由于nfs网络文件系统无法访问,导致statx hang住了,所以无法连接。
为避免再次出现这样的问题,可以参照文档1316251.1提供的办法,不将nfs挂载在根目录/下
mkdir /nfs
mkdir /nfs/nbu_nfs_102_mount
mount /nfs/nbu_nfs_102_mount
ln -s /nfs/nbu_nfs_102_mount /nbu_nfs_102
mkdir /nfs
mkdir /nfs/nbu_nfs_102_mount
mount /nfs/nbu_nfs_102_mount
ln -s /nfs/nbu_nfs_102_mount /nbu_nfs_102