因重建oracle用户和组引起的ORA-27121错误

今天在做恢复测试的时候遇到ORA-27121的错误，具体的表现为用sqlplus登录oracle的时候提示Connected to an idle instance，但是通过ps -ef仍然能够看到oracle的进程存在。

一、故障现象

oracle@bfdx-dsg:[/oracle]rman target / nocatalog

Recovery Manager: Release 10.2.0.2.0 - Production on Fri May 16 09:52:17 2008

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

connected to target database (not started)

RMAN> run {
2>  allocate channel t1 type 'sbt_tape';
3>  send 'NB_ORA_SERV=backsrv,NB_ORA_CLIENT=sx_kf_svc';
4>  allocate channel t2 type 'sbt_tape';
5>  send 'NB_ORA_SERV=backsrv,NB_ORA_CLIENT=sx_kf_svc';
6>  restore database;
7>  release channel t1;
8>  release channel t2;
9>  }

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of allocate command at 05/16/2008 09:52:31
RMAN-06403: could not obtain a fully authorized session
ORA-01034: ORACLE not available
ORA-27121: unable to determine size of shared memory segment
IBM AIX RISC System/6000 Error: 13: Permission denied

RMAN> exit
Recovery Manager complete.

oracle@bfdx-dsg:[/oracle]id
uid=300(oracle) gid=300(dba) groups=1(staff)

oracle@bfdx-dsg:[/oracle]sqlplus /nolog

SQL*Plus: Release 10.2.0.2.0 - Production on Fri May 16 09:53:13 2008
Copyright (c) 1982, 2005, Oracle.  All Rights Reserved.

SQL> conn /as sysdba
Connected to an idle instance.

SQL> startup nomount
ORA-27121: unable to determine size of shared memory segment
IBM AIX RISC System/6000 Error: 13: Permission denied

SQL> exit
Disconnected

oracle@bfdx-dsg:[/oracle/app/oracle/admin/sibss/bdump]ps -ef|grep ora
  oracle 290896      1   0   May 14      -  0:30 ora_ckpt_sibss
  oracle 430180 639092   0 09:53:27  pts/1  0:00 sqlplus
  oracle 450618      1   0   May 14      -  0:02 ora_lgwr_sibss
  oracle 503824 671794   0 09:55:05  pts/0  0:00 ps -ef
  oracle 512110      1   0   May 14      -  0:03 ora_dbw0_sibss
  oracle 520252      1   0   May 14      -  0:04 ora_mman_sibss
  oracle 532528      1   0   May 14      -  0:02 ora_psp0_sibss
  oracle 561394      1   0   May 14      -  0:28 ora_pmon_sibss
  oracle 565438      1   0   May 14      -  0:02 ora_smon_sibss
  oracle 610504 430180   0 09:53:27      -  0:00 oraclesibss (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
  oracle 622730      1   0   May 14      -  0:00 ora_reco_sibss
  oracle 639092 635114   0 09:53:21  pts/1  0:00 -ksh
  oracle 647392      1   0   May 14      -  0:04 ora_mmon_sibss
  oracle 651310 671794   0 09:55:05  pts/0  0:00 grep ora
  oracle 671794 663674   0 09:52:00  pts/0  0:00 -ksh
  oracle 680056      1   0   May 14      -  0:01 ora_mmnl_sibss
  oracle 692432      1   0   May 14      -  0:01 ora_cjq0_sibss

oracle@bfdx-dsg:[/oracle/app/oracle/admin/sibss/bdump]sqlplus '/as sysdba'

SQL*Plus: Release 10.2.0.2.0 - Production on Fri May 16 09:55:15 2008
Copyright (c) 1982, 2005, Oracle.  All Rights Reserved.
Connected to an idle instance.

SQL> shutdown immediate
ORA-01034: ORACLE not available
ORA-27121: unable to determine size of shared memory segment
IBM AIX RISC System/6000 Error: 13: Permission denied

SQL> exit
Disconnected

二、查看日志文件，没有发现报错

oracle@bfdx-dsg:[/oracle]echo $ORACLE_BASE
/oracle/app/oracle

oracle@bfdx-dsg:[/oracle/app/oracle/admin/sibss/bdump]tail -f *.log
Setting recovery target incarnation to 2
Wed May 14 15:21:21 2008
Successful mount of redo thread 1, with mount id 1604448301
Wed May 14 15:21:21 2008
Database mounted in Exclusive Mode
Completed: ALTER DATABASE   MOUNT
Fri May 16 09:53:33 2008
Starting ORACLE instance (normal)
Fri May 16 09:54:01 2008
Starting ORACLE instance (normal)

三、初步判断是共享内存段出现了问题

oracle@bfdx-dsg:[/oracle/app/oracle/admin/sibss/bdump]ipcs -pmb
IPC status from /dev/mem as of Fri May 16 09:56:15 BEIDT 2008
T        ID     KEY        MODE       OWNER    GROUP     SEGSZ  CPID  LPID
Shared Memory:
m         0 0x58002034 --rw-rw-rw-     root   system 134217728 61596 278664
m   1048577 0x52e74b4f --rw-rw-rw-   imnadm   imnadm     36028 422096 422096
m   1048578 0x9308e451 --rw-rw-rw-   imnadm   imnadm     97948 422096 422096
m   1048579 0xe4663d62 --rw-rw-rw-   imnadm   imnadm        96 422096 422096
m         4 0xc76283cc --rw-rw-rw-   imnadm   imnadm     42268 422096 422096
m         5 0x298ee665 --rw-rw-rw-   imnadm   imnadm      2844 422096 422096
m         6 0xffffffff --rw-rw----     root   system      4096 381128 381128
m         7 0x78000019 --rw-rw-rw-     root   system  16777216 483570 483570
m         8 0x0d003a64 --rw-rw----     root   system      1440 344086 635114
m   8388617 0xc4d142b0 --rw-r-----      205      204 6442467328 618702 638976
m   1048586 0x00005643 --rw-rw-rw-     root   system      1024 659660 688262
m   1048587 0x00005654 --rw-rw-rw-     root   system      1024 659660 688262

在做测试的时候，由于测试机的oracle用户和dba组的id和生产库不一致，所以删除后重建了。
m   8388617 0xc4d142b0 --rw-r-----      205      204 6442467328 618702 638976
这个共享段的205和204分别是删除之前的oracle用户和dba组id。

四、解决步骤

--杀掉oracle相关的进程
oracle@bfdx-dsg:[/oracle]ps -ef|grep $ORACLE_SID| /
> grep -v grep|awk '{print $2}'|xargs -i kill -9 {}

oracle@bfdx-dsg:[/oracle]ps -ef|grep ora
  oracle 430180 639092   0 09:53:27  pts/1  0:00 sqlplus
  oracle 639092 635114   0 09:53:21  pts/1  0:00 -ksh
  oracle 671800 663674   0 10:00:38  pts/0  0:00 -ksh
  oracle 680058 671800   0 10:01:05  pts/0  0:00 grep ora
  oracle 692434 671800   0 10:01:05  pts/0  0:00 ps -ef

--清除数据库占用的内存段
oracle@bfdx-dsg:[/oracle]ipcrm -m 8388617
ipcrm: 0515-021 shmid(8388617): The user does not have the appropriate privileges to perform this operation.

oracle@bfdx-dsg:[/oracle]exit

bfdx-dsg:root:/acct>ipcs -pmb
IPC status from /dev/mem as of Fri May 16 10:03:14 BEIDT 2008
T        ID     KEY        MODE       OWNER    GROUP     SEGSZ  CPID  LPID
Shared Memory:
m         0 0x58002034 --rw-rw-rw-     root   system 134217728 61596 278664
m   1048577 0x52e74b4f --rw-rw-rw-   imnadm   imnadm     36028 422096 422096
m   1048578 0x9308e451 --rw-rw-rw-   imnadm   imnadm     97948 422096 422096
m   1048579 0xe4663d62 --rw-rw-rw-   imnadm   imnadm        96 422096 422096
m         4 0xc76283cc --rw-rw-rw-   imnadm   imnadm     42268 422096 422096
m         5 0x298ee665 --rw-rw-rw-   imnadm   imnadm      2844 422096 422096
m         6 0xffffffff --rw-rw----     root   system      4096 381128 381128
m         7 0x78000019 --rw-rw-rw-     root   system  16777216 483570 483570
m         8 0x0d003a64 --rw-rw----     root   system      1440 344086 635114
m   8388617 0xc4d142b0 --rw-r-----      205      204 6442467328 618702 692432
m   1048586 0x00005643 --rw-rw-rw-     root   system      1024 659660 688262
m   1048587 0x00005654 --rw-rw-rw-     root   system      1024 659660 688262

bfdx-dsg:root:/acct>ipcrm -m 8388617
bfdx-dsg:root:/acct>su - oracle
oracle@bfdx-dsg:[/oracle]sqlplus '/as sysdba'

SQL*Plus: Release 10.2.0.2.0 - Production on Fri May 16 10:03:35 2008
Copyright (c) 1982, 2005, Oracle.  All Rights Reserved.
Connected to an idle instance.

SQL> startup nomount
ORACLE instance started.

Total System Global Area 6442450944 bytes
Fixed Size                  2080904 bytes
Variable Size            2013269880 bytes
Database Buffers         4412407808 bytes
Redo Buffers               14692352 bytes

OK，问题解决。