记录一例 ORA-27300 ORA-27301 ORA-27302

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/xxzhaobb/article/details/80323574

 今天在看崔华的《基于Oracle的SQL优化》一书。看到一个案例,这个案例和之前自己碰到的现象很像,可惜不是自己处理的,而且故障也没有重现过。所以具体原因无法考证。仅仅把书上的解决方法和自己碰到的现象记录下来。

以下图片内容摘自《基于Oracle的SQL优化》一书







--------- 以下内容为发生故障时候,同事收集到的信息

Wed Feb 07 04:53:17 2018
Process startup failed, error stack:
Errors in file /u01/app/oracle/diag/rdbms/sae/sae2/trace/sae2_psp0_6554062.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Process J003 died, see its trace file
kkjcre1p: unable to spawn jobq slave process 
Errors in file /u01/app/oracle/diag/rdbms/sae/sae2/trace/sae2_cjq0_4587790.trc:
Process startup failed, error stack:
Errors in file /u01/app/oracle/diag/rdbms/sae/sae2/trace/sae2_psp0_6554062.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Wed Feb 07 04:53:22 2018
Process J003 died, see its trace file
kkjcre1p: unable to spawn jobq slave process 
Errors in file /u01/app/oracle/diag/rdbms/sae/sae2/trace/sae2_cjq0_4587790.trc:
Wed Feb 07 04:53:27 2018
Process startup failed, error stack:
Errors in file /u01/app/oracle/diag/rdbms/sae/sae2/trace/sae2_psp0_6554062.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Process J003 died, see its trace file

errpt看到的错误信息

$ errpt
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
A6DF45AA   0207131418 I O RMCdaemon      The daemon is started.
C5C09FFA   0207131218 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
9DBCFDEE   0207131418 T O errdemon       ERROR LOGGING TURNED ON
C5C09FFA   0207073818 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207073818 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207073818 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207073818 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207072618 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207072518 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207072518 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207060118 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207060118 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207060118 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207060118 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207060018 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207060018 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207060018 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207060018 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207055318 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207055318 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207054918 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207054918 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207054918 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207054918 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207054718 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207054718 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207054018 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207054018 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207054018 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
C5C09FFA   0207054018 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED
BC3BE5A3   0207045218 P S SRC            SOFTWARE PROGRAM ERROR
4B436A3D   1224082617 T H fscsi1         LINK ERROR
8650BE3F   1128024817 I H ent4           ETHERCHANNEL RECOVERY
42F2355C   1128024717 T H ent0           PROBLEM RESOLVED
59224136   1128024717 P H ent4           ETHERCHANNEL FAILOVER
F1814D51   1128024717 T H ent0           ETHERNET DOWN



Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
              12845134
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
           0
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Wed Feb  7 06:00:09 BEIST 2018
Sequence Number: 697
Machine Id:      00FA51814C00
Node Id:         exporc2
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
               7930094
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
           0
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Wed Feb  7 06:00:09 BEIST 2018
Sequence Number: 696
Machine Id:      00FA51814C00
Node Id:         exporc2
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
              11403714
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
           0
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Wed Feb  7 06:00:09 BEIST 2018
Sequence Number: 695
Machine Id:      00FA51814C00
Node Id:         exporc2
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
               7602464
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
           0
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Wed Feb  7 05:53:33 BEIST 2018
Sequence Number: 694
Machine Id:      00FA51814C00
Node Id:         exporc2
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
              12583146
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
           0
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Wed Feb  7 05:53:31 BEIST 2018
Sequence Number: 693
Machine Id:      00FA51814C00
Node Id:         exporc2
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
               9372060
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
           0
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Wed Feb  7 05:49:03 BEIST 2018
Sequence Number: 692
Machine Id:      00FA51814C00
Node Id:         exporc2
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
               3539928
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
           0
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Wed Feb  7 05:49:01 BEIST 2018
Sequence Number: 691
Machine Id:      00FA51814C00
Node Id:         exporc2
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
               5178002
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
           0
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Wed Feb  7 05:49:01 BEIST 2018
Sequence Number: 690
Machine Id:      00FA51814C00
Node Id:         exporc2
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
               9699744
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
           0
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Wed Feb  7 05:49:01 BEIST 2018
Sequence Number: 689
Machine Id:      00FA51814C00
Node Id:         exporc2
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
               7996270
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
           0
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Wed Feb  7 05:47:06 BEIST 2018
Sequence Number: 688
Machine Id:      00FA51814C00
Node Id:         exporc2
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
               8651320
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
           0
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Wed Feb  7 05:47:06 BEIST 2018
Sequence Number: 687
Machine Id:      00FA51814C00
Node Id:         exporc2
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
              13369476
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
           0
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Wed Feb  7 05:40:39 BEIST 2018
Sequence Number: 686
Machine Id:      00FA51814C00
Node Id:         exporc2
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
               9634226
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
           0
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Wed Feb  7 05:40:39 BEIST 2018
Sequence Number: 685
Machine Id:      00FA51814C00
Node Id:         exporc2
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
               6816194
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
           0
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Wed Feb  7 05:40:39 BEIST 2018
Sequence Number: 684
Machine Id:      00FA51814C00
Node Id:         exporc2
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
               5505686
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
           0
---------------------------------------------------------------------------
LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Wed Feb  7 05:40:39 BEIST 2018
Sequence Number: 683
Machine Id:      00FA51814C00
Node Id:         exporc2
Class:           S
Type:            PERM
WPAR:            Global
Resource Name:   SYSVMM          

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
oracle
USER'S PROCESS ID:
               8651666
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
           0

--- 后记

通过查询MOS,发下的确有类似的文章,提出是AIX的一个bug,不过个人感觉bug的可能性不是很大。因为此事情过去了几个月了,再也没有重现。感觉还是应用程序方面耗光了PGA。可惜当时是别人处理的,重启后,故障消失。也没有抓取当时的AWR,可惜.........


END

oracle自动关闭,提示ORA-27300

10-24

系统是centos7,oracle自动关闭后内存和硬盘使用情况:rnrn```rn[root@wjdb ~]# free -mrntotal used free shared buffers cachedrnMem: 64267 62984 1283 16 277 60243rn-/+ buffers/cache: 2462 61804rnSwap: 35839 0 35839rnrn[root@wjdb ~]# df -hlrnFilesystem Size Used Avail Use% Mounted onrn/dev/vda1 40G 27G 11G 73% /rndevtmpfs 32G 0 32G 0% /devrntmpfs 32G 84K 32G 1% /dev/shmrntmpfs 32G 17M 32G 1% /runrntmpfs 32G 0 32G 0% /sys/fs/cgrouprn/dev/mapper/centosVG-tmp 25G 45M 24G 1% /tmprn/dev/mapper/centosVG-rac 1.1T 958G 113G 90% /datarntmpfs 6.3G 4.0K 6.3G 1% /run/user/0rn```rnrnrn网上说修改logind.conf文件,但问题依旧rnrnrn```rn[root@wjdb ~]# cat /etc/systemd/logind.conf rn[Login]rnRemoveIPC=onrn```rn登录oracle出现如下错误rnrn```rnLast login: Wed Oct 24 18:30:35 CST 2018 on pts/0rnrn[oracle@wjdb ~]$ sqlplus / as sysdbarnrnSQL*Plus: Release 11.2.0.1.0 Production on Wed Oct 24 18:54:06 2018rnCopyright (c) 1982, 2009, Oracle. All rights reserved.rnConnected to an idle instance.rnrnSQL> select * from dual;rnselect * from dualrn*rnERROR at line 1:rnORA-01034: ORACLE not availablernProcess ID: 0rnSession ID: 0 Serial number: 0rn rn```rnrnrnrn检查trace目录的日志文件,如下:rnrnrnrn```rn Starting background process CJQ0rnMon Oct 22 23:45:08 2018rnCJQ0 started with pid=43, OS id=2409rnArchived Log entry 2777 added for thread 1 sequence 2783 ID 0xa5f27e dest 1:rnMon Oct 22 23:45:08 2018rndb_recovery_file_dest_size of 409600 MB is 52.01% used. This is arnuser-specified limit on the amount of space that will be used by thisrndatabase for recovery-related files, and does not reflect the amount ofrnspace available in the underlying filesystem or ASM diskgroup.rnSetting Resource Manager plan SCHEDULER[0x3003]:DEFAULT_MAINTENANCE_PLAN via scheduler windowrnSetting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameterrnMon Oct 22 23:45:11 2018rnStarting background process VKRMrnMon Oct 22 23:45:11 2018rnVKRM started with pid=41, OS id=2411rnMon Oct 22 23:53:18 2018rnStarting background process SMCOrnMon Oct 22 23:53:18 2018rnSMCO started with pid=1030, OS id=12564rnTue Oct 23 02:00:00 2018rnClearing Resource Manager plan via parameterrnTue Oct 23 08:00:17 2018rnThread 1 advanced to log sequence 2785 (LGWR switch)rn Current log# 4 seq# 2785 mem# 0: /data/oracle/oradata/wjdb/redo04.logrnTue Oct 23 08:00:30 2018rnArchived Log entry 2778 added for thread 1 sequence 2784 ID 0xa5f27e dest 1:rnTue Oct 23 16:02:53 2018rnThread 1 advanced to log sequence 2786 (LGWR switch)rn Current log# 1 seq# 2786 mem# 0: /data/oracle/oradata/wjdb/redo01.logrnTue Oct 23 16:03:06 2018rnArchived Log entry 2779 added for thread 1 sequence 2785 ID 0xa5f27e dest 1:rnTue Oct 23 19:07:01 2018rnErrors in file /data/oracle/diag/rdbms/wjdb/wjdb/trace/wjdb_j000_20537.trc:rnORA-27157: OS post/wait facility removedrnORA-27300: OS system dependent operation:semop failed with status: 43rnORA-27301: OS failure message: Identifier removedrnORA-27302: failure occurred at: sskgpwwait1rnTue Oct 23 19:07:01 2018rnErrors in file /data/oracle/diag/rdbms/wjdb/wjdb/trace/wjdb_dbrm_2317.trc:rnORA-27157: OS post/wait facility removedrnORA-27300: OS system dependent operation:semop failed with status: 43rnORA-27301: OS failure message: Identifier removedrnORA-27302: failure occurred at: sskgpwwait1rnDBRM (ospid: 2317): terminating the instance due to error 27157rnTue Oct 23 19:07:01 2018rnErrors in file /data/oracle/diag/rdbms/wjdb/wjdb/trace/wjdb_j001_20539.trc:rnORA-27157: OS post/wait facility removedrnORA-27300: OS system dependent operation:semop failed with status: 43rnORA-27301: OS failure message: Identifier removedrnORA-27302: failure occurred at: sskgpwwait1rnTue Oct 23 19:07:01 2018rnErrors in file /data/oracle/diag/rdbms/wjdb/wjdb/trace/wjdb_q000_2418.trc:rnORA-27157: OS post/wait facility removedrnORA-27300: OS system dependent operation:semop failed with status: 43rnORA-27301: OS failure message: Identifier removedrnORA-27302: failure occurred at: sskgpwwait1rnInstance terminated by DBRM, pid = 2317rnErrors in file /data/oracle/diag/rdbms/wjdb/wjdb/trace/wjdb_dbrm_2317.trc:rnORA-27300: OS system dependent operation:semctl failed with status: 22rnORA-27301: OS failure message: Invalid argumentrnORA-27302: failure occurred at: sskgpwrm1rnORA-27157: OS post/wait facility removedrnORA-27300: OS system dependent operation:semop failed with status: 43rnORA-27301: OS failure message: Identifier removedrnORA-27302: failure occurred at: sskgpwwait1rnTue Oct 23 19:07:04 2018rnInstance termination got error 27120 from SGA destruction.rn Error cleared. Process exiting.rnrnwjdb_j000_20537.trc内容如下rnrn7f30aa2c8000-7f30aa2cb000 r-xp 00000000 fd:01 143235 /usr/lib64/libdl-2.17.sorn7f30aa2cb000-7f30aa4ca000 ---p 00003000 fd:01 143235 /usr/lib64/libdl-2.17.sorn7f30aa4ca000-7f30aa4cb000 r-xp 00002000 fd:01 143235 /usr/lib64/libdl-2.17.sorn7f30aa4cb000-7f30aa4cc000 rwxp 00003000 fd:01 143235 /usr/lib64/libdl-2.17.sorn7f30aa4cc000-7f30aa4cd000 r-xp 00000000 fd:01 136896 /usr/lib64/libaio.so.1.0.1rn7f30aa4cd000-7f30aa6cc000 ---p 00001000 fd:01 136896 /usr/lib64/libaio.so.1.0.1rn7f30aa6cc000-7f30aa6cd000 r-xp 00000000 fd:01 136896 /usr/lib64/libaio.so.1.0.1rn7f30aa6cd000-7f30aa6ce000 rwxp 00001000 fd:01 136896 /usr/lib64/libaio.so.1.0.1rn7f30aa6ce000-7f30aa73d000 r-xp 00000000 fc:01 37623267 /data/oracle/product/11.2.0/db_1/lib/libasmclnt11.sorn7f30aa73d000-7f30aa83d000 ---p 0006f000 fc:01 37623267 /data/oracle/product/11.2.0/db_1/lib/libasmclnt11.sorn7f30aa83d000-7f30aa84f000 rwxp 0006f000 fc:01 37623267 /data/oracle/product/11.2.0/db_1/lib/libasmclnt11.sorn7f30aa84f000-7f30aa850000 rwxp 00000000 00:00 0rn7f30aa850000-7f30aa85b000 r-xp 00000000 fc:01 37622977 /data/oracle/product/11.2.0/db_1/lib/libocrutl11.sorn7f30aa85b000-7f30aa95a000 ---p 0000b000 fc:01 37622977 /data/oracle/product/11.2.0/db_1/lib/libocrutl11.sorn7f30aa95a000-7f30aa95b000 rwxp 0000a000 fc:01 37622977 /data/oracle/product/11.2.0/db_1/lib/libocrutl11.sorn7f30aa95b000-7f30aa9f3000 r-xp 00000000 fc:01 37622976 /data/oracle/product/11.2.0/db_1/lib/libocrb11.sorn7f30aa9f3000-7f30aaaf3000 ---p 00098000 fc:01 37622976 /data/oracle/product/11.2.0/db_1/lib/libocrb11.sorn7f30aaaf3000-7f30aaaf5000 rwxp 00098000 fc:01 37622976 /data/oracle/product/11.2.0/db_1/lib/libocrb11.sorn7f30aaaf5000-7f30aab8e000 r-xp 00000000 fc:01 37622975 /data/oracle/product/11.2.0/db_1/lib/libocr11.sorn7f30aab8e000-7f30aac8d000 ---p 00099000 fc:01 37622975 /data/oracle/product/11.2.0/db_1/lib/libocr11.sorn7f30aac8d000-7f30aac8f000 rwxp 00098000 fc:01 37622975 /data/oracle/product/11.2.0/db_1/lib/libocr11.sorn7f30aac8f000-7f30aac91000 r-xp 00000000 fc:01 37619552 /data/oracle/product/11.2.0/db_1/lib/libskgxn2.sorn7f30aac91000-7f30aad90000 ---p 00002000 fc:01 37619552 /data/oracle/product/11.2.0/db_1/lib/libskgxn2.sorn7f30aad90000-7f30aad91000 rwxp 00001000 fc:01 37619552 /data/oracle/product/11.2.0/db_1/lib/libskgxn2.sorn7f30aad91000-7f30ab1b9000 r-xp 00000000 fc:01 37622972 /data/oracle/product/11.2.0/db_1/lib/libhasgen11.sorn7f30ab1b9000-7f30ab2b9000 ---p 00428000 fc:01 37622972 /data/oracle/product/11.2.0/db_1/lib/libhasgen11.sorn7f30ab2b9000-7f30ab2cb000 rwxp 00428000 fc:01 37622972 /data/oracle/product/11.2.0/db_1/lib/libhasgen11.sorn7f30ab2cb000-7f30ab2cf000 rwxp 00000000 00:00 0rn7f30ab2cf000-7f30ab2f1000 r-xp 00000000 fc:01 37619532 /data/oracle/product/11.2.0/db_1/lib/libdbcfg11.sorn7f30ab2f1000-7f30ab3f0000 ---p 00022000 fc:01 37619532 /data/oracle/product/11.2.0/db_1/lib/libdbcfg11.sorn7f30ab3f0000-7f30ab3f2000 rwxp 00021000 fc:01 37619532 /data/oracle/product/11.2.0/db_1/lib/libdbcfg11.sorn7f30ab3f2000-7f30ab409000 r-xp 00000000 fc:01 37622974 /data/oracle/product/11.2.0/db_1/lib/libclsra11.sorn7f30ab409000-7f30ab509000 ---p 00017000 fc:01 37622974 /data/oracle/product/11.2.0/db_1/lib/libclsra11.sorn7f30ab509000-7f30ab50a000 rwxp 00017000 fc:01 37622974 /data/oracle/product/11.2.0/db_1/lib/libclsra11.sorn7f30ab50a000-7f30ab78f000 r-xp 00000000 fc:01 37623084 /data/oracle/product/11.2.0/db_1/lib/libnnz11.sorn7f30ab78f000-7f30ab88e000 ---p 00285000 fc:01 37623084 /data/oracle/product/11.2.0/db_1/lib/libnnz11.sorn7f30ab88e000-7f30ab8d1000 rwxp 00284000 fc:01 37623084 /data/oracle/product/11.2.0/db_1/lib/libnnz11.sorn7f30ab8d1000-7f30ab8d2000 rwxp 00000000 00:00 0rn7f30ab8d2000-7f30ab8d9000 r-xp 00000000 fd:01 150366 /usr/lib64/librt-2.17.sorn7f30ab8d9000-7f30abad8000 ---p 00007000 fd:01 150366 /usr/lib64/librt-2.17.sorn7f30abad8000-7f30abad9000 r-xp 00006000 fd:01 150366 /usr/lib64/librt-2.17.sorn7f30abad9000-7f30abada000 rwxp 00007000 fd:01 150366 /usr/lib64/librt-2.17.sorn7f30abada000-7f30abb91000 r-xp 00000000 fc:01 37618703 /data/oracle/product/11.2.0/db_1/lib/libskgxp11.sorn7f30abb91000-7f30abc91000 ---p 000b7000 fc:01 37618703 /data/oracle/product/11.2.0/db_1/lib/libskgxp11.sorn7f30abc91000-7f30abc93000 rwxp 000b7000 fc:01 37618703 /data/oracle/product/11.2.0/db_1/lib/libskgxp11.sorn7f30abc93000-7f30abce2000 r-xp 00000000 fc:01 37623266 /data/oracle/product/11.2.0/db_1/lib/libcell11.sorn7f30abce2000-7f30abde1000 ---p 0004f000 fc:01 37623266 /data/oracle/product/11.2.0/db_1/lib/libcell11.sorn7f30abde1000-7f30abde9000 rwxp 0004e000 fc:01 37623266 /data/oracle/product/11.2.0/db_1/lib/libcell11.sorn7f30abde9000-7f30abe0a000 r-xp 00000000 fd:01 132827 /usr/lib64/ld-2.17.sorn7f30abe10000-7f30abe45000 r-xs 00000000 fd:01 401548 /var/db/nscd/hostsrn7f30abe45000-7f30abef7000 rwxp 00000000 00:00 0rn7f30abf04000-7f30abf06000 rwxs 00000000 00:0a 44872319 /[aio] (deleted)rn7f30abf06000-7f30abf07000 rwxp 00000000 00:00 0rn7f30abf07000-7f30abf08000 r-xp 00000000 fc:01 37618636 /data/oracle/product/11.2.0/db_1/lib/libodmd11.sorn7f30abf08000-7f30ac008000 ---p 00001000 fc:01 37618636 /data/oracle/product/11.2.0/db_1/lib/libodmd11.sorn7f30ac008000-7f30ac009000 rwxp 00001000 fc:01 37618636 /data/oracle/product/11.2.0/db_1/lib/libodmd11.sorn7f30ac009000-7f30ac00a000 rwxp 00000000 00:00 0rn7f30ac00a000-7f30ac00b000 r-xp 00021000 fd:01 132827 /usr/lib64/ld-2.17.sorn7f30ac00b000-7f30ac00c000 rwxp 00022000 fd:01 132827 /usr/lib64/ld-2.17.sorn7f30ac00c000-7f30ac00d000 rwxp 00000000 00:00 0rn7fffa52c6000-7fffa52fa000 rwxp 00000000 00:00 0 [stack]rn7fffa53fe000-7fffa5400000 r-xp 00000000 00:00 0 [vdso]rnffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]rn******************* End of process map dump ************rnUnexpected error 27157 in job slave processrnORA-27157: OS post/wait facility removedrnORA-27300: OS system dependent operation:semop failed with status: 43rnORA-27301: OS failure message: Identifier removedrnORA-27302: failure occurred at: sskgpwwait1rnrnrnwjdb_dbrm_2317.trc内容如下rnrn7f2cbbad0000-7f2cbbad1000 rwxp 00007000 fd:01 150366 /usr/lib64/librt-2.17.sorn7f2cbbad1000-7f2cbbb88000 r-xp 00000000 fc:01 37618703 /data/oracle/product/11.2.0/db_1/lib/libskgxp11.sorn7f2cbbb88000-7f2cbbc88000 ---p 000b7000 fc:01 37618703 /data/oracle/product/11.2.0/db_1/lib/libskgxp11.sorn7f2cbbc88000-7f2cbbc8a000 rwxp 000b7000 fc:01 37618703 /data/oracle/product/11.2.0/db_1/lib/libskgxp11.sorn7f2cbbc8a000-7f2cbbcd9000 r-xp 00000000 fc:01 37623266 /data/oracle/product/11.2.0/db_1/lib/libcell11.sorn7f2cbbcd9000-7f2cbbdd8000 ---p 0004f000 fc:01 37623266 /data/oracle/product/11.2.0/db_1/lib/libcell11.sorn7f2cbbdd8000-7f2cbbde0000 rwxp 0004e000 fc:01 37623266 /data/oracle/product/11.2.0/db_1/lib/libcell11.sorn7f2cbbde0000-7f2cbbe01000 r-xp 00000000 fd:01 132827 /usr/lib64/ld-2.17.sorn7f2cbbe07000-7f2cbbe3c000 r-xs 00000000 fd:01 401548 /var/db/nscd/hostsrn7f2cbbe3c000-7f2cbbeee000 rwxp 00000000 00:00 0rn7f2cbbefa000-7f2cbbefb000 rwxs 00000000 fc:01 37492061 /data/oracle/product/11.2.0/db_1/dbs/hc_wjdb.datrn7f2cbbefb000-7f2cbbefd000 rwxs 00000000 00:0a 26080 /[aio] (deleted)rn7f2cbbefd000-7f2cbbefe000 rwxp 00000000 00:00 0rn7f2cbbefe000-7f2cbbeff000 r-xp 00000000 fc:01 37618636 /data/oracle/product/11.2.0/db_1/lib/libodmd11.sorn7f2cbbeff000-7f2cbbfff000 ---p 00001000 fc:01 37618636 /data/oracle/product/11.2.0/db_1/lib/libodmd11.sorn7f2cbbfff000-7f2cbc000000 rwxp 00001000 fc:01 37618636 /data/oracle/product/11.2.0/db_1/lib/libodmd11.sorn7f2cbc000000-7f2cbc001000 rwxp 00000000 00:00 0rn7f2cbc001000-7f2cbc002000 r-xp 00021000 fd:01 132827 /usr/lib64/ld-2.17.sorn7f2cbc002000-7f2cbc003000 rwxp 00022000 fd:01 132827 /usr/lib64/ld-2.17.sorn7f2cbc003000-7f2cbc004000 rwxp 00000000 00:00 0rn7fff386d3000-7fff38705000 rwxp 00000000 00:00 0 [stack]rn7fff387fe000-7fff38800000 r-xp 00000000 00:00 0 [vdso]rnffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]rn******************* End of process map dump ************rnerror 27157 detected in background processrnORA-27157: OS post/wait facility removedrnORA-27300: OS system dependent operation:semop failed with status: 43rnORA-27301: OS failure message: Identifier removedrnORA-27302: failure occurred at: sskgpwwait1rnrn*** 2018-10-23 19:07:01.746rnDBRM (ospid: 2317): terminating the instance due to error 27157rnrn*** 2018-10-23 19:07:04.048rnORA-27300: OS system dependent operation:semctl failed with status: 22rnORA-27301: OS failure message: Invalid argumentrnORA-27302: failure occurred at: sskgpwrm1rnORA-27157: OS post/wait facility removedrnORA-27300: OS system dependent operation:semop failed with status: 43rnORA-27301: OS failure message: Identifier removedrnORA-27302: failure occurred at: sskgpwwait1rnskgm error 27120: errno = 22, info = 5, 229378, 0, 0, 229378, 0, 11rnskgm error 27120: errno = 22, info = 5, 262147, 0, 0, 262147, 0, 11rnskgm error 27120: errno = 22, info = 5, 294916, 0, 0, 294916, 0, 11rnskgm error 27120: errno = 22, info = 5, 327685, 0, 0, 327685, 0, 11rnskgm error 27120: errno = 22, info = 5, 360454, 0, 0, 360454, 0, 11rnskgm error 27120: errno = 22, info = 5, 393223, 0, 0, 393223, 0, 11rnskgm error 27120: errno = 22, info = 5, 425992, 0, 0, 425992, 0, 11rnskgm error 27120: errno = 22, info = 5, 458761, 0, 0, 458761, 0, 11rnskgm error 27120: errno = 22, info = 5, 491530, 0, 0, 491530, 0, 11rnskgm error 27120: errno = 22, info = 5, 524299, 0, 0, 524299, 0, 11rnskgm error 27120: errno = 22, info = 5, 557068, 0, 0, 557068, 0, 11rnskgm error 27120: errno = 22, info = 5, 589837, 0, 0, 589837, 0, 11rnskgm error 27120: errno = 22, info = 5, 622606, 0, 0, 622606, 0, 11rnskgm error 27120: errno = 22, info = 5, 655375, 0, 0, 655375, 0, 11rnskgm error 27120: errno = 22, info = 5, 688144, 0, 0, 688144, 0, 11rnskgm error 27120: errno = 22, info = 5, 720913, 0, 0, 720913, 0, 11rnskgm error 27120: errno = 22, info = 5, 753682, 0, 0, 753682, 0, 11rnskgm error 27120: errno = 22, info = 5, 786451, 0, 0, 786451, 0, 11rnskgm error 27120: errno = 22, info = 5, 819220, 0, 0, 819220, 0, 11rnskgm error 27120: errno = 22, info = 5, 851989, 0, 0, 851989, 0, 11rnskgm error 27120: errno = 22, info = 5, 884758, 0, 0, 884758, 0, 11rnskgm error 27120: errno = 22, info = 5, 917527, 0, 0, 917527, 0, 11rnskgm error 27120: errno = 22, info = 5, 950296, 0, 0, 950296, 0, 11rnskgm error 27120: errno = 22, info = 5, 983065, 0, 0, 983065, 0, 11rnskgm error 27120: errno = 22, info = 5, 1015834, 0, 0, 1015834, 0, 11rnskgm error 27120: errno = 22, info = 5, 1048603, 0, 0, 1048603, 0, 11rnskgm error 27120: errno = 22, info = 5, 1081372, 0, 0, 1081372, 0, 11rnrnrnwjdb_j001_20539.trc内容如下rnrn7f9d46ac3000-7f9d46b32000 r-xp 00000000 fc:01 37623267 /data/oracle/product/11.2.0/db_1/lib/libasmclnt11.sorn7f9d46b32000-7f9d46c32000 ---p 0006f000 fc:01 37623267 /data/oracle/product/11.2.0/db_1/lib/libasmclnt11.sorn7f9d46c32000-7f9d46c44000 rwxp 0006f000 fc:01 37623267 /data/oracle/product/11.2.0/db_1/lib/libasmclnt11.sorn7f9d46c44000-7f9d46c45000 rwxp 00000000 00:00 0rn7f9d46c45000-7f9d46c50000 r-xp 00000000 fc:01 37622977 /data/oracle/product/11.2.0/db_1/lib/libocrutl11.sorn7f9d46c50000-7f9d46d4f000 ---p 0000b000 fc:01 37622977 /data/oracle/product/11.2.0/db_1/lib/libocrutl11.sorn7f9d46d4f000-7f9d46d50000 rwxp 0000a000 fc:01 37622977 /data/oracle/product/11.2.0/db_1/lib/libocrutl11.sorn7f9d46d50000-7f9d46de8000 r-xp 00000000 fc:01 37622976 /data/oracle/product/11.2.0/db_1/lib/libocrb11.sorn7f9d46de8000-7f9d46ee8000 ---p 00098000 fc:01 37622976 /data/oracle/product/11.2.0/db_1/lib/libocrb11.sorn7f9d46ee8000-7f9d46eea000 rwxp 00098000 fc:01 37622976 /data/oracle/product/11.2.0/db_1/lib/libocrb11.sorn7f9d46eea000-7f9d46f83000 r-xp 00000000 fc:01 37622975 /data/oracle/product/11.2.0/db_1/lib/libocr11.sorn7f9d46f83000-7f9d47082000 ---p 00099000 fc:01 37622975 /data/oracle/product/11.2.0/db_1/lib/libocr11.sorn7f9d47082000-7f9d47084000 rwxp 00098000 fc:01 37622975 /data/oracle/product/11.2.0/db_1/lib/libocr11.sorn7f9d47084000-7f9d47086000 r-xp 00000000 fc:01 37619552 /data/oracle/product/11.2.0/db_1/lib/libskgxn2.sorn7f9d47086000-7f9d47185000 ---p 00002000 fc:01 37619552 /data/oracle/product/11.2.0/db_1/lib/libskgxn2.sorn7f9d47185000-7f9d47186000 rwxp 00001000 fc:01 37619552 /data/oracle/product/11.2.0/db_1/lib/libskgxn2.sorn7f9d47186000-7f9d475ae000 r-xp 00000000 fc:01 37622972 /data/oracle/product/11.2.0/db_1/lib/libhasgen11.sorn7f9d475ae000-7f9d476ae000 ---p 00428000 fc:01 37622972 /data/oracle/product/11.2.0/db_1/lib/libhasgen11.sorn7f9d476ae000-7f9d476c0000 rwxp 00428000 fc:01 37622972 /data/oracle/product/11.2.0/db_1/lib/libhasgen11.sorn7f9d476c0000-7f9d476c4000 rwxp 00000000 00:00 0rn7f9d476c4000-7f9d476e6000 r-xp 00000000 fc:01 37619532 /data/oracle/product/11.2.0/db_1/lib/libdbcfg11.sorn7f9d476e6000-7f9d477e5000 ---p 00022000 fc:01 37619532 /data/oracle/product/11.2.0/db_1/lib/libdbcfg11.sorn7f9d477e5000-7f9d477e7000 rwxp 00021000 fc:01 37619532 /data/oracle/product/11.2.0/db_1/lib/libdbcfg11.sorn7f9d477e7000-7f9d477fe000 r-xp 00000000 fc:01 37622974 /data/oracle/product/11.2.0/db_1/lib/libclsra11.sorn7f9d477fe000-7f9d478fe000 ---p 00017000 fc:01 37622974 /data/oracle/product/11.2.0/db_1/lib/libclsra11.sorn7f9d478fe000-7f9d478ff000 rwxp 00017000 fc:01 37622974 /data/oracle/product/11.2.0/db_1/lib/libclsra11.sorn7f9d478ff000-7f9d47b84000 r-xp 00000000 fc:01 37623084 /data/oracle/product/11.2.0/db_1/lib/libnnz11.sorn7f9d47b84000-7f9d47c83000 ---p 00285000 fc:01 37623084 /data/oracle/product/11.2.0/db_1/lib/libnnz11.sorn7f9d47c83000-7f9d47cc6000 rwxp 00284000 fc:01 37623084 /data/oracle/product/11.2.0/db_1/lib/libnnz11.sorn7f9d47cc6000-7f9d47cc7000 rwxp 00000000 00:00 0rn7f9d47cc7000-7f9d47cce000 r-xp 00000000 fd:01 150366 /usr/lib64/librt-2.17.sorn7f9d47cce000-7f9d47ecd000 ---p 00007000 fd:01 150366 /usr/lib64/librt-2.17.sorn7f9d47ecd000-7f9d47ece000 r-xp 00006000 fd:01 150366 /usr/lib64/librt-2.17.sorn7f9d47ece000-7f9d47ecf000 rwxp 00007000 fd:01 150366 /usr/lib64/librt-2.17.sorn7f9d47ecf000-7f9d47f86000 r-xp 00000000 fc:01 37618703 /data/oracle/product/11.2.0/db_1/lib/libskgxp11.sorn7f9d47f86000-7f9d48086000 ---p 000b7000 fc:01 37618703 /data/oracle/product/11.2.0/db_1/lib/libskgxp11.sorn7f9d48086000-7f9d48088000 rwxp 000b7000 fc:01 37618703 /data/oracle/product/11.2.0/db_1/lib/libskgxp11.sorn7f9d48088000-7f9d480d7000 r-xp 00000000 fc:01 37623266 /data/oracle/product/11.2.0/db_1/lib/libcell11.sorn7f9d480d7000-7f9d481d6000 ---p 0004f000 fc:01 37623266 /data/oracle/product/11.2.0/db_1/lib/libcell11.sorn7f9d481d6000-7f9d481de000 rwxp 0004e000 fc:01 37623266 /data/oracle/product/11.2.0/db_1/lib/libcell11.sorn7f9d481de000-7f9d481ff000 r-xp 00000000 fd:01 132827 /usr/lib64/ld-2.17.sorn7f9d4823a000-7f9d482ec000 rwxp 00000000 00:00 0rn7f9d482f9000-7f9d482fb000 rwxs 00000000 00:0a 44870643 /[aio] (deleted)rn7f9d482fb000-7f9d482fc000 rwxp 00000000 00:00 0rn7f9d482fc000-7f9d482fd000 r-xp 00000000 fc:01 37618636 /data/oracle/product/11.2.0/db_1/lib/libodmd11.sorn7f9d482fd000-7f9d483fd000 ---p 00001000 fc:01 37618636 /data/oracle/product/11.2.0/db_1/lib/libodmd11.sorn7f9d483fd000-7f9d483fe000 rwxp 00001000 fc:01 37618636 /data/oracle/product/11.2.0/db_1/lib/libodmd11.sorn7f9d483fe000-7f9d483ff000 rwxp 00000000 00:00 0rn7f9d483ff000-7f9d48400000 r-xp 00021000 fd:01 132827 /usr/lib64/ld-2.17.sorn7f9d48400000-7f9d48401000 rwxp 00022000 fd:01 132827 /usr/lib64/ld-2.17.sorn7f9d48401000-7f9d48402000 rwxp 00000000 00:00 0rn7fff9ed9c000-7fff9edcf000 rwxp 00000000 00:00 0 [stack]rn7fff9edfe000-7fff9ee00000 r-xp 00000000 00:00 0 [vdso]rnffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]rn******************* End of process map dump ************rnUnexpected error 27157 in job slave processrnORA-27157: OS post/wait facility removedrnORA-27300: OS system dependent operation:semop failed with status: 43rnORA-27301: OS failure message: Identifier removedrnORA-27302: failure occurred at: sskgpwwait1rnrnwjdb_q000_2418.trc内容如下rnrn7f71f86ef000-7f71f86f0000 rwxp 00000000 00:00 0rn7f71f86f0000-7f71f86fb000 r-xp 00000000 fc:01 37622977 /data/oracle/product/11.2.0/db_1/lib/libocrutl11.sorn7f71f86fb000-7f71f87fa000 ---p 0000b000 fc:01 37622977 /data/oracle/product/11.2.0/db_1/lib/libocrutl11.sorn7f71f87fa000-7f71f87fb000 rwxp 0000a000 fc:01 37622977 /data/oracle/product/11.2.0/db_1/lib/libocrutl11.sorn7f71f87fb000-7f71f8893000 r-xp 00000000 fc:01 37622976 /data/oracle/product/11.2.0/db_1/lib/libocrb11.sorn7f71f8893000-7f71f8993000 ---p 00098000 fc:01 37622976 /data/oracle/product/11.2.0/db_1/lib/libocrb11.sorn7f71f8993000-7f71f8995000 rwxp 00098000 fc:01 37622976 /data/oracle/product/11.2.0/db_1/lib/libocrb11.sorn7f71f8995000-7f71f8a2e000 r-xp 00000000 fc:01 37622975 /data/oracle/product/11.2.0/db_1/lib/libocr11.sorn7f71f8a2e000-7f71f8b2d000 ---p 00099000 fc:01 37622975 /data/oracle/product/11.2.0/db_1/lib/libocr11.sorn7f71f8b2d000-7f71f8b2f000 rwxp 00098000 fc:01 37622975 /data/oracle/product/11.2.0/db_1/lib/libocr11.sorn7f71f8b2f000-7f71f8b31000 r-xp 00000000 fc:01 37619552 /data/oracle/product/11.2.0/db_1/lib/libskgxn2.sorn7f71f8b31000-7f71f8c30000 ---p 00002000 fc:01 37619552 /data/oracle/product/11.2.0/db_1/lib/libskgxn2.sorn7f71f8c30000-7f71f8c31000 rwxp 00001000 fc:01 37619552 /data/oracle/product/11.2.0/db_1/lib/libskgxn2.sorn7f71f8c31000-7f71f9059000 r-xp 00000000 fc:01 37622972 /data/oracle/product/11.2.0/db_1/lib/libhasgen11.sorn7f71f9059000-7f71f9159000 ---p 00428000 fc:01 37622972 /data/oracle/product/11.2.0/db_1/lib/libhasgen11.sorn7f71f9159000-7f71f916b000 rwxp 00428000 fc:01 37622972 /data/oracle/product/11.2.0/db_1/lib/libhasgen11.sorn7f71f916b000-7f71f916f000 rwxp 00000000 00:00 0rn7f71f916f000-7f71f9191000 r-xp 00000000 fc:01 37619532 /data/oracle/product/11.2.0/db_1/lib/libdbcfg11.sorn7f71f9191000-7f71f9290000 ---p 00022000 fc:01 37619532 /data/oracle/product/11.2.0/db_1/lib/libdbcfg11.sorn7f71f9290000-7f71f9292000 rwxp 00021000 fc:01 37619532 /data/oracle/product/11.2.0/db_1/lib/libdbcfg11.sorn7f71f9292000-7f71f92a9000 r-xp 00000000 fc:01 37622974 /data/oracle/product/11.2.0/db_1/lib/libclsra11.sorn7f71f92a9000-7f71f93a9000 ---p 00017000 fc:01 37622974 /data/oracle/product/11.2.0/db_1/lib/libclsra11.sorn7f71f93a9000-7f71f93aa000 rwxp 00017000 fc:01 37622974 /data/oracle/product/11.2.0/db_1/lib/libclsra11.sorn7f71f93aa000-7f71f962f000 r-xp 00000000 fc:01 37623084 /data/oracle/product/11.2.0/db_1/lib/libnnz11.sorn7f71f962f000-7f71f972e000 ---p 00285000 fc:01 37623084 /data/oracle/product/11.2.0/db_1/lib/libnnz11.sorn7f71f972e000-7f71f9771000 rwxp 00284000 fc:01 37623084 /data/oracle/product/11.2.0/db_1/lib/libnnz11.sorn7f71f9771000-7f71f9772000 rwxp 00000000 00:00 0rn7f71f9772000-7f71f9779000 r-xp 00000000 fd:01 150366 /usr/lib64/librt-2.17.sorn7f71f9779000-7f71f9978000 ---p 00007000 fd:01 150366 /usr/lib64/librt-2.17.sorn7f71f9978000-7f71f9979000 r-xp 00006000 fd:01 150366 /usr/lib64/librt-2.17.sorn7f71f9979000-7f71f997a000 rwxp 00007000 fd:01 150366 /usr/lib64/librt-2.17.sorn7f71f997a000-7f71f9a31000 r-xp 00000000 fc:01 37618703 /data/oracle/product/11.2.0/db_1/lib/libskgxp11.sorn7f71f9a31000-7f71f9b31000 ---p 000b7000 fc:01 37618703 /data/oracle/product/11.2.0/db_1/lib/libskgxp11.sorn7f71f9b31000-7f71f9b33000 rwxp 000b7000 fc:01 37618703 /data/oracle/product/11.2.0/db_1/lib/libskgxp11.sorn7f71f9b33000-7f71f9b82000 r-xp 00000000 fc:01 37623266 /data/oracle/product/11.2.0/db_1/lib/libcell11.sorn7f71f9b82000-7f71f9c81000 ---p 0004f000 fc:01 37623266 /data/oracle/product/11.2.0/db_1/lib/libcell11.sorn7f71f9c81000-7f71f9c89000 rwxp 0004e000 fc:01 37623266 /data/oracle/product/11.2.0/db_1/lib/libcell11.sorn7f71f9c89000-7f71f9caa000 r-xp 00000000 fd:01 132827 /usr/lib64/ld-2.17.sorn7f71f9ce5000-7f71f9d97000 rwxp 00000000 00:00 0rn7f71f9da4000-7f71f9da6000 rwxs 00000000 00:0a 28167 /[aio] (deleted)rn7f71f9da6000-7f71f9da7000 rwxp 00000000 00:00 0rn7f71f9da7000-7f71f9da8000 r-xp 00000000 fc:01 37618636 /data/oracle/product/11.2.0/db_1/lib/libodmd11.sorn7f71f9da8000-7f71f9ea8000 ---p 00001000 fc:01 37618636 /data/oracle/product/11.2.0/db_1/lib/libodmd11.sorn7f71f9ea8000-7f71f9ea9000 rwxp 00001000 fc:01 37618636 /data/oracle/product/11.2.0/db_1/lib/libodmd11.sorn7f71f9ea9000-7f71f9eaa000 rwxp 00000000 00:00 0rn7f71f9eaa000-7f71f9eab000 r-xp 00021000 fd:01 132827 /usr/lib64/ld-2.17.sorn7f71f9eab000-7f71f9eac000 rwxp 00022000 fd:01 132827 /usr/lib64/ld-2.17.sorn7f71f9eac000-7f71f9ead000 rwxp 00000000 00:00 0rn7fff92da9000-7fff92ddc000 rwxp 00000000 00:00 0 [stack]rn7fff92dfe000-7fff92e00000 r-xp 00000000 00:00 0 [vdso]rnffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]rn******************* End of process map dump ************rnKSV 27157 error in slave processrnrn*** 2018-10-23 19:07:01.765rnORA-27157: OS post/wait facility removedrnORA-27300: OS system dependent operation:semop failed with status: 43rnORA-27301: OS failure message: Identifier removedrnORA-27302: failure occurred at: sskgpwwait1rnOPIRIP: Uncaught error 447. Error stack:rnORA-00447: fatal error in background processrnORA-27157: OS post/wait facility removedrnORA-27300: OS system dependent operation:semop failed with status: 43rnORA-27301: OS failure message: Identifier removedrnORA-27302: failure occurred at: sskgpwwait1rnrn```rnrnrn

没有更多推荐了,返回首页

私密
私密原因:
请选择设置私密原因
  • 广告
  • 抄袭
  • 版权
  • 政治
  • 色情
  • 无意义
  • 其他
其他原因:
120
出错啦
系统繁忙,请稍后再试

关闭