Troubleshooting Oracle Background Processes

Master Note: Troubleshooting Oracle Background Processes (Doc ID 1509616.1) To Bottom Modified:26-Sep-2013Type:TROUBLESHOOTING In this Document   Purpose   Troubleshooting Steps   Concepts   Issues Involving Oracle Background Processes   ORA- errors related to Background Processes   ORA-443 background process did not start   ORA-444 background process failed while starting   ORA-445 background process did not start after seconds   ORA-446 background process started when not expected   ORA-447 fatal error in background process   ORA-448 normal completion of background process   ORA-449 background process unexpectedly terminated with error   ORA-450 background process '%s' did not start   Generic issues not related to a specific background process   CJQ0 / Jnnn ( Job Queue Coordinator Process / Job Queue Slave Processes )   Issues Specific to CJQ0 / Jnnn   CKPT ( Checkpoint Process )   Issues Specific to CKPT   DBWn ( Database Writer Process )   Issues Specific to DBWn   EMNC / Ennn ( EMON Coordinator Process / EMON Slave Process )   Issues Specific to EMNC / Ennn   LGWR ( Log Writer Process )   Issues Specific to LGWR   MMON / Mnnn ( Manageability Monitor Process / MMON Slave Process )   Issues Specific to MMON / Mnnn   PMON ( Process Monitor )   Issues Specific to PMON   QMNC / QMNn ( AQ Coordinator Process / AQ Server Class Process )   Issues Specific to QMNC / QMNn   RECO ( Recoverer Process )   Issues Specific to RECO   SMON ( System Monitor Process )   Issues Specific to SMON   VKTM ( Virtual Keeper of Time Process )   Issues Specific to VKTM   Wnnn ( Space Management Slave Process )   Issues Specific to Wnnn   Helpful Articles on Background Processes   Further Diagnostics   References APPLIES TO: Oracle Database - Enterprise Edition - Version 9.2.0.1 and later Information in this document applies to any platform. PURPOSE This document is intended to assist Database Administrators resolve issues encountered involving Oracle Background Processes. It mainly applies to a single instance environment and looks at a basic typical database configuration. TROUBLESHOOTING STEPS Concepts To maximize performance and accommodate many users, a multiprocess Oracle Database system uses background processes. Background processes consolidate functions that would otherwise be handled by multiple database programs running for each user process. Background processes asynchronously perform I/O and monitor other Oracle Database processes to provide increased parallelism for better performance and reliability. Issues Involving Oracle Background Processes Oracle Database creates background processes automatically when a database instance starts. An instance can have many background processes, not all of which always exist in every database configuration. The following information helps to outline some of the known issues involving common Oracle background processes, and has been grouped by background process where possible.   ORA- errors related to Background Processes ORA-443 background process did not start           See Note 18442.1 - OERR: ORA 443 background process <name> did not start ORA-444 background process failed while starting           See Note 18443.1 - OERR: ORA 444 background process <name> failed while starting ORA-445 background process did not start after seconds           See Note 18444.1 - OERR: ORA 445 background process <name> did not start after <num> seconds ORA-446 background process started when not expected           See Note 18445.1 - OERR: ORA 446 background process started when not expected ORA-447 fatal error in background process           See Note 18446.1 - OERR: ORA 447 fatal error in background process ORA-448 normal completion of background process           See Note 18447.1 - OERR: ORA 448 normal completion of background process ORA-449 background process unexpectedly terminated with error           See Note 18448.1 - OERR: ORA 449 background process <name> unexpectedly terminated with error ORA-450 background process '%s' did not start           See Note 287652.1 - OERR: ORA-450 background process '%s' did not start   Generic issues not related to a specific background process Note 1379200.1 - Troubleshooting Guide (TSG) - ksvcreate: Process(xxxx) creation failed / ORA-00445: background process "xxxx" did not start after n seconds Note 1349496.1 - Background Process is Killed with ORA-00028 and Instance is Terminated Note 1345364.1 - ORA-00445: Background Process "xxxx" Did Not Start After 120 Seconds Note 954245.1 - BACKGROUND PROCESSES START VERY SLOWLY AND SOMETIMES ORA-445 IS RAISED Note 730287.1 - Database Crashes With ORA-00449 and ORA-00601 Note 375517.1 - Background Process Terminated, e.g. ORA-471, Accompanied By Alert Log Warning "Low open file descriptor limit"[ Note 605280.1 - Database Fails With Message User Terminating Instance Due To Error 472 Note 779552.1 - ORA-00494 Or ORA-600 [2103] During High Load After 10.2.0.4 Note 1029808.6 - BACKGROUND PROCESS DIES, NO TRACE FILE GENERATED Note 166409.1 - Successfully Connect to Database Even if Background Processes are Killed   CJQ0 / Jnnn ( Job Queue Coordinator Process / Job Queue Slave Processes ) CJQ0 is automatically started and stopped as needed by Oracle Scheduler. The JOB_QUEUE_PROCESSES initialization parameter specifies the maximum number of processes that can be created for the execution of jobs. CJQ0 starts only as many job queue processes as required by the number of jobs to run and available resources. Job slave processes are created or awakened by the job coordinator when it is time for a job to be executed. Job slaves gather all the metadata required to run the job from the data dictionary. The slave processes start a database session as the owner of the job, execute triggers, and then execute the job. After the job is complete, the slave processes commit and then execute appropriate triggers and close the session. The slave can repeat this operation in case additional jobs need to be run. Issues Specific to CJQ0 / Jnnn Note 1467179.1 - ORA-00450, ORA-00443, background process "CJQ0" did not start Note 1449639.1 - ORA-00450: background process 'CJQ0' did not start Note 979217.1 - Messages In Alert Log - Starting/Stopping Background Process CJQ0 Note 413399.1 - CJQ Process Starts Even When job_queue_processes=0 Note 854214.1 - CJQ Process Fails With ORA-7445 [kkjqawi+353] Note 1292151.1 - ORA-7445 [Kggibr] Crashes of CJQ Process Note 8285404.8 - Bug 8285404 - CJQ errors with ORA-1000 during window switches Note 9167430.8 - Bug 9167430 - CJQ job queue process SQL uses FULL TABLE SCAN of OBJ$ Note 344275.1 - Kkjcre1p: Unable To Spawn Jobq Slave Process, Error 1089 Note 564957.1 - DBMS_SCHEDULER NOT RUNNING THE JOBS, CJQ WAITING FOR 'ENQ: JS - WDW OP' Note 416244.1 - ORA-00610 And/Or "unable to spawn jobq slave process " And/Or "Process(<>) creation failed" In The Alert Log And/Or TNS-12518/ TNS-12500 In Listener Log Note 436509.1 - ORA-610 INTERNAL ERROR CODE, KKJCRE1P: UNABLE TO SPAWN JOBQ Note 756389.1 - TNS-12514,TNS-12523,TNS-12528,J000 died kkjcre1p: unable to spawn jobq slave process Note 790397.1 - PROCESS J000 And M000 Die   CKPT ( Checkpoint Process ) At specific times CKPT starts a checkpoint request by messaging DBWn to begin writing dirty buffers. On completion of individual checkpoint requests, CKPT updates data file headers and control files to record most recent checkpoint. Issues Specific to CKPT Note 1151601.1 - CKPT: terminating instance due to error 472 in Linux on System Z Note 7385253.8 - Bug 7385253 - Slow Truncate / DBWR uses high CPU / CKPT blocks on RO enqueue   DBWn ( Database Writer Process ) The primary responsibility of DBWn is to write data blocks to disk. DBWn also handles checkpoints, file open synchronization, and logging of Block Written records. In many cases the blocks that DBWn writes are scattered throughout the disk. Thus, the writes tend to be slower than the sequential writes performed by LGWR. DBWn performs multiblock writes when possible to improve efficiency. The number of blocks written in a multiblock write varies by operating system. The DB_WRITER_PROCESSES initialization parameter specifies the number of DBWn processes (DBW0-DBW9 and DBWa-DBWz). The database selects an appropriate default setting for this parameter or adjusts a user-specified setting based on the number of CPUs and processor groups. Issues Specific to DBWn Note 243248.1 - Database Startup fails with ORA-443 Background Process "DBW0" did not Start Note 1065267.6 - ORA-445 Background Process DBW0 Did Not Start After 120 Seconds Note 1461704.1 - ORA-00600 [kcbbxsv_2] Errors And DBW0: Terminating Instance Due To Error 471 ( Note 6005113.8 - Bug 6005113 - OERI[kclrwrite_7] in DBWn process followed by instance crash in RAC   EMNC / Ennn ( EMON Coordinator Process / EMON Slave Process ) EMNC coordinates event management and notification activity in the database, including Streams Event Notifications, Continuous Query Notifications, and Fast Application Notifications. The database event management and notification load is distributed among the EMON slave processes. These processes work on the system notifications in parallel, offering a capability to process a larger volume of notifications, a faster response time, and a lower shared memory use for staging notifications. Issues Specific to EMNC / Ennn Note 1302050.1 - Shut Down Immediate Ora-00443 Background Process Emno Didn't Start Note 329863.1 - The Event Monitor Process reports an ORA-00600 [15239] when using OCI AQ Notification Note 1076346.1 - EMN process crash with ORA-07445 [nserrbd()+129] Note 105067.1 - Event Monitor Process: Architecture and Known Issues   LGWR ( Log Writer Process ) Redo log entries are generated in the redo log buffer of the system global area (SGA). LGWR writes the redo log entries sequentially into a redo log file. If the database has a multiplexed redo log, then LGWR writes the redo log entries to a group of redo log files. Issues Specific to LGWR Note 470.1 - Error ORA-00470: LGWR process terminated with error Note 431246.1 - Pmon terminated instance due to LGWR termination ORA-470 Note 1089733.1 - ORA-00494: enqueue [CF] held for too long (more than 900 seconds); LGWR:Terminating Instance Due to Error 2103 Note 9556189.8 - Bug 9556189 - LGWR hangs for long periods using DNFS - CF waits likely Note 6193945.8 - Bug 6193945 - High LGWR CPU use and long 'log file sync' latency in RAC Note 6074620.8 - Bug 6074620 - LGWR unconditionally writes to trace file Note 5896963.8 - Bug 5896963 - High LGWR CPU and longer "log file sync" with fix for bug 5065930 Note 5118272.8 - Bug 5118272 - Instance crash with ORA-600 [1433], [60] due to lgwr filling up msg blocks (Messages could be for LGWR or ARCH) Note 813473.1 - LGWR Uses Async I/O Inspite Of Setting _lgwr_async_io=False Note 1278149.1 - Intermittent Long 'log file sync' Waits, LGWR Posting Long Write Times, I/O Portion of Wait Minimal   MMON / Mnnn ( Manageability Monitor Process / MMON Slave Process ) MMON performs many tasks related to manageability, including taking Automatic Workload Repository snapshots and performing Automatic Database Diagnostic Monitor analysis. Mnnn performs manageability tasks dispatched to them by MMON. Tasks performed include taking Automatic Workload Repository snapshots and Automatic Database Diagnostic Monitor analysis. Issues Specific to MMON / Mnnn Note 1233079.1 - Ksvcreate: Process(M001) Creation Failed , Database hang Note 846217.1 - Getting Ora-00600 [12826] In MMON Background Slave (m000) Process Note 359950.1 - Trace Files Reporting ORA-1476 For m000 Background Process Note 1174758.1 - M000 tracefiles containing: ket_aba_validate : Error 15649 After 11.1 Upgrade Note 555124.1 - High Cpu Usage By M000 Process And Kewrafc Code 5 In Mmon Trace File Note 13833511.8 - Bug 13833511 - M000 process spins after a tablespace is renamed and new tablespace is created with original name Note 13358531.8 - Bug 13358531 - MMON slave (M000) may hold "TH" enqueue too long   PMON ( Process Monitor ) PMON periodically performs cleanup of all the following: Processes that died abnormally Sessions that were killed Detached transactions that have exceeded their idle timeout Detached network connections which have exceeded their idle timeout In addition, PMON monitors, spawns, and stops the following as needed: Dispatcher and shared server processes Job queue processes Pooled server processes for database resident connection pooling Restartable background processes PMON is also responsible for registering information about the instance and dispatcher processes with the network listener. Issues Specific to PMON Note 465770.1 - Instance Terminated By PMON Due To Error ORA-00443 Note 789919.1 - Instance crash by PMON due to ORA-00484 ; Dump shows 'failed with errno=4' Note 744619.1 - PMON crashing with: ORA-07445: exception encountered: CORE Dump [kfiofib_detach()+84] Note 1130713.1 - Pmon Spins While Cleaning Dead Process Note 468740.1 - "Pmon Failed To Acquire Latch" Messages in Alert Log -Database Hung Note 976714.1 - 'PMON failed to acquire latch, see PMON dump' in Alert Log - How To Diagnose Note 1138755.1 - PMON Processes Utilizing Excessive CPU [100%] in 10gR2   QMNC / QMNn ( AQ Coordinator Process / AQ Server Class Process ) QMNC is responsible for facilitating various background activities required by AQ and Oracle Streams: time management of messages, management of nonpersistent queues, cleanup of resources, and so on. QMNC dynamically spawns Qnnn processes as needed for performing these tasks. Note that if the AQ_TM_PROCESSES initialization parameter is set to 0, this process will not start. The database writes the following message to the alert log: WARNING: AQ_TM_PROCESSES is set to 0. System might be adversely affected. Qnnn acts as a slave process for QMNC and carry out tasks assigned by QMNC. The number of these processes is dynamically managed by QMNC based on load. Issues Specific to QMNC / QMNn Note 793632.1 - Restarting Dead Queue Monitor Process upgrade from 9.2 to 10.2 Note 839527.1 - Continuous Warning 'Restarting Dead Background Process Qmn0' in Alert.log Note 395137.1 - Repeated : Restarting dead background process QMNC recorded in the alert.log file   RECO ( Recoverer Process ) RECO uses the information in the pending transaction table to finalize the status of in-doubt transactions. At timed intervals, the local RECO attempts to connect to remote databases and automatically complete the commit or rollback of the local portion of any pending distributed transactions. All transactions automatically resolved by RECO are removed from the pending transaction table. Issues Specific to RECO Note 761852.1 - RECO Process Fails with ORA-476 or ORA-444 Note 1447285.1 - RECO Process Crashes Instance After ORA-2068 ORA-7445 [kksfft()+109] ORA-476   SMON ( System Monitor Process ) SMON performs many database maintenance tasks, including the following: Creates and manages the temporary tablespace metadata Reclaims space used by orphaned temporary segments Maintains the undo tablespace by onlining, offlining, and shrinking the undo segments based on undo space usage statistics Cleans up the data dictionary when it is in a transient and inconsistent state Maintains the SCN to time mapping table used to support Oracle Flashback features Issues Specific to SMON Note 155413.1 - SMON Background Process is Running for Long Time Note 761094.1 - SMON: Failed to acquire SEG2, skipping transaction Note 1073704.1 - SMON Consumes High CPU With Excessive Redo Log Generation Note 821743.1 - SMON Trace File Reports Messages "SMON: about to recover undo segment X" "SMON: mark undo segment X as available" Note 7529174.8 - Bug 7529174 - Deadlock / hang between SMON and foreground process Note 1076161.6 - Shutdown Normal or Shutdown Immediate Hangs. SMON disabling TX Recovery Note 414242.1 - Database Hangs Because SMON Is Taking 100% CPU Doing Transaction Recovery Note 549000.1 - ORA-600 [6006] ORA-600 [6856] During Startup Instance, Followed by Termination by SMON   VKTM ( Virtual Keeper of Time Process ) VKTM acts as a time publisher for an Oracle instance. VKTM publishes two sets of time: a wall clock time using a seconds interval and a higher resolution time (which is not wall clock time) for interval measurements. The VKTM timer service centralizes time tracking and offloads multiple timer calls from other clients. Issues Specific to VKTM Note 558374.1 - Database Startup Fails With ORA-00443: Background Process "VKTM" Did Not Start Note 837105.1 - The Database Server Is Creating A Lot Of Trace Files From the VKTM Process Note 1347586.1 - Time Drift Detected. Please Check Vktm Trace File For More Details Note 1185093.1 - VKTM Trace Files Generated With KSTMCHKDRIFT "BACKWARD DRIFT ENDED AT"   Wnnn ( Space Management Slave Process ) Wnnn processes are slave processes dynamically spawned by SMCO to perform space management tasks in the background. These tasks include preallocating space into locally managed tablespace and SecureFiles segments based on space usage growth analysis, and reclaiming space from dropped segments. At most 10 Wnnn slaves can run on one database instance. After being started, the slave acts as an autonomous agent. After it finishes task execution, it automatically picks up another task from the queue. The process terminates itself after being idle for a long time. Issues Specific to Wnnn Note 1492880.1 - Wnnn processes consuming high CPU Note 13871316.8 - Bug 13871316 - Wnnn space background process may be periodically CPU bound Note 7661251.8 - Bug 7661251 - A process may spin causing ORA-445 background process "wnnn" did not start .." with fix for bug 6858062 Note 10237773.8 - Bug 10237773 - ORA-600 [kcbz_check_objd_typ] / ORA-600 [ktecgsc:kcbz_objdchk]   Helpful Articles on Background Processes Note 91062.1 - Database Writer and Buffer Management Note 305662.1 - Master Note for AQ Queue Monitor Process (QMON) Note 743773.1 - SMCO (Space Management Coordinator) For Autoextend On Datafiles And How To Disable/Enable Note 268197.1 - New Background Processes In 10g Note 444149.1 - New Background Processes In 11g Note 331000.1 - What are the 'PSP0' and 'LNS1' Background Processes Running In 10g Release 2? Note 1038976.6 - HOW TO FIND BACKGROUND PROCESS RUNNING ON NT Note 564989.1 - How To Truncate a Background Trace File Without Bouncing the Database Note 164768.1 - Troubleshooting: High CPU Utilization Note 250262.1 - RDA 4 - Health Check / Validation Engine Guide   FURTHER DIAGNOSTICS If you were not able to resolve the issue with the details provided in this document, please raise a Service Request for further assistance from Oracle Support. REFERENCES NOTE:1503146.1 - Master Note: Overview of Oracle Background Processes NOTE:250262.1 - RDA - Health Check / Validation Engine Guide     Related         Products       Oracle Database Products > Oracle Database > Oracle Database > Oracle Database - Enterprise Edition > RDBMS > Background processes   Keywords       BACKGROUND PROCESSES Back to Top