宕机 重启oracle数据库,RAC一节点宕机重启问题

本帖最后由 zyo320 于 2012-11-16 12:44 编辑

RAC双节点，数据库版本：10.2.0.1 系统 Linux  2.6.18-128.7NFS #1 SMP

节点2：

alert信息：

Shutting down instance (abort)

License high water mark = 193

Instance terminated by USER, pid = 31102

Fri Nov 16 04:23:07 2012

Starting ORACLE instance (normal)

LICENSE_MAX_SESSION = 0

LICENSE_SESSIONS_WARNING = 0

Interface type 1 eth1 192.168.100.0 configured from OCR for use as a cluster interconnect

Interface type 1 eth0 10.0.0.0 configured from OCR for use as  a public interface

Picked latch-free SCN scheme 1

Autotune of undo retention is turned on.

LICENSE_MAX_USERS = 0

SYS auditing is disabled

ksdpec: called for event 13740 prior to event group initialization

Starting up ORACLE RDBMS Version: 10.2.0.1.0.

System parameters with non-default values:

processes                = 1000

sessions                 = 1105

__shared_pool_size       = 603979776

__large_pool_size        = 16777216

__java_pool_size         = 16777216

__streams_pool_size      = 0

spfile                   = +FOODSAFE_DISK/safedb/spfilesafedb.ora

sga_target               = 5016387584

control_files            = +FOODSAFE_DISK/safedb/controlfile/current.256.759414295

db_block_size            = 8192

__db_cache_size          = 4362076160

db_writer_processes      = 3

compatible               = 10.2.0.1.0

log_archive_dest_1       = location=/dbrac2_arch

log_archive_dest_2       =

log_archive_max_processes= 4

standby_archive_dest     =

log_checkpoint_interval  = 0

log_checkpoint_timeout   = 0

db_file_multiblock_read_count= 16

cluster_database         = TRUE

cluster_database_instances= 2

db_create_file_dest      = +FOODSAFE_DISK

thread                   = 2

fast_start_mttr_target   = 300

instance_number          = 2

undo_management          = AUTO

undo_tablespace          = UNDOTBS2

_kgl_large_heap_warning_threshold= 52428800

remote_login_passwordfile= EXCLUSIVE

db_domain                =

dispatchers              = (PROTOCOL=TCP) (SERVICE=SAFEDBXDB)

local_listener           = (ADDRESS=(PROTOCOL=TCP)(HOST=10.0.0.123)(PORT=1521))

remote_listener          = LISTENERS_SAFEDB

job_queue_processes      = 10

background_dump_dest     = /opt/ora10g/admin/SAFEDB/bdump

user_dump_dest           = /opt/ora10g/admin/SAFEDB/udump

core_dump_dest           = /opt/ora10g/admin/SAFEDB/cdump

audit_file_dest          = /opt/ora10g/admin/SAFEDB/adump

db_name                  = SAFEDB

open_cursors             = 300

pga_aggregate_target     = 1669332992

Cluster communication is configured to use the following interface(s) for this instance

192.168.100.101

Fri Nov 16 04:23:08 2012

cluster interconnect IPC version:Oracle UDP/IP

IPC Vendor 1 proto 2

PMON started with pid=2, OS id=31270

DIAG started with pid=3, OS id=31272

PSP0 started with pid=4, OS id=31274

LMON started with pid=5, OS id=31276

LMD0 started with pid=6, OS id=31278

LMS0 started with pid=7, OS id=31280

LMS1 started with pid=8, OS id=31284

LMS2 started with pid=9, OS id=31288

LMS3 started with pid=10, OS id=31292

MMAN started with pid=11, OS id=31296

DBW0 started with pid=12, OS id=31298

DBW1 started with pid=13, OS id=31300

DBW2 started with pid=14, OS id=31302

LGWR started with pid=15, OS id=31304

CKPT started with pid=16, OS id=31306

SMON started with pid=17, OS id=31308

RECO started with pid=18, OS id=31335

CJQ0 started with pid=19, OS id=31337

MMON started with pid=20, OS id=31339

Fri Nov 16 04:23:09 2012

starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...

MMNL started with pid=21, OS id=31341

Fri Nov 16 04:23:09 2012

starting up 1 shared server(s) ...

Fri Nov 16 04:23:09 2012

lmon registered with NM - instance id 2 (internal mem no 1)

Fri Nov 16 04:23:10 2012

Reconfiguration started (old inc 0, new inc 48)

pseudo shared rm latch used

List of nodes:

0 1

Global Resource Directory frozen

* allocate domain 0, invalid = TRUE

Communication channels reestablished

* domain 0 valid = 0 according to instance 0

Fri Nov 16 04:23:11 2012

Master broadcasted resource hash value bitmaps

Non-local Process blocks cleaned out

Fri Nov 16 04:23:11 2012

LMS 1: 0 GCS shadows cancelled, 0 closed

Fri Nov 16 04:23:11 2012

LMS 2: 0 GCS shadows cancelled, 0 closed

Fri Nov 16 04:23:11 2012

LMS 3: 0 GCS shadows cancelled, 0 closed

Fri Nov 16 04:23:11 2012

LMS 0: 0 GCS shadows cancelled, 0 closed

Set master node info

Submitted all remote-enqueue requests

Dwn-cvts replayed, VALBLKs dubious

All grantable enqueues granted

Fri Nov 16 04:23:12 2012

LMS 2: 0 GCS shadows traversed, 0 replayed

Fri Nov 16 04:23:12 2012

LMS 0: 0 GCS shadows traversed, 0 replayed

Fri Nov 16 04:23:12 2012

LMS 3: 0 GCS shadows traversed, 0 replayed

Fri Nov 16 04:23:12 2012

LMS 1: 0 GCS shadows traversed, 0 replayed

Fri Nov 16 04:23:12 2012

Submitted all GCS remote-cache requests

Fix write in gcs resources

Reconfiguration complete

LCK0 started with pid=24, OS id=31422

Fri Nov 16 04:23:13 2012

ALTER DATABASE   MOUNT

Fri Nov 16 04:23:13 2012

Starting background process ASMB

ASMB started with pid=26, OS id=31427

Starting background process RBAL

RBAL started with pid=27, OS id=31441

Fri Nov 16 04:23:23 2012

SUCCESS: diskgroup FOODSAFE_DISK was mounted

Fri Nov 16 04:23:27 2012

Setting recovery target incarnation to 2

Fri Nov 16 04:23:27 2012

Successful mount of redo thread 2, with mount id 2685604006

Fri Nov 16 04:23:27 2012

Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)

Completed: ALTER DATABASE   MOUNT

Fri Nov 16 04:23:27 2012

ALTER DATABASE OPEN

Picked broadcast on commit scheme to generate SCNs

Fri Nov 16 04:23:28 2012

LGWR: STARTING ARCH PROCESSES

ARC0 started with pid=29, OS id=31802

ARC1 started with pid=30, OS id=31804

ARC2 started with pid=31, OS id=31806

Fri Nov 16 04:23:28 2012

ARC0: Archival started

ARC1: Archival started

ARC2: Archival started

ARC3: Archival started

LGWR: STARTING ARCH PROCESSES COMPLETE

ARC3 started with pid=32, OS id=31808

Fri Nov 16 04:23:28 2012

Thread 2 opened at log sequence 1956

Current log# 8 seq# 1956 mem# 0: +FOODSAFE_DISK/safedb/onlinelog/group_8.290.798134725

Successful open of redo thread 2

Fri Nov 16 04:23:28 2012

ARC2: STARTING ARCH PROCESSES

Fri Nov 16 04:23:28 2012

ARC1: Becoming the 'no FAL' ARCH

ARC1: Becoming the 'no SRL' ARCH

Fri Nov 16 04:23:28 2012

ARC0: Becoming the heartbeat ARCH

Fri Nov 16 04:23:28 2012

SMON: enabling cache recovery

Fri Nov 16 04:23:28 2012

ARC4: Archival started

ARC2: STARTING ARCH PROCESSES COMPLETE

ARC4 started with pid=33, OS id=31810

Fri Nov 16 04:23:29 2012

Successfully onlined Undo Tablespace 4.

Fri Nov 16 04:23:29 2012

SMON: enabling tx recovery

Fri Nov 16 04:23:29 2012

Database Characterset is ZHS16GBK

replication_dependency_tracking turned off (no async multimaster replication found)

Starting background process QMNC

QMNC started with pid=38, OS id=31871

Fri Nov 16 04:23:31 2012

Completed: ALTER DATABASE OPEN

Fri Nov 16 04:23:40 2012

ALTER SYSTEM SET service_names='SAFEDB','foodsafe' SCOPE=MEMORY SID='SAFEDB2';

Fri Nov 16 04:29:28 2012

Shutting down archive processes

Fri Nov 16 04:29:33 2012

ARCH shutting down

ARC4: Archival stopped

节点1：alert信息

Fri Nov 16 04:23:08 2012

Reconfiguration started (old inc 44, new inc 46)

List of nodes:

0

Global Resource Directory frozen

* dead instance detected - domain 0 invalid = TRUE

Communication channels reestablished

Master broadcasted resource hash value bitmaps

Non-local Process blocks cleaned out

Fri Nov 16 04:23:08 2012

LMS 2: 0 GCS shadows cancelled, 0 closed

Fri Nov 16 04:23:08 2012

LMS 1: 0 GCS shadows cancelled, 0 closed

Fri Nov 16 04:23:08 2012

LMS 3: 0 GCS shadows cancelled, 0 closed

Fri Nov 16 04:23:08 2012

LMS 0: 0 GCS shadows cancelled, 0 closed

Set master node info

Submitted all remote-enqueue requests

Dwn-cvts replayed, VALBLKs dubious

All grantable enqueues granted

Post SMON to start 1st pass IR

Fri Nov 16 04:23:08 2012

LMS 3: 90512 GCS shadows traversed, 0 replayed

Fri Nov 16 04:23:08 2012

LMS 2: 90141 GCS shadows traversed, 0 replayed

Fri Nov 16 04:23:08 2012

LMS 1: 90903 GCS shadows traversed, 0 replayed

Fri Nov 16 04:23:08 2012

LMS 0: 93644 GCS shadows traversed, 0 replayed

Fri Nov 16 04:23:08 2012

Submitted all GCS remote-cache requests

Post SMON to start 1st pass IR

Fix write in gcs resources

Fri Nov 16 04:23:08 2012

Instance recovery: looking for dead threads

Fri Nov 16 04:23:08 2012

Beginning instance recovery of 1 threads

Reconfiguration complete

Fri Nov 16 04:23:10 2012

Reconfiguration started (old inc 46, new inc 48)

List of nodes:

0 1

Global Resource Directory frozen

Communication channels reestablished

Master broadcasted resource hash value bitmaps

Non-local Process blocks cleaned out

Fri Nov 16 04:23:11 2012

LMS 1: 0 GCS shadows cancelled, 0 closed

Fri Nov 16 04:23:11 2012

LMS 0: 0 GCS shadows cancelled, 0 closed

Fri Nov 16 04:23:11 2012

LMS 3: 0 GCS shadows cancelled, 0 closed

Fri Nov 16 04:23:11 2012

LMS 2: 0 GCS shadows cancelled, 0 closed

Set master node info

Submitted all remote-enqueue requests

Dwn-cvts replayed, VALBLKs dubious

All grantable enqueues granted

Fri Nov 16 04:23:12 2012

LMS 0: 8788 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 3: 8973 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 2: 8844 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 1: 8839 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 0: 8782 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 3: 8965 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 1: 8739 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 2: 9006 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 0: 8892 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 3: 8968 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 1: 8891 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 2: 8887 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 0: 8766 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 3: 9032 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 1: 8866 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 2: 8933 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 0: 8852 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 3: 8895 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 2: 8895 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 1: 8798 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 3: 9082 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 2: 8927 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 0: 8806 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 1: 8801 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 2: 9056 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 0: 8922 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 3: 8939 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 1: 8854 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 2: 8936 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 0: 8765 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 1: 8848 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 3: 9039 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 0: 8872 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 2: 8951 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 1: 8768 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 3: 8957 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 0: 8793 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 2: 8935 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 1: 8773 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 3: 9111 GCS shadows traversed, 4001 replayed

Fri Nov 16 04:23:12 2012

LMS 2: 768 GCS shadows traversed, 334 replayed

Fri Nov 16 04:23:12 2012

LMS 0: 5407 GCS shadows traversed, 2454 replayed

Fri Nov 16 04:23:12 2012

LMS 1: 2724 GCS shadows traversed, 1235 replayed

Fri Nov 16 04:23:12 2012

LMS 3: 549 GCS shadows traversed, 247 replayed

Fri Nov 16 04:23:12 2012

Submitted all GCS remote-cache requests

Post SMON to start 1st pass IR

Fix write in gcs resources

Reconfiguration complete

Fri Nov 16 04:23:13 2012

parallel recovery started with 15 processes

Fri Nov 16 04:23:13 2012

Started redo scan

Fri Nov 16 04:23:13 2012

Completed redo scan

212 redo blocks read, 56 data blocks need recovery

Fri Nov 16 04:23:13 2012

Started redo application at

Thread 2: logseq 1954, block 119345

Fri Nov 16 04:23:13 2012

Recovery of Online Redo Log: Thread 2 Group 7 Seq 1954 Reading mem 0

Mem# 0 errs 0: +FOODSAFE_DISK/safedb/onlinelog/group_7.291.798134677

Fri Nov 16 04:23:13 2012

Completed redo application

Fri Nov 16 04:23:13 2012

Completed instance recovery at

Thread 2: logseq 1954, block 119557, scn 1438521085

53 data blocks read, 57 data blocks written, 212 redo blocks read

Switch log for thread 2 to sequence 1955

Fri Nov 16 04:23:13 2012

ARC3: Archiving disabled thread 2 sequence 1955

Fri Nov 16 04:23:18 2012

2012-11-16 12:44 上传

点击文件名下载附件

186.41 KB, 下载次数: 13