mysql脑裂_记载一个脑裂现象之后，从库报错！

本帖最后由 mchdba 于 2012-9-10 11:10 编辑

11:30:04 db33 snmpd[17556]: Connection from UDP: [xxxx]:39555

11:30:04 db33 snmpd[17556]: Connection from UDP: [xxxx]:39555

11:32:17 db33 kernel: block drbd0: PingAck did not arrive in time.

11:32:17 db33 kernel: block drbd0: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )

11:32:17 db33 kernel: block drbd0: asender terminated

11:32:17 db33 kernel: block drbd0: Terminating asender thread

11:32:17 db33 kernel: block drbd0: short read expecting header on sock: r=-512

11:32:17 db33 kernel: block drbd0: Creating new current UUID

11:32:17 db33 kernel: block drbd0: Connection closed

11:32:17 db33 kernel: block drbd0: conn( NetworkFailure -> Unconnected )

11:32:17 db33 kernel: block drbd0: receiver (re)started

11:32:17 db33 kernel: block drbd0: conn( Unconnected -> WFConnection )

11:32:18 db33 openais[30882]: [TOTEM] The token was lost in the OPERATIONAL state.

11:32:18 db33 openais[30882]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes).

11:32:18 db33 openais[30882]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).

11:32:18 db33 openais[30882]: [TOTEM] entering GATHER state from 2.

11:32:38 db33 openais[30882]: [TOTEM] entering GATHER state from 0.

11:32:38 db33 openais[30882]: [TOTEM] Creating commit token because I am the rep.

11:32:38 db33 openais[30882]: [TOTEM] Saving state aru 46 high seq received 46

11:32:38 db33 openais[30882]: [TOTEM] Storing new sequence id for ring 160

11:32:38 db33 openais[30882]: [TOTEM] entering COMMIT state.

11:32:38 db33 openais[30882]: [TOTEM] entering RECOVERY state.

11:32:38 db33 openais[30882]: [TOTEM] position [0] member xxxx:

11:32:38 db33 openais[30882]: [TOTEM] previous ring seq 348 rep xxxx

11:32:38 db33 openais[30882]: [TOTEM] aru 46 high delivered 46 received flag 1

11:32:38 db33 openais[30882]: [TOTEM] Did not need to originate any messages in recovery.

11:32:38 db33 openais[30882]: [TOTEM] Sending initial ORF token

11:32:38 db33 openais[30882]: [CLM  ] CLM CONFIGURATION CHANGE

11:32:38 db33 fenced[30901]: xx.com not a cluster member after 0 sec post_fail_delay

11:32:38 db33 kernel: dlm: closing connection to node 2

11:32:38 db33 openais[30882]: [CLM  ] New Configuration:

11:32:38 db33 fenced[30901]: fencing node "xx.com"

11:32:38 db33 openais[30882]: [CLM  ]         r(0) ip(xxxx)

11:32:38 db33 openais[30882]: [CLM  ] Members Left:

11:32:38 db33 openais[30882]: [CLM  ]         r(0) ip(xxxx)

11:32:38 db33 openais[30882]: [CLM  ] Members Joined:

11:32:38 db33 openais[30882]: [CLM  ] CLM CONFIGURATION CHANGE

11:32:38 db33 openais[30882]: [CLM  ] New Configuration:

11:32:38 db33 openais[30882]: [CLM  ]         r(0) ip(xxxx)

11:32:38 db33 openais[30882]: [CLM  ] Members Left:

11:32:38 db33 openais[30882]: [CLM  ] Members Joined:

11:32:38 db33 openais[30882]: [SYNC ] This node is within the primary component and will provide service.

11:32:38 db33 openais[30882]: [TOTEM] entering OPERATIONAL state.

11:32:38 db33 openais[30882]: [CLM  ] got nodejoin message xxxx

11:32:38 db33 openais[30882]: [CPG  ] got joinlist message from node 1

11:32:39 db33 shutdown[32750]: shutting down for system halt

11:32:39 db33 pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found

11:32:42 db33 modclusterd: shutdown succeeded

11:32:42 db33 rgmanager: [569]: Shutting down Cluster Service Manager...

11:32:42 db33 clurgmgrd[30936]: Shutting down

11:37:08 db33 syslogd 1.4.1: restart.

11:37:08 db33 kernel: klogd 1.4.1, log source = /proc/kmsg started.

11:37:08 db33 kernel: Linux version 2.6.18-194.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Fri Apr

2 14:58:14 EDT 2010

11:37:08 db33 kernel: Command line: ro root=/dev/VolGroup00/LogVol00 rhgb quiet

11:37:08 db33 kernel: BIOS-provided physical RAM map:

11:37:08 db33 kernel:  BIOS-e820: 0000000000010000 - 0000000000095400 (usable)

11:37:08 db33 kernel:  BIOS-e820: 0000000000095400 - 00000000000a0000 (reserved)

11:37:08 db33 kernel:  BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)

11:37:08 db33 kernel:  BIOS-e820: 0000000000100000 - 000000007f600000 (usable)

11:37:08 db33 kernel:  BIOS-e820: 000000007f600000 - 000000007f62c000 (ACPI data)

11:37:08 db33 kernel:  BIOS-e820: 000000007f62c000 - 000000007f62d000 (usable)

11:37:08 db33 kernel:  BIOS-e820: 000000007f62d000 - 000000007f79e000 (reserved)

11:37:08 db33 kernel:  BIOS-e820: 000000007f79e000 - 000000007f7a0000 (ACPI data)

11:37:08 db33 kernel:  BIOS-e820: 000000007f7a0000 - 0000000090000000 (reserved)

11:37:08 db33 kernel:  BIOS-e820: 00000000fec00000 - 00000000fee10000 (reserved)

11:37:08 db33 kernel:  BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved)

11:37:09 db33 kernel:  BIOS-e820: 0000000100000000 - 0000000480000000 (usable)

11:37:09 db33 kernel:  BIOS-e820: 0000002000000000 - 0000002400000000 (usable)

11:37:09 db33 kernel:  BIOS-e820: 0000004000000000 - 0000004400000000 (usable)

11:37:09 db33 kernel:  BIOS-e820: 0000006000000000 - 0000006400000000 (usable)

11:37:09 db33 kernel:  BIOS-e820: 0000008000000000 - 0000008400000000 (usable)

11:37:09 db33 kernel:  BIOS-e820: 000000a000000000 - 000000a400000000 (usable)

11:37:09 db33 kernel:  BIOS-e820: 000000c000000000 - 000000c400000000 (usable)

11:37:09 db33 kernel:  BIOS-e820: 000000e000000000 - 000000e400000000 (usable)

11:37:09 db33 kernel: DMI 2.6 present.

11:37:09 db33 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0

11:37:09 db33 kernel: SRAT: PXM 0 -> APIC 1 -> Node 0

11:37:09 db33 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0

11:37:09 db33 kernel: SRAT: PXM 0 -> APIC 3 -> Node 0

11:37:09 db33 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0

11:37:09 db33 kernel: SRAT: PXM 0 -> APIC 5 -> Node 0

11:37:09 db33 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0

11:37:09 db33 kernel: SRAT: PXM 0 -> APIC 7 -> Node 0

............................. 很多SRAT信息，忽略过掉

11:37:09 db33 kernel: SRAT: PXM 7 -> APIC 246 -> Node 7

11:37:09 db33 kernel: SRAT: PXM 7 -> APIC 247 -> Node 7

11:37:09 db33 kernel: ACPI: [SRAT:0x00] ignored 1 entries of 256 found

11:37:09 db33 kernel: SRAT: Node 0 PXM 0 0-80000000

11:37:09 db33 kernel: SRAT: Node 0 PXM 0 0-480000000

11:37:09 db33 kernel: SRAT: Node 1 PXM 1 2000000000-2400000000

11:37:09 db33 kernel: SRAT: Node 2 PXM 2 4000000000-4400000000

11:37:09 db33 kernel: SRAT: Node 3 PXM 3 6000000000-6400000000

11:37:09 db33 kernel: SRAT: Node 4 PXM 4 8000000000-8400000000

11:37:09 db33 kernel: SRAT: Node 5 PXM 5 a000000000-a400000000

11:37:09 db33 kernel: SRAT: Node 6 PXM 6 c000000000-c400000000

11:37:09 db33 kernel: SRAT: Node 7 PXM 7 e000000000-e400000000

11:37:09 db33 kernel: Bootmem setup node 0 0000000000000000-0000000480000000

11:37:09 db33 kernel: Bootmem setup node 1 0000002000000000-0000002400000000

11:37:09 db33 kernel: Bootmem setup node 2 0000004000000000-0000004400000000

11:37:09 db33 kernel: Bootmem setup node 3 0000006000000000-0000006400000000

11:37:09 db33 kernel: Bootmem setup node 4 0000008000000000-0000008400000000

11:37:09 db33 kernel: Bootmem setup node 5 000000a000000000-000000a400000000

11:37:09 db33 kernel: Bootmem setup node 6 000000c000000000-000000c400000000

11:37:09 db33 kernel: Bootmem setup node 7 000000e000000000-000000e400000000

11:37:09 db33 kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissible range

11:37:09 db33 kernel: disabling kdump

11:37:09 db33 kernel: ACPI: PM-Timer IO Port: 0x908

11:37:09 db33 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)

11:37:09 db33 kernel: Processor #0 6:14 APIC version 21

11:37:09 db33 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)

11:37:09 db33 kernel: Processor #2 6:14 APIC version 21

11:37:09 db33 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)

11:37:09 db33 kernel: Processor #4 6:14 APIC version 21

11:37:09 db33 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)

11:37:09 db33 kernel: Processor #6 6:14 APIC version 21

11:37:09 db33 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x10] enabled)

11:37:09 db33 kernel: Processor #16 6:14 APIC version 21

11:37:09 db33 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0x41] enabled)

11:37:09 db33 kernel: Processor #65 6:14 APIC version 21

11:37:09 db33 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0x43] enabled)

11:37:09 db33 kernel: Processor #67 6:14 APIC version 21

11:37:09 db33 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0x45] enabled)

11:37:09 db33 kernel: Processor #69 6:14 APIC version 21

11:37:09 db33 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0x47] enabled)

11:37:09 db33 kernel: Processor #71 6:14 APIC version 21

11:37:09 db33 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0x51] enabled)

11:37:09 db33 kernel: Processor #81 6:14 APIC version 21

11:37:09 db33 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0x53] enabled)

11:37:09 db33 kernel: Processor #83 6:14 APIC version 21

11:37:09 db33 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0x55] enabled)

11:37:09 db33 kernel: Processor #85 6:14 APIC version 21

11:37:09 db33 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0x57] enabled)

11:37:09 db33 kernel: Processor #87 6:14 APIC version 21

11:37:09 db33 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0x61] enabled)

11:37:09 db33 kernel: Processor #97 6:14 APIC version 21

11:37:09 db33 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0x63] enabled)

11:37:09 db33 kernel: Processor #99 6:14 APIC version 21

11:37:09 db33 kernel: Processor #245 6:14 APIC version 21

11:37:09 db33 kernel: ACPI: LAPIC (acpi_id[0x7f] lapic_id[0xf7] enabled)

11:37:09 db33 kernel: Processor #247 6:14 APIC version 21

11:37:09 db33 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])

11:37:09 db33 kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])

11:37:09 db33 kernel: IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23

11:37:09 db33 kernel: ACPI: IOAPIC (id[0x00] address[0xfec08000] gsi_base[24])

11:37:09 db33 kernel: IOAPIC[1]: apic_id 0, version 32, address 0xfec08000, GSI 24-47

11:37:09 db33 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)

11:37:09 db33 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)

11:37:09 db33 kernel: Setting APIC routing to clustered

11:37:09 db33 kernel: ACPI: HPET id: 0x8086a201 base: 0xfed00000

11:37:09 db33 kernel: Using ACPI (MADT) for SMP configuration information

11:37:09 db33 kernel: Nosave address range: 0000000000095000 - 0000000000096000

11:37:09 db33 kernel: Nosave address range: 0000000000096000 - 00000000000a0000

11:37:09 db33 kernel: Nosave address range: 00000000000a0000 - 00000000000f0000

11:37:09 db33 kernel: Nosave address range: 00000000000f0000 - 0000000000100000

11:37:09 db33 kernel: Nosave address range: 000000007f600000 - 000000007f62c000

11:37:09 db33 kernel: Nosave address range: 000000007f62d000 - 000000007f79e000

11:37:09 db33 kernel: Nosave address range: 000000007f79e000 - 000000007f7a0000

11:37:09 db33 kernel: Nosave address range: 000000007f7a0000 - 0000000090000000

11:37:09 db33 kernel: Nosave address range: 0000000090000000 - 00000000fec00000

11:37:09 db33 kernel: Nosave address range: 00000000fec00000 - 00000000fee10000

11:37:09 db33 kernel: Nosave address range: 00000000fee10000 - 00000000ff800000

11:37:09 db33 kernel: Nosave address range: 00000000ff800000 - 0000000100000000

11:37:09 db33 kernel: Nosave address range: 0000000480000000 - 0000002000000000

11:37:09 db33 kernel: Nosave address range: 0000002400000000 - 0000004000000000

11:37:09 db33 kernel: Nosave address range: 0000004400000000 - 0000006000000000

11:37:09 db33 kernel: Nosave address range: 0000006400000000 - 0000008000000000

11:37:09 db33 kernel: Nosave address range: 0000008400000000 - 000000a000000000

11:37:09 db33 kernel: Nosave address range: 000000a400000000 - 000000c000000000

11:37:09 db33 kernel: Nosave address range: 000000c400000000 - 000000e000000000

11:37:09 db33 kernel: Allocating PCI resources starting at 98000000 (gap: 90000000:6ec00000)

11:37:09 db33 kernel: SMP: Allowing 128 CPUs, 0 hotplug CPUs

11:37:09 db33 kernel: Built 8 zonelists.  Total pages: 33084402

11:37:09 db33 kernel: Kernel command line: ro root=/dev/VolGroup00/LogVol00 rhgb quiet

11:37:09 db33 kernel: Initializing CPU#0

11:37:09 db33 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes)

11:37:09 db33 kernel: Console: colour VGA+ 80x25

11:37:09 db33 kernel: Dentry cache hash table entries: 16777216 (order: 15, 134217728 bytes)

11:37:09 db33 kernel: Inode-cache hash table entries: 8388608 (order: 14, 67108864 bytes)

11:37:09 db33 kernel: Checking aperture...

11:37:09 db33 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)

11:37:09 db33 kernel: Placing software IO TLB between 0xd0a1000 - 0x110a1000

11:37:09 db33 kernel: Memory: 132066100k/956301312k available (2577k kernel code, 2140900k reserved, 1305k data, 212k init)

11:37:09 db33 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3990.18 BogoMIPS (lpj=1995094)

11:37:09 db33 kernel: Security Framework v1.0.0 initialized

11:37:09 db33 kernel: SELinux:  Initializing.

11:37:09 db33 kernel: selinux_register_security:  Registering secondary module capability

11:37:09 db33 kernel: Capability LSM initialized as secondary

11:37:09 db33 kernel: Mount-cache hash table entries: 256

11:37:09 db33 kernel: CPU: L3 cache: 18432K

11:37:09 db33 kernel: CPU 0/0 -> Node 0

11:37:09 db33 kernel: using mwait in idle threads.

11:37:09 db33 kernel: CPU: Physical Processor ID: 0

11:37:09 db33 kernel: CPU: Processor Core ID: 0

11:37:09 db33 kernel: CPU0: Thermal monitoring enabled (TM1)

11:37:09 db33 kernel: SMP alternatives: switching to UP code

11:37:09 db33 kernel: ACPI: Core revision 20060707

11:37:09 db33 kernel: Using local APIC timer interrupts.

11:37:09 db33 kernel: Detected 8.312 MHz APIC timer.

11:37:09 db33 kernel: SMP alternatives: switching to SMP code

11:37:09 db33 kernel: Booting processor 1/128 APIC 0x2

11:37:09 db33 kernel: Initializing CPU#1

11:37:09 db33 kernel: Calibrating delay using timer specific routine.. 3989.96 BogoMIPS (lpj=1994981)

11:37:09 db33 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K

11:37:09 db33 kernel: CPU: L2 cache: 256K

11:37:09 db33 kernel: CPU: L3 cache: 18432K

11:37:09 db33 kernel: CPU 1/2 -> Node 0

11:37:09 db33 kernel: CPU: Physical Processor ID: 0

11:37:09 db33 kernel: CPU: Processor Core ID: 1

11:37:09 db33 kernel: CPU1: Thermal monitoring enabled (TM1)

11:37:09 db33 kernel: Intel(R) Xeon(R) CPU           X6550  @ 2.00GHz stepping 06

11:37:09 db33 kernel: SMP alternatives: switching to SMP code

11:37:09 db33 kernel: Booting processor 2/128 APIC 0x4

11:37:09 db33 kernel: Initializing CPU#2

11:37:09 db33 kernel: Calibrating delay using timer specific routine.. 3989.92 BogoMIPS (lpj=1994963)

11:37:09 db33 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K

11:37:09 db33 kernel: CPU: L2 cache: 256K

11:37:09 db33 kernel: CPU: L3 cache: 18432K

11:37:09 db33 kernel: CPU 2/4 -> Node 0

11:37:09 db33 kernel: CPU: Physical Processor ID: 0

11:37:09 db33 kernel: CPU: Processor Core ID: 2

11:37:09 db33 kernel: CPU2: Thermal monitoring enabled (TM1)

11:37:09 db33 kernel: Intel(R) Xeon(R) CPU           X6550  @ 2.00GHz stepping 06

11:37:09 db33 kernel: SMP alternatives: switching to SMP code

11:37:09 db33 kernel: Booting processor 3/128 APIC 0x6

11:37:09 db33 kernel: Initializing CPU#3

11:37:09 db33 kernel: Calibrating delay using timer specific routine.. 3989.93 BogoMIPS (lpj=1994965)

11:37:09 db33 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K

11:37:09 db33 kernel: CPU: L2 cache: 256K

11:37:09 db33 kernel: CPU: L3 cache: 18432K

11:37:09 db33 kernel: CPU 3/6 -> Node 0

11:37:09 db33 kernel: CPU: Physical Processor ID: 0

11:37:09 db33 kernel: CPU: Processor Core ID: 3

11:37:09 db33 kernel: CPU3: Thermal monitoring enabled (TM1)

11:37:09 db33 kernel: Intel(R) Xeon(R) CPU           X6550  @ 2.00GHz stepping 06

11:37:09 db33 kernel: SMP alternatives: switching to SMP code

11:37:09 db33 kernel: Booting processor 4/128 APIC 0x10

11:37:09 db33 kernel: Initializing CPU#4

11:37:09 db33 kernel: Calibrating delay using timer specific routine.. 3989.92 BogoMIPS (lpj=1994963)

11:37:09 db33 kernel: CPU: L1 I cache: 32K, L1 D cache: 32K

11:37:09 db33 kernel: CPU: L2 cache: 256K

11:37:09 db33 kernel: CPU: L3 cache: 18432K

11:37:09 db33 kernel: CPU 4/10 -> Node 0

11:37:09 db33 kernel: CPU: Physical Processor ID: 0

11:37:09 db33 kernel: CPU: Processor Core ID: 8

11:37:09 db33 kernel: CPU4: Thermal monitoring enabled (TM1)

11:37:09 db33 kernel: Intel(R) Xeon(R) CPU           X6550  @ 2.00GHz stepping 06

......

这是db系统的日志，记载了脑裂信息。