How to re-add the failed member to raid

[root@qht174 ~]# cat /etc/redhat-release

CentOS release 6.2 (Final)

 

[root@qht174 ~]# uname -a

Linux qht174 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux

 

 

1. Disk A and B as members of array /dev/md3, the initial status is normal.

 

 [root@qht174 ~]# cat /proc/mdstat

Personalities : [raid1]

md3 : active raid1 sdb5[3] sda5[2]

      62953977 blocks super 1.2 [2/2] [UU]

 

1. Remove the Disk B from array.

 

 [root@qht174 ~]# mdadm /dev/md3 -f /dev/sdb5

mdadm: set /dev/sdb5 faulty in /dev/md3

[root@qht174 ~]# mdadm /dev/md3 -r /dev/sdb5

mdadm: hot removed /dev/sdb5 from /dev/md3

[root@qht174 ~]# cat /proc/mdstat

Personalities : [raid1]

md3 : active raid1 sda5[2]

      62953977 blocks super 1.2 [2/1] [U_]

 

unused devices: <none>

 

1. Power off server and replace disk B with disk C, then power on the server.

 

1. Below steps are add disk C to the array.

 

[root@qht174 /]# sfdisk -d /dev/sda | sfdisk --force /dev/sdb

[root@qht174 /]# fdisk -l

 

Disk /dev/sdb: 80.0 GB, 80026361856 bytes

255 heads, 63 sectors/track, 9729 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0xd3b7d3b7

 

   Device Boot      Start         End      Blocks   Id  System

/dev/sdb1   *           1          64      512000   83  Linux

Partition 1 does not end on cylinder boundary.

/dev/sdb2              64        1370    10485760   83  Linux

/dev/sdb3            1370        1892     4194304   82  Linux swap / Solaris

/dev/sdb4            1892        9729    62955104+   5  Extended

/dev/sdb5            1892        9729    62955073   fd  Linux raid autodetect

 

Disk /dev/sda: 80.0 GB, 80026361856 bytes

255 heads, 63 sectors/track, 9729 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0xd3b7d3b7

 

   Device Boot      Start         End      Blocks   Id  System

/dev/sda1   *           1          64      512000   83  Linux

Partition 1 does not end on cylinder boundary.

/dev/sda2              64        1370    10485760   83  Linux

/dev/sda3            1370        1892     4194304   82  Linux swap / Solaris

/dev/sda4            1892        9729    62955104+   5  Extended

/dev/sda5            1892        9729    62955073   fd  Linux raid autodetect

 

Disk /dev/md3: 64.5 GB, 64464872448 bytes

2 heads, 4 sectors/track, 15738494 cylinders

Units = cylinders of 8 * 512 = 4096 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x00000000

 

Disk /dev/md3 doesn't contain a valid partition table

 

[root@qht174 /]# mdadm /dev/md3 -a /dev/sdb5

mdadm: added /dev/sdb5

 

[root@qht174 /]#  cat /proc/mdstat

Personalities : [raid1]

md3 : active raid1 sdb5[3] sda5[2]

      62953977 blocks super 1.2 [2/2] [UU]

 

unused devices: <none>

 

1. Remove Disk C from array after disks are synchronized

 

 [root@qht174 /]# mdadm /dev/md3 -f /dev/sdb5

mdadm: set /dev/sdb5 faulty in /dev/md3

[root@qht174 /]# mdadm /dev/md3 -r /dev/sdb5

mdadm: hot removed /dev/sdb5 from /dev/md3

[root@qht174 /]# cat /proc/mdstat

Personalities : [raid1]

md3 : active raid1 sda5[2]

      62953977 blocks super 1.2 [2/1] [U_]

 

unused devices: <none>

 

1. Power off and replace Disk C with Disk B, then power on the server.

 

 [root@qht174 ~]#  cat /proc/mdstat

Personalities : [raid1]

md3 : active raid1 sda5[2]

      62953977 blocks super 1.2 [2/1] [U_]

 

unused devices: <none>

 

1. Try to add /dev/sdb5 to the array, but it’s not success, likes below:

 

 [root@qht174 ~]# mdadm /dev/md3 -a /dev/sdb5

mdadm: /dev/sdb5 reports being an active member for /dev/md3, but a --re-add fails.

mdadm: not performing --add as that would convert /dev/sdb5 in to a spare.

mdadm: To make this a spare, use "mdadm --zero-superblock /dev/sdb5" first.

 

[root@qht174 ~]# mdadm --examine /dev/sdb5

/dev/sdb5:

          Magic : a92b4efc

        Version : 1.2

    Feature Map : 0x0

     Array UUID : b29493ff:ef61ac43:e135b452:2c460d3b

           Name : qht174:3  (local to host qht174)

  Creation Time : Mon May  6 16:41:25 2019

     Raid Level : raid1

   Raid Devices : 2

 

 Avail Dev Size : 125908098 (60.04 GiB 64.46 GB)

     Array Size : 125907954 (60.04 GiB 64.46 GB)

  Used Dev Size : 125907954 (60.04 GiB 64.46 GB)

    Data Offset : 2048 sectors

   Super Offset : 8 sectors

          State : clean

    Device UUID : f112e378:18bc5522:2ba5bb9b:a2f4ae9d

 

    Update Time : Tue May  7 10:23:29 2019

       Checksum : 7b641820 - correct

         Events : 89

 

 

   Device Role : Active device 1

   Array State : AA ('A' == active, '.' == missing)

 

--the Array State is ‘AA’，so we cannot re-add any member to this array.

 

1. Erase the superblock on this drive like it says.

 [root@qht174 ~]# mdadm --zero-superblock /dev/sdb5

[root@qht174 ~]# mdadm --examine /dev/sdb5

mdadm: No md superblock detected on /dev/sdb5.

 

[root@qht174 ~]# mdadm --examine /dev/sda5

/dev/sda5:

          Magic : a92b4efc

        Version : 1.2

    Feature Map : 0x0

     Array UUID : b29493ff:ef61ac43:e135b452:2c460d3b

           Name : qht174:3  (local to host qht174)

  Creation Time : Mon May  6 16:41:25 2019

     Raid Level : raid1

   Raid Devices : 2

 

 Avail Dev Size : 125908098 (60.04 GiB 64.46 GB)

     Array Size : 125907954 (60.04 GiB 64.46 GB)

  Used Dev Size : 125907954 (60.04 GiB 64.46 GB)

    Data Offset : 2048 sectors

   Super Offset : 8 sectors

          State : clean

    Device UUID : bde7bcb4:85a27efa:b1991efc:68e673f1

 

    Update Time : Tue May  7 11:03:25 2019

       Checksum : 438bc322 - correct

         Events : 120

 

 

   Device Role : Active device 0

   Array State : A. ('A' == active, '.' == missing)

 

--Now the Array State is ‘A.’, it means missing one member in this array.

 

1. Then go straight to add it back to the array.

[root@qht174 ~]# mdadm /dev/md3 -a /dev/sdb5

mdadm: added /dev/sdb5

 

1. Check out the array status in the end, it seems normal.

 [root@qht174 ~]#  cat /proc/mdstat

Personalities : [raid1]

md3 : active raid1 sdb5[3] sda5[2]

      62953977 blocks super 1.2 [2/2] [UU]

 

unused devices: <none>

 

[root@qht174 ~]# mdadm --examine /dev/sda5

/dev/sda5:

          Magic : a92b4efc

        Version : 1.2

    Feature Map : 0x0

     Array UUID : b29493ff:ef61ac43:e135b452:2c460d3b

           Name : qht174:3  (local to host qht174)

  Creation Time : Mon May  6 16:41:25 2019

     Raid Level : raid1

   Raid Devices : 2

 

 Avail Dev Size : 125908098 (60.04 GiB 64.46 GB)

     Array Size : 125907954 (60.04 GiB 64.46 GB)

  Used Dev Size : 125907954 (60.04 GiB 64.46 GB)

    Data Offset : 2048 sectors

   Super Offset : 8 sectors

          State : clean

    Device UUID : bde7bcb4:85a27efa:b1991efc:68e673f1

 

    Update Time : Tue May  7 11:35:13 2019

       Checksum : 438ecaaa - correct

         Events : 141

 

 

   Device Role : Active device 0

   Array State : AA ('A' == active, '.' == missing)

[root@qht174 ~]#  mdadm --examine /dev/sdb5

/dev/sdb5:

          Magic : a92b4efc

        Version : 1.2

    Feature Map : 0x0

     Array UUID : b29493ff:ef61ac43:e135b452:2c460d3b

           Name : qht174:3  (local to host qht174)

  Creation Time : Mon May  6 16:41:25 2019

     Raid Level : raid1

   Raid Devices : 2

 

 Avail Dev Size : 125908098 (60.04 GiB 64.46 GB)

     Array Size : 125907954 (60.04 GiB 64.46 GB)

  Used Dev Size : 125907954 (60.04 GiB 64.46 GB)

    Data Offset : 2048 sectors

   Super Offset : 8 sectors

          State : clean

    Device UUID : d6990c02:6cc7a01e:a7cf623d:c244ecb1

 

    Update Time : Tue May  7 11:35:13 2019

       Checksum : b6bd35f8 - correct

         Events : 141

 

 

   Device Role : Active device 1

   Array State : AA ('A' == active, '.' == missing)