Mdthbs

It looks like my system is suffering from some sort of catastrophic failure, and I'm panicking at the moment, not sure what to do.

I had a 3 drive raid 10 array. I noticed this morning that I was having trouble accessing the array (all my photos are on there). I checked mdadm and it said that one drive was dropped from the array (drive 2). I think this may have been because the computer was shutdown accidentally (there was a blackout), and the drive was kicked as a result.

I tried adding the drive back, and that worked. Then I checked the progress of the rebuild with mdstat, and it is rebuilding at 10 kb/s. Yes. It would take years to rebuild a 3 TB drive at that speed. I check dmseg, and I was getting a bunch of I/O errors for drive 3. It seems that drive 3 has some kind of hardware failure. I checked the drive's health with the disks tool (I think it's from gnome, gnome-disk-tool or something), and it had almost 6000 bad sectors, but apart from that it said it was OK.

So now I'm panicking, thinking that the drive 2 was actually still good and now when I readded it, it's resynching, probably destroying the good data it has. I then tried turning off the computer (I read it was save to do so even when mdstat is resyncing).

Unfortunately, the shut-off is not working. Pressing "ESC" showed me the terminal and it's displaying a bunch of "print_req_error: I/O error, dev sdd, sector ...." type errors. I don't know what to do. Just wait? It's been going on for 30 minutes like this.

Any advice?

It's difficult to answer your question, and this is too long for a comment, so just some general pointers.

So now I'm panicking, thinking that the drive 2 was actually still good and now when I readded it, it's resynching, probably destroying the good data it has.

Unless there is a kernel bug, re-adding a disk (in the same role and same offset it had before) does not "destroy" data. It just re-writes most of the same data that was already there, no harm done.

• the role might change if there was more than one drive missing from the array


• the offset usually only changes if you add sdx when it was sdx1 before


• if very unlucky, offset might also change if it was in a weird state before

The main problem about kicked drive, even if the drive was innocent, is that it's no longer part of the array. As soon as the array is mounted in write mode, data on the array is modified, and the data on the kicked drive is not updated along with it, so it becomes outdated and as such no longer "good".

I checked the drive's health with the disks tool (I think it's from gnome, gnome-disk-tool or something), and it had almost 6000 bad sectors, but apart from that it said it was OK.

You can't do data recovery if your drives have issues. If those 6000 bad sectors didn't appear over night, you should have replaced that drive a long time ago. RAIDs die if you don't selftest, monitor and actually replace ASAP any drives that are going bad.

Get new drives, use ddrescue to copy what you can from the old drives, then use copy-on-write overlays for data recovery experiments. With overlays you can write without modifying the original (so you don't have to re-do the disk copy and don't need a copy of the copy neither). But overlays, too, require drives that work, you can't do it with drives that have errors.