Drive failure? ddrescue to the rescue!

January 25, 2011 Storage, RAID & Recovery

Archived post, originally published January 25, 2011. Kept for reference — commands and package names reflect the distributions of the time and may have changed.

A server's primary hard drive crashed hard one fine day. Yes, a non-RAIDed "server" (*sigh*). The last thing heard from it was screams of read-only, and it was gone.

We got the drive mounted via a SATA dock to a new machine. At this point, all attempts at mounting and fscking failed, and dmesg was spewing out horribly ugly ext errors that would make any administrator cringe...

sd 2:0:0:0: SCSI error: return code = 0x08000002
sdc: Current: sense key: Medium Error
    Add. Sense: Unrecovered read error

end_request: I/O error, dev sdc, sector 214199
JBD: Failed to read block at offset 24684
JBD: recovery failed
EXT3-fs: error loading journal.

It looked pretty hopeless, and we were about to look to not-so-fresh backups and fail over. Then I remembered a magical tool called "ddrescue". I had used it previously in a crisis-situation and it had proved to save my ass without complaint. Could it work in a server environment dealing with a 1TB drive?

hmmmmmmmmmmmmmmm.........

[Disclaimer] Before proceeding, I must again warn all readers that you should always use caution when modifying a file system or disk. If you do not have access to the console (though NOT recommended, but sometimes necessary) be sure you perform any disk operations inside of a screen:

[~]# screen ddrescue -r3 /dev/sdc1 /home2/rescued.image log

Breakdown:

-r3 defines # of retries to give a bad block before calling it hopeless, in this case retry 3 times and move on.

/dev/sdc1 is the drive/partition I'm trying to recover data from

/home2/rescued.image is where the recovered data goes

and lastly, "log" simply tells the program where to log (`pwd`/log in this case)

And off it goes...

Initial status (read from logfile)
rescued:     8416 MB,  errsize:   94720 B,  current rate:   30146 kB/s
   ipos:     8417 MB,   errors:       1,    average rate:   14700 kB/s
   opos:     8417 MB,     time from last successful read:       0 s
Copying non-tried blocks...

After it's done all of your dirty work, you're going to want to clean up or "fsck" the image prior to mounting it:

# e2fsck -y /home2/rescued.image

The -y flag negates the need to answer "yes" to all that it's fixing.

After fsck, simply mount the image:

# mount -o loop /home2/rescued.image /mnt

Want to try saving your own data, and save thousands of dollars on data recovery by doing it yourself? Here ya go:

Obtaining ddrescue:

Download the RPM's for any RHEL/CentOS based systems here:
http://packages.sw.be/ddrescue/

For all the Debian/Ubuntu Linuxers out there, simply use aptitude:

[eriks@jaded: ~]$ sudo apt-get install ddrescue

In a hurry? Want to speed up the process? Learn how to ionice and renice the ddrescue process here: