Drive failure? ddrescue to the rescue!

A server’s primary hard drive crashed hard one fine day. Yes, a non-RAIDed “server” (*sigh*). The last thing heard from it was screams of read-only, and it was gone.

We got the drive mounted via a SATA dock to a new machine. At this point, all attempts at mounting and fscking failed, and dmesg was spewing out horribly ugly ext errors that would make any administrator cringe…

sd 2:0:0:0: SCSI error: return code = 0x08000002
sdc: Current: sense key: Medium Error
Add. Sense: Unrecovered read error

end_request: I/O error, dev sdc, sector 214199
JBD: Failed to read block at offset 24684
JBD: recovery failed
EXT3-fs: error loading journal.

It looked pretty hopeless, and we were about to look to not-so-fresh backups and fail over. Then I remembered a magical tool called “ddrescue“. I had used it previously in a crisis-situation and it had proved to save my ass without complaint. Could it work in a server environment dealing with a 1TB drive?

hmmmmmmmmmmmmmmm………

[Disclaimer] Before proceeding, I must again warn all readers that you should always use caution when modifying a file system or disk. If you do not have access to the console (though NOT recommended, but sometimes necessary) be sure you perform any disk operations inside of a screen:

[~]# screen ddrescue -r3 /dev/sdc1 /home2/rescued.image log

Breakdown:

-r3 defines # of retries to give a bad block before calling it hopeless, in this case retry 3 times and move on.

/dev/sdc1 is the drive/partition I’m trying to recover data from

/home2/rescued.image is where the recovered data goes

and lastly, “log” simply tells the program where to log (pwd/log in this case)

And off it goes…

Initial status (read from logfile)
rescued: 8416 MB, errsize: 94720 B, current rate: 30146 kB/s
ipos: 8417 MB, errors: 1, average rate: 14700 kB/s
opos: 8417 MB, time from last successful read: 0 s
Copying non-tried blocks…

After it’s done all of your dirty work, you’re going to want to clean up or “fsck” the image prior to mounting it:

# e2fsck -y /home2/rescued.image

The -y flag negates the need to answer “yes” to all that it’s fixing.

After fsck, simply mount the image:

# mount -o loop /home2/rescued.image /mnt

Want to try saving your own data, and save thousands of dollars on data recovery by doing it yourself? Here ya go:

Obtaining ddrescue:

Download the RPM’s for any RHEL/CentOS based systems here:
http://packages.sw.be/ddrescue/

For all the Debian/Ubuntu Linuxers out there, simply use aptitude:

[eriks@jaded: ~]$ sudo apt-get install ddrescue

In a hurry? Want to speed up the process? Learn how to ionice and renice the ddrescue process here: http://erikimh.com/nice

Share on Facebook

Tweet