inside the mind of a linux admin

Drive failure? ddrescue to the rescue!

A server’s primary hard drive crashed hard one fine day. Yes, a non-RAIDed “server” (*sigh*). The last thing heard from it was screams of read-only, and it was gone.

We got the drive mounted via a SATA dock to a new machine. At this point, all attempts at mounting and fscking failed, and dmesg was spewing out horribly ugly ext errors that would make any administrator cringe…

sd 2:0:0:0: SCSI error: return code = 0x08000002
sdc: Current: sense key: Medium Error
Add. Sense: Unrecovered read error

end_request: I/O error, dev sdc, sector 214199
JBD: Failed to read block at offset 24684
JBD: recovery failed
EXT3-fs: error loading journal.

It looked pretty hopeless, and we were about to look to not-so-fresh backups and fail over. Then I remembered a magical tool called “ddrescue“. I had used it previously in a crisis-situation and it had proved to save my ass without complaint. Could it work in a server environment dealing with a 1TB drive?

hmmmmmmmmmmmmmmm………

[Disclaimer] Before proceeding, I must again warn all readers that you should always use caution when modifying a file system or disk. If you do not have access to the console (though NOT recommended, but sometimes necessary) be sure you perform any disk operations inside of a screen:

[~]# screen ddrescue -r3 /dev/sdc1 /home2/rescued.image log

Breakdown:

  • -r3 defines # of retries to give a bad block before calling it hopeless, in this case retry 3 times and move on.
  • /dev/sdc1 is the drive/partition I’m trying to recover data from
  • /home2/rescued.image is where the recovered data goes
  • and lastly, “log” simply tells the program where to log (pwd/log in this case)
  • And off it goes…

    Initial status (read from logfile)
    rescued: 8416 MB, errsize: 94720 B, current rate: 30146 kB/s
    ipos: 8417 MB, errors: 1, average rate: 14700 kB/s
    opos: 8417 MB, time from last successful read: 0 s
    Copying non-tried blocks…

    After it’s done all of your dirty work, you’re going to want to clean up or “fsck” the image prior to mounting it:

    # e2fsck -y /home2/rescued.image

    The -y flag negates the need to answer “yes” to all that it’s fixing.

    After fsck, simply mount the image:

    # mount -o loop /home2/rescued.image /mnt



    Want to try saving your own data, and save thousands of dollars on data recovery by doing it yourself? Here ya go:



    Obtaining ddrescue:

    Download the RPM’s for any RHEL/CentOS based systems here:
    http://packages.sw.be/ddrescue/

    For all the Debian/Ubuntu Linuxers out there, simply use aptitude:

    [eriks@jaded: ~]$ sudo apt-get install ddrescue



    In a hurry? Want to speed up the process? Learn how to ionice and renice the ddrescue process here: http://erikimh.com/nice

    Related Posts

    synergy: How to enable crypto (encryption) and generate SSL certificate

    The newer Linux versions of the popular mouse/keyboard sharing application “synergy” now has built in encryption. Here’s how to configure it: Just simply passing the –enable-crypto flag on your synergy server without having a proper SSL certificate will result in the inability to connect to clients and generate an error message similar to this in […]

    Read More

    Change Number Pad Delete (dot) key from a comma in Ubuntu Linux

    I recently purchased a new keyboard and updated to the latest Ubuntu, I’m also an avid user of the number pad for quick input when dealing with spreadsheets or accounting. I found that my num pad’s delete key (“.”) was outputting a comma (“,”) instead. Pretty annoying? I agree, but this can be very easily […]

    Read More

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    This site uses Akismet to reduce spam. Learn how your comment data is processed.

    Tweeter button Facebook button Myspace button