e r i k i m h d o t c o m

Recently, I’ve seen cPanel upgrade throwing errors during upcp such as:

# /scripts/upcp
/scripts/upcp syntax OK
Running Futex Check/Fix……Done
Undefined subroutine &Cpanel::Update::automatic_updates_enabled called at /scripts/upcp line 273.

Fix:

wget -O /root/updatenow.static http://httpupdate.cpanel.net/cpanelsync/RELEASE/scripts/updatenow.static && perl /root/updatenow.static –manual

Then force the upgrade:

/scripts/upcp –force

Share on Facebook

Ever mistakenly pipe output to a file with special characters that you couldn’t remove?

-rw-r–r– 1 eriks eriks 4 2011-02-10 22:37 –fooface

Good luck. Anytime you pass any sort of command to this file, it’s going to interpret it as a flag. You can’t fool rm, echo, sed, or anything else into actually deeming this a file at this point. You do, however, have a inode for every file.

Traditional methods fail:

[eriks@jaded: ~]$ rm -f –fooface
rm: unrecognized option ‘–fooface’
Try `rm ./–fooface’ to remove the file `–fooface’.
Try `rm –help’ for more information.
[eriks@jaded: ~]$ rm -f ‘–fooface’
rm: unrecognized option ‘–fooface’
Try `rm ./–fooface’ to remove the file `–fooface’.
Try `rm –help’ for more information.

So now what, do you live forever with this annoyance of a file sitting inside your filesystem, never to be removed or touched again? Nah.

We can remove a file, simply by an inode number, but first we must find out the file inode number:

$ ls -il | grep foo

Output:

[eriks@jaded: ~]$ ls -il | grep foo
508160 drwxr-xr-x 3 eriks eriks 4096 2010-10-27 18:13 foo3
500724 -rw-r–r– 1 eriks eriks 4 2011-02-10 22:37 –fooface
589907 drwxr-xr-x 2 eriks eriks 4096 2010-11-22 18:52 tempfoo
589905 drwxr-xr-x 2 eriks eriks 4096 2010-11-22 18:48 tmpfoo

The number you see prior to the file permission set is actually the inode # of the file itself.

Hint: 500724 is inode number we want removed.

Now use find command to delete file by inode:

# find . -inum 500724 -exec rm -i {} \;

There she is.

[eriks@jaded: ~]$ find . -inum 500724 -exec rm -i {} \;
rm: remove regular file `./–fooface’? y

And done.

Share on Facebook

Recently I’ve had to do extensive work with Dell PowerEdge servers, and specifically Dell’s that use the LSI MegaRAID controllers. Anyone who has ever had to work with the LSI RAID controllers knows that the MegaCLI provided by LSI is the most cryptic command line utility in existence. MegaCli is available for Linux, DOS, Windows, Netware and Solaris.

I’ve compiled a “cheat sheet” if you will, of some common commands that will help you decipher the necessary beast of a utility that is MegaCLI.

Check it out: http://erikimh.com/megacli-cheatsheet

Share on Facebook

A server’s primary hard drive crashed hard one fine day. Yes, a non-RAIDed “server” (*sigh*). The last thing heard from it was screams of read-only, and it was gone.

We got the drive mounted via a SATA dock to a new machine. At this point, all attempts at mounting and fscking failed, and dmesg was spewing out horribly ugly ext errors that would make any administrator cringe…

sd 2:0:0:0: SCSI error: return code = 0×08000002
sdc: Current: sense key: Medium Error
Add. Sense: Unrecovered read error

end_request: I/O error, dev sdc, sector 214199
JBD: Failed to read block at offset 24684
JBD: recovery failed
EXT3-fs: error loading journal.

It looked pretty hopeless, and we were about to look to not-so-fresh backups and fail over. Then I remembered a magical tool called “ddrescue“. I had used it previously in a crisis-situation and it had proved to save my ass without complaint. Could it work in a server environment dealing with a 1TB drive?

hmmmmmmmmmmmmmmm………

[Disclaimer] Before proceeding, I must again warn all readers that you should always use caution when modifying a file system or disk. If you do not have access to the console (though NOT recommended, but sometimes necessary) be sure you perform any disk operations inside of a screen:

[~]# screen ddrescue -r3 /dev/sdc1 /home2/rescued.image log

Breakdown:

And off it goes…

Initial status (read from logfile)
rescued: 8416 MB, errsize: 94720 B, current rate: 30146 kB/s
ipos: 8417 MB, errors: 1, average rate: 14700 kB/s
opos: 8417 MB, time from last successful read: 0 s
Copying non-tried blocks…

After it’s done all of your dirty work, you’re going to want to clean up or “fsck” the image prior to mounting it:

# e2fsck -y /home2/rescued.image

The -y flag negates the need to answer “yes” to all that it’s fixing.

After fsck, simply mount the image:

# mount -o loop /home2/rescued.image /mnt

Want to try saving your own data, and save thousands of dollars on data recovery by doing it yourself? Here ya go:

Obtaining ddrescue:

Download the RPM’s for any RHEL/CentOS based systems here:
http://packages.sw.be/ddrescue/

For all the Debian/Ubuntu Linuxers out there, simply use aptitude:

[eriks@jaded: ~]$ sudo apt-get install ddrescue

In a hurry? Want to speed up the process? Learn how to ionice and renice the ddrescue process here: http://erikimh.com/nice

Share on Facebook

If rpm / yum command hangs incessantly during operations or you see error messages, it likely means your RPM database has become corrupted.

/var/lib/rpm/ stores the RPM database information, so to resolve you can just delete and rebuild the database using one of the two methods below like so:

Method # 1

rm -f /var/lib/rpm/__db*
rpm –rebuilddb -v -v

Method # 2
If you are still getting errors, then try your luck with following commands:

# mv Packages Packages-BAKUP
# db_dump Packages-BAKUP | db_load Packages
# rpm -qa
# rpm –rebuilddb

Share on Facebook

When setting up Maildir in Debian 5.0 or Ubuntu 10.x, getting the generic mail applications working correctly can be extremely tricky. When you type “mail” in a default/stock installation, it is attempting to read mbox data and will not read your “Maildir”. Annoying, since the stock configuration for mail delivery in most mail servers is now Maildir.

First, Install the ‘mailutils’ package, not the bsd mailx package.

apt-get install mailutils

Do not touch the mailutils /etc/mail.rc file; there are some promising looking variables here like mailbox-type but these do not do what is desired. When the MAIL environment variable is set correctly mailutils will automatically detect the maildir box type.

Do not touch /etc/login.defs; the promising looking lines in here are deprecated.

Edit files in /etc/pam.d, changing the pam_mail.so lines as shown (assuming ~/Maildir as the selected location):

login: default login shells
session optional pam_mail.so dir=~/Maildir standard

su: set up for proper MAIL when using su; nopen does not show ‘new mail’ message
session optional pam_mail.so dir=~/Maildir nopen

sshd: set up MAIL properly when logging in with ssh; this line probably has a noenv by default which needs removed to set this properly
session optional pam_mail.so dir=~/Maildir standard

Share on Facebook

So you have a drive that’s gone bad, and you cannot mount it. How can you recover a bad superblock from a corrupted ext3 partition to get back your data? Linux ext2/3 filesystem stores superblocks at different locations so it is possible to recover data from a corrupted partition. Your error will likely look similar to this:

#mount /dev/sda2
/dev/sda2: Input/output error
mount: /dev/sda2: can’t read superblock

or

# mount /dev/sdb1 /mnt
mount: special device /dev/sdb1 does not exist

Note: Your device may be different, I provided two separate examples from an sda partition and sdb partition, but check your server and become familiar with its’ drives before messing with your file system at all.

Mount partition using alternate superblock

Find out superblock location for /dev/sda2:

# dumpe2fs /dev/sda2 | grep superblock

WARNING! Make sure file system is UNMOUNTED.

If your system will give you a terminal type the following command, else boot Linux system from rescue disk (boot from 1st CD/DVD. At boot: prompt type command linux rescue)

Sample output:

Primary superblock at 0, Group descriptors at 1-6
Backup superblock at 32768, Group descriptors at 32769-32774
Backup superblock at 98304, Group descriptors at 98305-98310
Backup superblock at 163840, Group descriptors at 163841-163846
Backup superblock at 229376, Group descriptors at 229377-229382
Backup superblock at 294912, Group descriptors at 294913-294918
Backup superblock at 819200, Group descriptors at 819201-819206
Backup superblock at 884736, Group descriptors at 884737-884742
Backup superblock at 1605632, Group descriptors at 1605633-1605638
Backup superblock at 2654208, Group descriptors at 2654209-2654214
Backup superblock at 4096000, Group descriptors at 4096001-4096006
Backup superblock at 7962624, Group descriptors at 7962625-7962630
Backup superblock at 11239424, Group descriptors at 11239425-11239430
Backup superblock at 20480000, Group descriptors at 20480001-20480006
Backup superblock at 23887872, Group descriptors at 23887873-23887878

Now check and repair a Linux file system using alternate superblock # 32768 on /dev/sda2:

# fsck -b 32768 /dev/sda2

Sample output:

fsck 1.40.2 (12-Jul-2007)
e2fsck 1.40.2 (12-Jul-2007)
/dev/sda2 was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong for group #241 (32254, counted=32253).
Fix? yes

Free blocks count wrong for group #362 (32254, counted=32248).
Fix? yes

Free blocks count wrong for group #368 (32254, counted=27774).
Fix? yes
……….
/dev/sda2: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sda2: 59586/30539776 files (0.6% non-contiguous), 3604682/61059048 blocks

Now try to mount file system using mount command:

# mount /dev/sda2 /mnt

You can also use superblock stored at 32768 to mount partition, enter:

# mount sb={alternative-superblock} /dev/device /mnt
# mount sb=32768 /dev/sda2 /mnt

Now , try to browse and access file system:

# cd /mnt
# mkdir test
# ls -l
# cp file /path/to/safe/location

Hopefully, this article will allow you to recover the critical data you need, and that you’ve learned your lesson on THE IMPORTANCE OF BACKUPS.

Share on Facebook