e r i k i m h d o t c o m

Recently I’ve had to do extensive work with Dell PowerEdge servers, and specifically Dell’s that use the LSI MegaRAID controllers. Anyone who has ever had to work with the LSI RAID controllers knows that the MegaCLI provided by LSI is the most cryptic command line utility in existence. MegaCli is available for Linux, DOS, Windows, Netware and Solaris.

I’ve compiled a “cheat sheet” if you will, of some common commands that will help you decipher the necessary beast of a utility that is MegaCLI.

Check it out: http://erikimh.com/megacli-cheatsheet

Share on Facebook

A server’s primary hard drive crashed hard one fine day. Yes, a non-RAIDed “server” (*sigh*). The last thing heard from it was screams of read-only, and it was gone.

We got the drive mounted via a SATA dock to a new machine. At this point, all attempts at mounting and fscking failed, and dmesg was spewing out horribly ugly ext errors that would make any administrator cringe…

sd 2:0:0:0: SCSI error: return code = 0×08000002
sdc: Current: sense key: Medium Error
Add. Sense: Unrecovered read error

end_request: I/O error, dev sdc, sector 214199
JBD: Failed to read block at offset 24684
JBD: recovery failed
EXT3-fs: error loading journal.

It looked pretty hopeless, and we were about to look to not-so-fresh backups and fail over. Then I remembered a magical tool called “ddrescue“. I had used it previously in a crisis-situation and it had proved to save my ass without complaint. Could it work in a server environment dealing with a 1TB drive?

hmmmmmmmmmmmmmmm………

[Disclaimer] Before proceeding, I must again warn all readers that you should always use caution when modifying a file system or disk. If you do not have access to the console (though NOT recommended, but sometimes necessary) be sure you perform any disk operations inside of a screen:

[~]# screen ddrescue -r3 /dev/sdc1 /home2/rescued.image log

Breakdown:

And off it goes…

Initial status (read from logfile)
rescued: 8416 MB, errsize: 94720 B, current rate: 30146 kB/s
ipos: 8417 MB, errors: 1, average rate: 14700 kB/s
opos: 8417 MB, time from last successful read: 0 s
Copying non-tried blocks…

After it’s done all of your dirty work, you’re going to want to clean up or “fsck” the image prior to mounting it:

# e2fsck -y /home2/rescued.image

The -y flag negates the need to answer “yes” to all that it’s fixing.

After fsck, simply mount the image:

# mount -o loop /home2/rescued.image /mnt

Want to try saving your own data, and save thousands of dollars on data recovery by doing it yourself? Here ya go:

Obtaining ddrescue:

Download the RPM’s for any RHEL/CentOS based systems here:
http://packages.sw.be/ddrescue/

For all the Debian/Ubuntu Linuxers out there, simply use aptitude:

[eriks@jaded: ~]$ sudo apt-get install ddrescue

In a hurry? Want to speed up the process? Learn how to ionice and renice the ddrescue process here: http://erikimh.com/nice

Share on Facebook

If rpm / yum command hangs incessantly during operations or you see error messages, it likely means your RPM database has become corrupted.

/var/lib/rpm/ stores the RPM database information, so to resolve you can just delete and rebuild the database using one of the two methods below like so:

Method # 1

rm -f /var/lib/rpm/__db*
rpm –rebuilddb -v -v

Method # 2
If you are still getting errors, then try your luck with following commands:

# mv Packages Packages-BAKUP
# db_dump Packages-BAKUP | db_load Packages
# rpm -qa
# rpm –rebuilddb

Share on Facebook

When setting up Maildir in Debian 5.0 or Ubuntu 10.x, getting the generic mail applications working correctly can be extremely tricky. When you type “mail” in a default/stock installation, it is attempting to read mbox data and will not read your “Maildir”. Annoying, since the stock configuration for mail delivery in most mail servers is now Maildir.

First, Install the ‘mailutils’ package, not the bsd mailx package.

apt-get install mailutils

Do not touch the mailutils /etc/mail.rc file; there are some promising looking variables here like mailbox-type but these do not do what is desired. When the MAIL environment variable is set correctly mailutils will automatically detect the maildir box type.

Do not touch /etc/login.defs; the promising looking lines in here are deprecated.

Edit files in /etc/pam.d, changing the pam_mail.so lines as shown (assuming ~/Maildir as the selected location):

login: default login shells
session optional pam_mail.so dir=~/Maildir standard

su: set up for proper MAIL when using su; nopen does not show ‘new mail’ message
session optional pam_mail.so dir=~/Maildir nopen

sshd: set up MAIL properly when logging in with ssh; this line probably has a noenv by default which needs removed to set this properly
session optional pam_mail.so dir=~/Maildir standard

Share on Facebook

So you have a drive that’s gone bad, and you cannot mount it. How can you recover a bad superblock from a corrupted ext3 partition to get back your data? Linux ext2/3 filesystem stores superblocks at different locations so it is possible to recover data from a corrupted partition. Your error will likely look similar to this:

#mount /dev/sda2
/dev/sda2: Input/output error
mount: /dev/sda2: can’t read superblock

or

# mount /dev/sdb1 /mnt
mount: special device /dev/sdb1 does not exist

Note: Your device may be different, I provided two separate examples from an sda partition and sdb partition, but check your server and become familiar with its’ drives before messing with your file system at all.

Mount partition using alternate superblock

Find out superblock location for /dev/sda2:

# dumpe2fs /dev/sda2 | grep superblock

WARNING! Make sure file system is UNMOUNTED.

If your system will give you a terminal type the following command, else boot Linux system from rescue disk (boot from 1st CD/DVD. At boot: prompt type command linux rescue)

Sample output:

Primary superblock at 0, Group descriptors at 1-6
Backup superblock at 32768, Group descriptors at 32769-32774
Backup superblock at 98304, Group descriptors at 98305-98310
Backup superblock at 163840, Group descriptors at 163841-163846
Backup superblock at 229376, Group descriptors at 229377-229382
Backup superblock at 294912, Group descriptors at 294913-294918
Backup superblock at 819200, Group descriptors at 819201-819206
Backup superblock at 884736, Group descriptors at 884737-884742
Backup superblock at 1605632, Group descriptors at 1605633-1605638
Backup superblock at 2654208, Group descriptors at 2654209-2654214
Backup superblock at 4096000, Group descriptors at 4096001-4096006
Backup superblock at 7962624, Group descriptors at 7962625-7962630
Backup superblock at 11239424, Group descriptors at 11239425-11239430
Backup superblock at 20480000, Group descriptors at 20480001-20480006
Backup superblock at 23887872, Group descriptors at 23887873-23887878

Now check and repair a Linux file system using alternate superblock # 32768 on /dev/sda2:

# fsck -b 32768 /dev/sda2

Sample output:

fsck 1.40.2 (12-Jul-2007)
e2fsck 1.40.2 (12-Jul-2007)
/dev/sda2 was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong for group #241 (32254, counted=32253).
Fix? yes

Free blocks count wrong for group #362 (32254, counted=32248).
Fix? yes

Free blocks count wrong for group #368 (32254, counted=27774).
Fix? yes
……….
/dev/sda2: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sda2: 59586/30539776 files (0.6% non-contiguous), 3604682/61059048 blocks

Now try to mount file system using mount command:

# mount /dev/sda2 /mnt

You can also use superblock stored at 32768 to mount partition, enter:

# mount sb={alternative-superblock} /dev/device /mnt
# mount sb=32768 /dev/sda2 /mnt

Now , try to browse and access file system:

# cd /mnt
# mkdir test
# ls -l
# cp file /path/to/safe/location

Hopefully, this article will allow you to recover the critical data you need, and that you’ve learned your lesson on THE IMPORTANCE OF BACKUPS.

Share on Facebook

So you’ve got a process that is taking forever to complete, and is killing your server.

Make it play nice.

What is nice?
You can run a program with modified scheduling priority using nice command (19 = least favorable):

# nice -n19 /path/to/your/script.sh

This will execute the script.sh with the lowest priority, meaning the system scheduler gives less resources to it. Values range from -19 to 19, where the lower the number the higher the priority.

Just because you started a process without nice or ionice, does not mean you can’t change its’ priority to “baby it” back to normalcy.

For more information on how to do this, check out renice (man renice).

=======================================================

Disk I/O nice
ionice is what you want to use when you’ve got a process that is driving your disks insane.

How do I use ionice command?

Linux refers the scheduling class using following number system and priorities:

Scheduling class Number Possible priority
real time 1 8 priority levels are defined denoting how big a time slice a given process will receive on each scheduling window
best-effort 2 0-7, with lower number being higher priority
idle 3 Nil ( does not take a priority argument)

To display the class and priority of the running process, enter:

# ionice -p {PID}
# ionice -p 1

Sample output:

none: prio 0

Dump full web server disk / mysql backup using best effort scheduling (2) and 7 priority:

# /usr/bin/ionice -c2 -n7 /root/scripts/my/script.sh

There is no “reionice”, but the same functionality can be achieved using the ionice tool, like so (where 25374 is the PID# you want to reionice):

# ionice -p25374 -c2 -n7

Finally, you can also combine both nice and ionice together:

# nice -n 19 ionice -c2 -n7 /path/to/shell.script

Share on Facebook

This article applies only to PERC 2/SC, 2/DC, 3/SC, 3/DCL, 3/DC, 3/QC, 4/SC, 4/DC, 4e/DC, 4/Di, and CERC ATA100/4ch controllers, and specifically for Dell PowerEdge servers. Be VERY careful when making any modifications to any disk array, as data loss is always possible.

After some planning and preparation, it is possible to migrate array and virtual disks from one controller, enclosure or server to another. You can perform array and virtual disk migration on PERC 2/SC, 2/DC, 3/SC, 3/DCL, 3/DC, 3/QC, 4/SC, 4/DC, 4e/DC, 4/Di, and CERC ATA100/4ch controllers as long as the following conditions are met:

PERC 3/SC, 3/DCL, 3/DC, 3/QC, 4/SC, 4/DC, 4e/DC, and 4/Di virtual disks cannot be migrated to a PERC 2/SC or PERC 2/DC controller. PERC 2/SC and PERC 2/DC virtual disks can be migrated to a PERC 3/DCL, 3/DC, 3/QC, 4/SC, 4/DC, 4e/DC, or 4/Di controller.

When moving the array disks from one enclosure to another, the SCSI ID for each disk must remain the same.

When moving the arrays disks from one enclosure to another or when moving an external enclosure from one server to another, the enclosure must be connected to the same channel number on the controller as in the original condition.

In the case where a virtual disk consists of array disks on multiple channels, each array disk must be migrated to an enclosure that is connected to the same channel number that the array disk or enclosure was originally connected to. This also prevents migration of disks on channel 1 of a PERC 2/DC controller to a PERC 2/SC controller because the PERC 2/SC has only the single channel 0.

WARNING: A virtual disk cannot be migrated between the PERC 2, PERC 2/Si, PERC 3/Si, PERC 3/Di, CERC SATA1.5/6ch, and CERC SATA1.5/2s controller family and the PERC 2/SC, 2/DC, 3/SC, 3/DCL, 3/DC, 3/QC, 4/SC, 4/DC, 4e/DC, 4/Di, and CERC ATA100/4ch controller family.

A virtual disk cannot be migrated from one controller to another unless the receiving controller’s configuration has been cleared.

Virtual disks from one controller cannot be merged with virtual disks from another controller.

To migrate virtual disks and volumes from one system to another, do the following:

Shut down the system that the array disks are being moved from.

If the receiving controller has a preexisting virtual disk configuration on attached array disks, use the following procedure for clearing the configuration:

Shut down the receiving server.

Remove all the array disks from the controller.

Start up the receiving server and clear the configuration from the controller BIOS. After making the changes, power down the server when it indicates to reboot the system.

If the receiving controller does not have a preexisting virtual disk configuration, then shut down its server.

Place the array disks into the new enclosure.

Start up the system connected to the receiving controller. When the system connected to the new enclosure comes up, use the BIOS access (Ctrl-M) to update the controller configuration with the information saved on the array disks. Exiting the BIOS access will require one more reboot of the system.

The migration is complete. The virtual disk is now manageable through Storage Management.

Use the following procedure to enter the controller BIOS configuration screens to save the new configuration:

Reboot the PowerEdge server.

Enter the PowerEdge RAID Controller BIOS by pressing Ctrl+M when prompted during the power on self test (POST) of the controller during the bootup sequence.

Select the Configure menu option and press Enter.

Select View/Add Configuration and press Enter.

This will read the data on the array disks to restore the RAID controller’s proprietary configuration information. Press Esc and save the new configuration when prompted.

Press the Esc key and save the new configuration when prompted.

Reboot the system.

Share on Facebook

As a system administrator, I’m always looking for new tools to make my job easier. I’ve been using ‘dstat’ now for about 3 months, and it’s something I don’t know how I lived so long without. This tool is quite versatile and informative, and interacts directly with the kernel to give you a nice colorful output of what’s happening on any linux machine.

There are a ton of flags that you can pass to dstat to get the data you want.

Note: most of the flags can be combined with eachother (depending on what you’re looking for) to have it fill your screen with live data. Be sure you have a wide terminal when combining flags, or dstat will detect it and trim valuable output.

Another favorite which combines a ton of information and is very helpful if you want to relate disk activity to network/CPU usage:

# dstat -tcndylp –top-cpu

(This shows you system time, cpu, net traffic, disk io, load avgs, proc stats, and most expensive CPU process)

Possible internal stats are:

aio, cpu, cpu24, disk, disk24, disk24old, epoch, fs, int, int24, io, ipc, load, lock, mem, net, page,
page24, proc, raw, socket, swap, swapold, sys, tcp, time, udp, unix, vm

Some of the most useful parts of dstat are actually built in “modules” which also interact directly with the kernel.

The I/O modules are as follows:

–disk-tps
per disk transactions per second (tps) stats
–disk-util
per disk utilization in percentage
–top-bio-adv
show most expensive block I/O process (incl. pid and other stats)
–top-io-adv
show most expensive I/O process (incl. pid and other stats)

So, to get a live output of how much disk activity is occurring and which processes are responsible, you could combine all of the above and run:

# dstat –top-io-adv –top-bio-adv –disk-util –disk-tps

Caveats: I/O accounting is only available in kernels >2.6.20.

Another of my favorites which combines a ton of information and is very helpful if you want to relate disk activity to network/CPU usage:

# dstat -tcndylp –top-cpu

(shows you system time, cpu, net traffic, disk io, load avgs, proc stats, and most expensive CPU process)

Here’s all the possible flags you can pass to dstat:

-a, –all
equals -cdngy (default)
-c, –cpu
enable cpu stats (system, user, idle, wait, hardware interrupt, software interrupt)
-C 0,3,total
include cpu0, cpu3 and total (when using -c/–cpu)
-d, –disk
enable disk stats (read, write)
-f, –full
expand -C, -D, -I, -N and -S discovery lists
-D total,hda
include total and hda (when using -d/–disk)
-g, –page
enable page stats (page in, page out)
-i, –int
enable interrupt stats
-I 5,10
include interrupt 5 and 10 (when using -i/–int)
-l, –load
enable load average stats (1 min, 5 mins, 15mins)
-m, –mem
enable memory stats (used, buffers, cache, free)
-n, –net
enable network stats (receive, send)
-N eth1,total
include eth1 and total (when using -n/–net)
-p, –proc
enable process stats (runnable, uninterruptible, new)
-r, –io
enable I/O request stats (read, write requests)
-s, –swap
enable swap stats (used, free)
-S swap1,total
include swap1 and total (when using -s/–swap)
-t, –time
enable time/date output
-T, –epoch
enable time counter (seconds since epoch)
-y, –sys
enable system stats (interrupts, context switches)
–fs enable filesystem stats (open files, inodes)
–ipc enable ipc stats (message queue, semaphores, shared memory)
–lock enable file lock stats (posix, flock, read, write)
–raw enable raw stats (raw sockets)
–socket
enable socket stats (total, tcp, udp, raw, ip-fragments)
–tcp enable tcp stats (listen, established, syn, time_wait, close)
–udp enable udp stats (listen, active)
–unix enable unix stats (datagram, stream, listen, active)
–vm enable vm stats (hard pagefaults, soft pagefaults, allocated, free)

Enjoy.

Share on Facebook