So you’ve got a process that is taking forever to complete, and is killing your server.

Make it play nice.

What is nice?
You can run a program with modified scheduling priority using nice command (19 = least favorable):

# nice -n19 /path/to/your/script.sh

This will execute the script.sh with the lowest priority, meaning the system scheduler gives less resources to it. Values range from -19 to 19, where the lower the number the higher the priority.

Just because you started a process without nice or ionice, does not mean you can’t change its’ priority to “baby it” back to normalcy.

For more information on how to do this, check out renice (man renice).

=======================================================

Disk I/O nice
ionice is what you want to use when you’ve got a process that is driving your disks insane.

How do I use ionice command?

Linux refers the scheduling class using following number system and priorities:

Scheduling class Number Possible priority
real time 1 8 priority levels are defined denoting how big a time slice a given process will receive on each scheduling window
best-effort 2 0-7, with lower number being higher priority
idle 3 Nil ( does not take a priority argument)

To display the class and priority of the running process, enter:

# ionice -p {PID}
# ionice -p 1

Sample output:

none: prio 0

Dump full web server disk / mysql backup using best effort scheduling (2) and 7 priority:

# /usr/bin/ionice -c2 -n7 /root/scripts/my/script.sh

There is no “reionice”, but the same functionality can be achieved using the ionice tool, like so (where 25374 is the PID# you want to reionice):

# ionice -p25374 -c2 -n7

Finally, you can also combine both nice and ionice together:

# nice -n 19 ionice -c2 -n7 /path/to/shell.script

Share on Facebook

This article applies only to PERC 2/SC, 2/DC, 3/SC, 3/DCL, 3/DC, 3/QC, 4/SC, 4/DC, 4e/DC, 4/Di, and CERC ATA100/4ch controllers, and specifically for Dell PowerEdge servers. Be VERY careful when making any modifications to any disk array, as data loss is always possible.

After some planning and preparation, it is possible to migrate array and virtual disks from one controller, enclosure or server to another. You can perform array and virtual disk migration on PERC 2/SC, 2/DC, 3/SC, 3/DCL, 3/DC, 3/QC, 4/SC, 4/DC, 4e/DC, 4/Di, and CERC ATA100/4ch controllers as long as the following conditions are met:

PERC 3/SC, 3/DCL, 3/DC, 3/QC, 4/SC, 4/DC, 4e/DC, and 4/Di virtual disks cannot be migrated to a PERC 2/SC or PERC 2/DC controller. PERC 2/SC and PERC 2/DC virtual disks can be migrated to a PERC 3/DCL, 3/DC, 3/QC, 4/SC, 4/DC, 4e/DC, or 4/Di controller.

When moving the array disks from one enclosure to another, the SCSI ID for each disk must remain the same.

When moving the arrays disks from one enclosure to another or when moving an external enclosure from one server to another, the enclosure must be connected to the same channel number on the controller as in the original condition.

In the case where a virtual disk consists of array disks on multiple channels, each array disk must be migrated to an enclosure that is connected to the same channel number that the array disk or enclosure was originally connected to. This also prevents migration of disks on channel 1 of a PERC 2/DC controller to a PERC 2/SC controller because the PERC 2/SC has only the single channel 0.

WARNING: A virtual disk cannot be migrated between the PERC 2, PERC 2/Si, PERC 3/Si, PERC 3/Di, CERC SATA1.5/6ch, and CERC SATA1.5/2s controller family and the PERC 2/SC, 2/DC, 3/SC, 3/DCL, 3/DC, 3/QC, 4/SC, 4/DC, 4e/DC, 4/Di, and CERC ATA100/4ch controller family.

A virtual disk cannot be migrated from one controller to another unless the receiving controller’s configuration has been cleared.

Virtual disks from one controller cannot be merged with virtual disks from another controller.

To migrate virtual disks and volumes from one system to another, do the following:

Shut down the system that the array disks are being moved from.

If the receiving controller has a preexisting virtual disk configuration on attached array disks, use the following procedure for clearing the configuration:

Shut down the receiving server.

Remove all the array disks from the controller.

Start up the receiving server and clear the configuration from the controller BIOS. After making the changes, power down the server when it indicates to reboot the system.

If the receiving controller does not have a preexisting virtual disk configuration, then shut down its server.

Place the array disks into the new enclosure.

Start up the system connected to the receiving controller. When the system connected to the new enclosure comes up, use the BIOS access (Ctrl-M) to update the controller configuration with the information saved on the array disks. Exiting the BIOS access will require one more reboot of the system.

The migration is complete. The virtual disk is now manageable through Storage Management.

Use the following procedure to enter the controller BIOS configuration screens to save the new configuration:

Reboot the PowerEdge server.

Enter the PowerEdge RAID Controller BIOS by pressing Ctrl+M when prompted during the power on self test (POST) of the controller during the bootup sequence.

Select the Configure menu option and press Enter.

Select View/Add Configuration and press Enter.

This will read the data on the array disks to restore the RAID controller’s proprietary configuration information. Press Esc and save the new configuration when prompted.

Press the Esc key and save the new configuration when prompted.

Reboot the system.

Share on Facebook

As a system administrator, I’m always looking for new tools to make my job easier. I’ve been using ‘dstat’ now for about 3 months, and it’s something I don’t know how I lived so long without. This tool is quite versatile and informative, and interacts directly with the kernel to give you a nice colorful output of what’s happening on any linux machine.

There are a ton of flags that you can pass to dstat to get the data you want.

Note: most of the flags can be combined with eachother (depending on what you’re looking for) to have it fill your screen with live data. Be sure you have a wide terminal when combining flags, or dstat will detect it and trim valuable output.

Another favorite which combines a ton of information and is very helpful if you want to relate disk activity to network/CPU usage:

# dstat -tcndylp –top-cpu

(This shows you system time, cpu, net traffic, disk io, load avgs, proc stats, and most expensive CPU process)

Possible internal stats are:

aio, cpu, cpu24, disk, disk24, disk24old, epoch, fs, int, int24, io, ipc, load, lock, mem, net, page,
page24, proc, raw, socket, swap, swapold, sys, tcp, time, udp, unix, vm

Some of the most useful parts of dstat are actually built in “modules” which also interact directly with the kernel.

The I/O modules are as follows:

–disk-tps
per disk transactions per second (tps) stats
–disk-util
per disk utilization in percentage
–top-bio-adv
show most expensive block I/O process (incl. pid and other stats)
–top-io-adv
show most expensive I/O process (incl. pid and other stats)

So, to get a live output of how much disk activity is occurring and which processes are responsible, you could combine all of the above and run:

# dstat –top-io-adv –top-bio-adv –disk-util –disk-tps

Caveats: I/O accounting is only available in kernels >2.6.20.

Another of my favorites which combines a ton of information and is very helpful if you want to relate disk activity to network/CPU usage:

# dstat -tcndylp –top-cpu

(shows you system time, cpu, net traffic, disk io, load avgs, proc stats, and most expensive CPU process)

Here’s all the possible flags you can pass to dstat:

-a, –all
equals -cdngy (default)
-c, –cpu
enable cpu stats (system, user, idle, wait, hardware interrupt, software interrupt)
-C 0,3,total
include cpu0, cpu3 and total (when using -c/–cpu)
-d, –disk
enable disk stats (read, write)
-f, –full
expand -C, -D, -I, -N and -S discovery lists
-D total,hda
include total and hda (when using -d/–disk)
-g, –page
enable page stats (page in, page out)
-i, –int
enable interrupt stats
-I 5,10
include interrupt 5 and 10 (when using -i/–int)
-l, –load
enable load average stats (1 min, 5 mins, 15mins)
-m, –mem
enable memory stats (used, buffers, cache, free)
-n, –net
enable network stats (receive, send)
-N eth1,total
include eth1 and total (when using -n/–net)
-p, –proc
enable process stats (runnable, uninterruptible, new)
-r, –io
enable I/O request stats (read, write requests)
-s, –swap
enable swap stats (used, free)
-S swap1,total
include swap1 and total (when using -s/–swap)
-t, –time
enable time/date output
-T, –epoch
enable time counter (seconds since epoch)
-y, –sys
enable system stats (interrupts, context switches)
–fs enable filesystem stats (open files, inodes)
–ipc enable ipc stats (message queue, semaphores, shared memory)
–lock enable file lock stats (posix, flock, read, write)
–raw enable raw stats (raw sockets)
–socket
enable socket stats (total, tcp, udp, raw, ip-fragments)
–tcp enable tcp stats (listen, established, syn, time_wait, close)
–udp enable udp stats (listen, active)
–unix enable unix stats (datagram, stream, listen, active)
–vm enable vm stats (hard pagefaults, soft pagefaults, allocated, free)

Enjoy.

Share on Facebook

Linux sometimes dumps a huge file when a script crashes. These core files can build up and eat away valuable disk space. Some other methods of deleting core files will damage your server. Here are a few simple commands I use to find and delete these core dump files safely.

Deleting files on Linux can be dangerous – one wrong character can destroy important files. I read a few other methods of removing core files that will potentially delete important files while leaving the core dumps on disk. Do not use these methods. The problem is that they are looking for files named “core” with no extension. If you do this you’re just trashing system files and leaving the dump files behind.

Linux core dump file names look like “core.4324″ – they have a three or four digit number as the extension. If your system has a different core dump format you’ll need to adjust these commands. We need a reliable way to find these files and only these files. We will need to find file names with the following criteria:

Linux find is a powerful tool, and with a little regular expression magic we can tackle all of these at once. Before deleting anything, we should do a test run with find to make sure we are not catching anything unwanted. Run the following find command to locate all core dump files in the current directory and all subdirectories.

find . -type f -regex “.*/core\.[0-9][0-9][0-9][0-9]$”

The regex in the above command matches the file name “core.xxxx” where “xxxx” is exactly four numbers 0-9. If the results look correct, we can delete all those dump files with the following command.

If your core files are dumping with three numbers, or five numbers, you will want to add another [0-9] or remove a [0-9] expression accordingly.

If you’ve seen your results, and you want them gone, go ahead and execute this:

find . -type f -regex “.*/core\.[0-9][0-9][0-9][0-9]$” -exec rm {} \;

The one downfall of this method is that it’s slower, since regular expressions take time to process. But this is a small price to pay for getting rid of these files in large numbers safely. If you see any errors with the above code or have any experiences to share please leave a comment below.

Share on Facebook

Encountered this gem when I was attempting to install an overly simple/normal perl module (HTML::Template, though all will fail as a result of this). This actually has nothing to do with cPanel, the kernel, or CPAN. Instead, you’re actually hitting a PAM ulimit that is exhausting all of the available reserved memory required to install this module.

Catching error: “Out of memory during \”large\” request for 1052672 bytes, total sbrk() is 106149888 bytes at /usr/local/lib/perl5/5.8.8/CPAN.pm line 5337.\cJ” at /usr/local/lib/perl5/5.8.8/CPAN.pm line 359
CPAN::shell() called at /usr/local/bin/cpan line 198
Going to read /home/xxxxx/.cpan/sources/authors/01mailrc.txt.gz
Out of memory during request for 16400 bytes, total sbrk() is 106149888 bytes!
Lockfile removed.

If you su into the user you’re attempting to install this module for and snap a ulimit -m (max memory size, rss) you’ll see:

max memory size (kbytes, -m) 200000

This is actually limiting the user to a specific amount of available memory, causing the install to fail miserably.

Here’s what you do to fix. Edit /etc/security/limits.conf and allow the user account to use an unlimited amount of memory (temporarily, if you wish):

username hard rss 0

Now do a ulimit -m:

max memory size (kbytes, -m) unlimited

Now, try again:

cpan -i ‘HTML::Template’

And watch it install.

Share on Facebook

While replacing a bad drive with a drive that used to be part of another RAID array configuration, the RAID refused to automatically rebuild, thinking that I might want to import the configuration from this disk (or that there’s data on there that I might need).

Simply inserting the drive doesn’t make the controller rebuild the array with that disk. Here’s how to manually make the drive get along with the rest of the new array:

(Note: this is a 64 bit server, so the MegaCli client I’m using is called “MegaCli64″, if you are not running x64, you can simply substitute the commands below with the path to your megacli binary.)

server:~# MegaCli64 -PDlist -aALL -a0
[...]
Enclosure Device ID: 32
Slot Number: 4
[...]
Firmware state: Unconfigured(bad)
[...]
Secured: Unsecured
Locked: Unlocked
Foreign State: Foreign
[...]

Based on the information obtained above, I now know that the disk drive I just replaced is [32:4] ([enclosureid:slotnumber]) and is currently being reported as ‘Unconfigured(bad)’.

To bring this drive back online run:

server:~# MegaCli64 -PDMakeGood -PhysDrv[32:4] -a0
Adapter: 0: EnclId-32 SlotId-4 state changed to Unconfigured-Good.

The controller will now recognize the disk as being a “foreign” one. This does not mean it was made in Japan (though, it likely was). It means it has detected some RAID configuration/data on it and thus, considers it as a disk being part of an array that may be imported into current controller configuration. Because of this, it will not automatically rebuild until you force it to.

Now you can ask the controller to scan for foreign configurations and remove them:

server:~# MegaCli64 -CfgForeign -Scan -a0
There are 1 foreign configuration(s) on controller 0.

server:~# MegaCli64 -CfgForeign -Clear -a0
Foreign configuration 0 is cleared on controller 0.

The disk should now be available for rebuilding into your new RAID array. To confirm, run this:

server:~# MegaCli64 -PDList -a0
[...]
Enclosure Device ID: 32
Slot Number: 4
[...]
Firmware state: Unconfigured(good), Spun Up
Foreign State: None
[...]

Excellent. We have a good, recognized (yet still unconfigured) drive now. Now we have all we need to add the disk back into the new array, and rebuild:

Get the disk [32:4] back into array 1, as disk 4:

server:~# MegaCli64 -PdReplaceMissing -PhysDrv[32:4] -array1 -row4 -a0
Adapter: 0: Missing PD at Array 1, Row 4 is replaced

And finally start rebuilding it:

server:~# megacli -PDRbld -Start -PhysDrv[32:4] -a0
Started rebuild progress on device(Encl-32 Slot-4)

Now, sit back, relax, grab a smoke and wait for it to rebuild itself into your new RAID array. Not so foreign anymore, huh?

-E

Share on Facebook

An excellent program for monitoring users and applications is psacct. This program will work in the background of your system recording what all users are doing on your system as well as the resources that are being consumed. I use it daily for resource abuse tracking, statistics generation, CPU usage trending, process identification and more.

Since I administer both Ubuntu/Debian and CentOS/RHEL servers, I’ll provide both methods of installation here:

CentOS:

yum install psacct

Ubuntu:

sudo apt-get install acct

The most useful command that now exists on your box is in /usr/bin and called “sa”.

SA has the following output fields:

cpu – sum of system and user time in cpu minutes
re – actual time in minutes
k – cpu-time averaged core usage, in 1k units
k*sec – cpu storage integral (kilo-core seconds)
u – user cpu time in cpu minutes
s – system time in cpu minutes

Start off by allowing process accounting to collect some data from your system. This should really be left for 24-48 hours, though if you’re too excited to start parsing the results, lets continue.

This will show you averages for the all the activity for this server overtime. This log file grows larger over time as more commands run, and the longer you allow it to collect data.

# sa -m

This will show you the sum of the system and user time in cpu minutes for specific commands.

# sa -u | grep username

This will give you a combined total for a specific user:

# sa -u |grep username|awk ‘BEGIN{TOTAL=0}{TOTAL=TOTAL+$2}END{print TOTAL}’

This option will show each of the programs on your server so you may evaluate, real time, memory usage and which programs are running:

# sa -a

Hint: append the “-c” flag to any of the options to view the highest percentile users first.

What is the difference between CPU Seconds and CP?

So:

cp = ((user_time+system_time)/60)/100

I know that this can be confusing, and it took myself and my organization some time to master the technology due to the lack of documentation available. I hope this article will also help someone else’s resource management lightbulb illuminate.

-e

Share on Facebook

Remote login using the SSH protocol is a common activity in my line of work. With the SSH protocol, the responsibility is on the SSH client to verify the identity of the host to which it is connecting. The host identify is established by its SSH host key. Typically, the host key is auto-created during initial SSH installation setup.

By default, the SSH client verifies the host key against a local file containing known, rustworthy machines. This provides protection against possible Man-In-The-Middle attacks. However, there are situations in which you want to bypass this verification step. This article explains how to disable host key checking using OpenSSH, a popular Free and Open-Source implementation of SSH.

When you login to a remote host for the first time, the remote host’s host key is most likely unknown to the SSH client. The default behavior is to ask the user to confirm the fingerprint of the host key.

$ ssh erik@192.168.0.100
The authenticity of host ’192.168.0.100 (192.168.0.100)’ can’t be established.
RSA key fingerprint is 3f:1b:f4:bd:c5:aa:c1:1f:bf:4e:2e:cf:53:fa:d8:59.
Are you sure you want to continue connecting (yes/no)?

If your answer is yes, the SSH client continues login, and stores the host key locally in the file ~/.ssh/known_hosts. You only need to validate the host key the first time around: in subsequent logins, you will not be prompted to confirm it again.

Yet, from time to time, when you try to remote login to the same host from the same origin, you may be refused with the following warning message:

$ ssh erik@192.168.0.100
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
3f:1b:f4:bd:c5:aa:c1:1f:bf:4e:2e:cf:53:fa:d8:59.
Please contact your system administrator.
Add correct host key in /home/erik/.ssh/known_hosts to get rid of this message.
Offending key in /home/erik/.ssh/known_hosts:3
RSA host key for 192.168.0.100 has changed and you have requested strict checking.
Host key verification failed.

There are multiple possible reasons why the remote host key changed. A Man-in-the-Middle attack is only one possible reason. Other possible reasons include:

OpenSSH was re-installed on the remote host but, for whatever reason, the original host key was not restored.
The remote host was replaced legitimately by another machine.

If you are sure that this is harmless, you can use either 1 of 2 methods below to trick openSSH to let you login. But be warned that you have become vulnerable to man-in-the-middle attacks.

The first method is to remove the remote host from the ~/.ssh/known_hosts file. Note that the warning message already tells you the line number in the known_hosts file that corresponds to the target remote host. The offending line in the above example is line 3(“Offending key in /home/erik/.ssh/known_hosts:3″)

You can use the following one liner to remove that one line (line 3) from the file.

$ sed -i 3d ~/.ssh/known_hosts

Note that with the above method, you will be prompted to confirm the host key fingerprint when you run ssh to login.

The second method uses two openSSH parameters:
StrictHostKeyCheckin, and UserKnownHostsFile.

This method tricks SSH by configuring it to use an empty known_hosts file, and NOT to ask you to confirm the remote host identity key.

$ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no erik@192.168.0.100
Warning: Permanently added ’192.168.0.100′ (RSA) to the list of known hosts.
erik@192.168.0.100′s password:

The UserKnownHostsFile parameter specifies the database file to use for storing the user host keys (default is ~/.ssh/known_hosts).

The /dev/null file is a special system device file that discards anything and everything written to it, and when used as the input file, returns End Of File immediately.

By configuring the null device file as the host key database, SSH is fooled into thinking that the SSH client has never connected to any SSH server before, and so will never run into a mismatched host key.

The parameter StrictHostKeyChecking specifies if SSH will automatically add new host keys to the host key database file. By setting it to no, the host key is automatically added, without user confirmation, for all first-time connection. Because of the null key database file, all connection is viewed as the first-time for any SSH server host. Therefore, the host key is automatically added to the host key database with no user confirmation. Writing the key to the /dev/null file discards the key and reports success.

Please refer to this excellent article about host keys and key checking.

By specifying the above 2 SSH options on the command line, you can bypass host key checking for that particular SSH login. If you want to bypass host key checking on a permanent basis, you need to specify those same options in the SSH configuration file.

You can edit the global SSH configuration file (/etc/ssh/ssh_config) if you want to make the changes permanent for all users.

If you want to target a particular user, modify the user-specific SSH configuration file (~/.ssh/config). The instructions below apply to both files.

Suppose you want to bypass key checking for a particular subnet (192.168.0.0/24).

Add the following lines to the beginning of the SSH configuration file.

Host 192.168.0.*
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null

Note that the configuration file should have a line like Host * followed by one or more parameter-value pairs. Host *means that it will match any host. Essentially, the parameters following Host * are the general defaults. Because the first matched value for each SSH parameter is used, you want to add the host-specific or subnet-specific parameters to the beginning of the file.

As a final word of caution, unless you know what you are doing, it is probably best to bypass key checking on a case by case basis, rather than making blanket permanent changes to the SSH configuration files.

Share on Facebook