Remote login using the SSH protocol is a common activity in my line of work. With the SSH protocol, the responsibility is on the SSH client to verify the identity of the host to which it is connecting. The host identify is established by its SSH host key. Typically, the host key is auto-created during initial SSH installation setup.

By default, the SSH client verifies the host key against a local file containing known, rustworthy machines. This provides protection against possible Man-In-The-Middle attacks. However, there are situations in which you want to bypass this verification step. This article explains how to disable host key checking using OpenSSH, a popular Free and Open-Source implementation of SSH.

When you login to a remote host for the first time, the remote host’s host key is most likely unknown to the SSH client. The default behavior is to ask the user to confirm the fingerprint of the host key.

$ ssh erik@192.168.0.100
The authenticity of host ’192.168.0.100 (192.168.0.100)’ can’t be established.
RSA key fingerprint is 3f:1b:f4:bd:c5:aa:c1:1f:bf:4e:2e:cf:53:fa:d8:59.
Are you sure you want to continue connecting (yes/no)?

If your answer is yes, the SSH client continues login, and stores the host key locally in the file ~/.ssh/known_hosts. You only need to validate the host key the first time around: in subsequent logins, you will not be prompted to confirm it again.

Yet, from time to time, when you try to remote login to the same host from the same origin, you may be refused with the following warning message:

$ ssh erik@192.168.0.100
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
3f:1b:f4:bd:c5:aa:c1:1f:bf:4e:2e:cf:53:fa:d8:59.
Please contact your system administrator.
Add correct host key in /home/erik/.ssh/known_hosts to get rid of this message.
Offending key in /home/erik/.ssh/known_hosts:3
RSA host key for 192.168.0.100 has changed and you have requested strict checking.
Host key verification failed.

There are multiple possible reasons why the remote host key changed. A Man-in-the-Middle attack is only one possible reason. Other possible reasons include:

OpenSSH was re-installed on the remote host but, for whatever reason, the original host key was not restored.
The remote host was replaced legitimately by another machine.

If you are sure that this is harmless, you can use either 1 of 2 methods below to trick openSSH to let you login. But be warned that you have become vulnerable to man-in-the-middle attacks.

The first method is to remove the remote host from the ~/.ssh/known_hosts file. Note that the warning message already tells you the line number in the known_hosts file that corresponds to the target remote host. The offending line in the above example is line 3(“Offending key in /home/erik/.ssh/known_hosts:3″)

You can use the following one liner to remove that one line (line 3) from the file.

$ sed -i 3d ~/.ssh/known_hosts

Note that with the above method, you will be prompted to confirm the host key fingerprint when you run ssh to login.

The second method uses two openSSH parameters:
StrictHostKeyCheckin, and UserKnownHostsFile.

This method tricks SSH by configuring it to use an empty known_hosts file, and NOT to ask you to confirm the remote host identity key.

$ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no erik@192.168.0.100
Warning: Permanently added ’192.168.0.100′ (RSA) to the list of known hosts.
erik@192.168.0.100′s password:

The UserKnownHostsFile parameter specifies the database file to use for storing the user host keys (default is ~/.ssh/known_hosts).

The /dev/null file is a special system device file that discards anything and everything written to it, and when used as the input file, returns End Of File immediately.

By configuring the null device file as the host key database, SSH is fooled into thinking that the SSH client has never connected to any SSH server before, and so will never run into a mismatched host key.

The parameter StrictHostKeyChecking specifies if SSH will automatically add new host keys to the host key database file. By setting it to no, the host key is automatically added, without user confirmation, for all first-time connection. Because of the null key database file, all connection is viewed as the first-time for any SSH server host. Therefore, the host key is automatically added to the host key database with no user confirmation. Writing the key to the /dev/null file discards the key and reports success.

Please refer to this excellent article about host keys and key checking.

By specifying the above 2 SSH options on the command line, you can bypass host key checking for that particular SSH login. If you want to bypass host key checking on a permanent basis, you need to specify those same options in the SSH configuration file.

You can edit the global SSH configuration file (/etc/ssh/ssh_config) if you want to make the changes permanent for all users.

If you want to target a particular user, modify the user-specific SSH configuration file (~/.ssh/config). The instructions below apply to both files.

Suppose you want to bypass key checking for a particular subnet (192.168.0.0/24).

Add the following lines to the beginning of the SSH configuration file.

Host 192.168.0.*
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null

Note that the configuration file should have a line like Host * followed by one or more parameter-value pairs. Host *means that it will match any host. Essentially, the parameters following Host * are the general defaults. Because the first matched value for each SSH parameter is used, you want to add the host-specific or subnet-specific parameters to the beginning of the file.

As a final word of caution, unless you know what you are doing, it is probably best to bypass key checking on a case by case basis, rather than making blanket permanent changes to the SSH configuration files.

Share on Facebook

The Linux Laptop Company sells professional grade Linux laptops with full hardware support under all flavors of Linux. Our clients range from worldwide corporations to everyday personal Linux desktop users. We are familiar with all of the latest technology and our engineers can customize every aspect of your system to meet all of your most demanding mobile computing needs. Full manufacturer warranties are provided by Dell®. We stand by all of our products and guarantee support for the life of your computer.

How do I know this to be true? Because I own the company. I have over sixteen years of experience in a wide range of Linux based organizations. As a result, I’ve hand picked a team of Linux experts to manage this company. Most have been using Linux from its very earliest days (Slackware in 1993). Besides being skilled in every aspect of system administration and provisioning, our staff knows exactly what our customers want. The reason for this, is we are all Linux users ourself. After realizing how difficult it was to find a decent customizable Linux notebook, we decided to make our own. Our Linux wizards are at your service before and after the sale.

Why is everyone switching to Linux? While the list of reasons can go on for pages, in two words… “Open Source”. Open source is publicly shared intellectual property. When it comes to Linux, brilliant minds from across the world contribute to your the operating system’s source code and does not have licensing restrictions that limit use, modification, or redistribution. Linux is the most popular example of an open source technology. This innovative way of using a computer is far superior to commercialized operating systems because of the Linux community. Until now, this type of freedom has only been available to the technically advanced.

Share on Facebook

You can change the default priority that an application runs with by starting it with the nice command, but if you want to change the priority of a process that is already running, the command to use is renice.

Renice can be used to change the priority of a single process, or of all the processes owned by a specified user. As with the nice command, the priority values range from -20 to +19 and negative numbers raise the priority of a task while positive numbers lower it. Yes, that seems a little backwards, but the higher the priority the less resources are devoted to it. Only the superuser can specify negative numbers (thus raising the priority of a process).

renice 5 some_process

This command will change the priority of some_process to 5.

renice -5 -u erik

will change the priority of all processes owned by user erik to -5.

renice -5 -u erik -p 699

will chance the priority of all processes owned by erik and process with PID 699 to -5.

Share on Facebook

This one’s here because I spent the better half of an hour troubleshooting this for a customer.

The error:

Fatal error: Out of memory (allocated 47448064) (tried to allocate 13472326 bytes) in
/usr/local/cpanel/3rdparty/lib/php/Net/SMTP.php on line 821

The cause:

Sending messages through Webmail with attachments larger than 10MB.

Incorrect diagnosis:

If you’ve stumbled upon this post as a result of furious internet searching for similar issues, you’ve likely found thousands of posts regarding the error indicating the problem lies in the php.ini file of cPanel or Horde. While the PHP memory limit for cPanel and Horde are defined in their own php.ini, this does not actually alleviate the error seen here. cPanel itself will actually kill off applications consuming more than what is defined in Tweak Settings.

The Fix:

WHM (as root) ->Tweak Settings > The maximum memory a cPanel process can use before it is killed off.

Change this to 512M or 0 to disable memory limits all together.

Done.

Share on Facebook

So, often I’ll come across something that I need done, but don’t want to waste time manually doing it. This particular instance, I had a list of hostnames which I needed to remove the trailing “.” (dot) from the PTR. Normally, if these hostnames were all simple name.TLD domains this would easily be accomplished with cut:

cut -d. -f1-2

This basically means cut by delimiter (-d), where the delimiter is defined as a period (.) and then print fields 1 and 2. Easy enough.

But, what if you have hostnames and subdomains with varying field lengths, such as: something.domain.com. and something.else.domain.com.

So — here’s two easy ways to get it done.

USING VI:
This one’s for you vi fans (and once again pains me to learn another way that vi is more capable than nano/pico):

:%s/.$//g

Run that in vi, and wha-la, all the trailing dots disappear. You can obviously do this for other characters that are not “dots” by replacing the . in the command above.

But wait, there’s still more ways to skin the cat dot.

USING SED:

sed. x=`echo $i|sed s/.$//`

Either way, this leaves you with an easy way to save yourself boatloads of time (depending on how long the list you’re trying to trim is!)

Hope this helps!

-E

Share on Facebook

For those of you who are unaware of the difference between a Megabyte (MB) and a Megabit (Mb), here’s a quick 101:

In my line of work, I often hear people confuse the two or even think that they’re the same.

There are 8 bits that go to make up a byte, so 10Mb != 10MB.

10MB/s actually = 80Mb/s just as 800Mb/s = 100MB/s

If a particular switch or port is transferring 3.2MB per second, you’re actually moving 25.6Mbit per second.

Quite a difference, indeed.

Share on Facebook

Warning: send(Net/SMTP.php) [function.send]: failed to open stream: No such file or directory in /usr/local/lib/php/Mail/smtp.php on line 206
Warning: send() [function.include]: Failed opening ‘Net/SMTP.php’ for inclusion (include_path=’/usr/local/cpanel/base/horde/lib:.:/usr/lib/php:/usr/local/lib/php’) in /usr/local/lib/php/Mail/smtp.php on line 206

Annoying, yeah?

# pear install Net_SMTP

voilà!

Share on Facebook

So your IMAP mail is failing. Your e-mail client disconnects with an error related to bad authentication or simply “connection closed by remote server”. Check your mail logs, and you find:

May 25 17:52:43 vps imapd: Failed to create cache file: maildirwatch (someone@somewhere.com)
May 25 17:52:43 vps imapd: Error: Input/output error
May 25 17:52:43 vps imapd: Check for proper operation and configuration
May 25 17:52:43 vps imapd: of the File Access Monitor daemon (famd).

Found this obscure error in the system logs for IMAP. The server does not run “famd”. Make any sense? Not really. Though verbose, the output is indicative of nothing related to famd, I/O, or the cache file.

I first restarted the courier-auth daemon, which alleviated the failed login issue (seen as authentication failed or connection closed on most IMAP clients).

Second, edit the file:

/usr/lib/courier-imap/etc/imapd

and make sure IMAP_USELOCKS and IMAP_ENHANCEDIDLE are both set to 0, after that restart courier-imap:

/etc/init.d/courier-imap restart

Done, and done.

Share on Facebook

I came across this strange issue from a Tier II escalation today. A Virtuozzo based virtual server had a problem with the apache web server refusing to start:

# service httpd restart
[Sat May 15 16:41:13 2010] [warn] NameVirtualHost x.x.x.x:80 has no VirtualHosts
httpd not running, trying to start

Lets see if it’s actually started:

# service httpd status
Looking up localhost
Making HTTP connection to localhost
Alert!: Unable to connect to remote host.

Lets figure out what’s going on… Here’s some information on the kernel and OS:

-bash-3.00# uname -a
Linux xxxx 2.6.9-023stab046.2-enterprise #1 SMP Mon Dec 10 15:22:33 MSK 2007 i686 i686 i386 GNU/Linux

Normal troubleshooting ensued from there, and you can use this as a basis for determining what’s actually going on with your own server.

1. ALWAYS check the Apache error logs

Take a look at the error logs (usually “/usr/local/apache/logs”) and see if you can find what’s causing the problem.

In this case, the error_log gave me some valuable information and a place to start.

[Sat May 15 17:03:19 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Sat May 15 17:03:19 2010] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
Configuration Failed

On some server environments, you may also see a similar error that says:

[emerg] (28)No space left on device: Couldn’t create accept lock

Seems pretty obvious, yeah? Not so much….

2. Check available disk space.

-bash-3.00# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vzfs 77G 21G 57G 27% /

So we have plenty of disk space. Why can’t apache create the lockfile, then? Next step, (especially on virtual environments):

3. Check your available inodes

-bash-3.00# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vzfs 489K 167K 323K 34% /

**Scratches Head** So, the filesystem and available inodes are fine too. Now this gets interesting. The problem is that apache didn’t shut down properly, and it’s left myriads of semaphore-arrays left, owned by the apache-user (nobody).

A semaphore is a programming concept that is frequently used to solve multi-threading problems. Think of semaphores as bouncers at a nightclub. There are a dedicated number of people that are allowed in the club at once. If the club is full no one is allowed to enter, but as soon as one person leaves another person might enter.

To see if this is your problem, run:

ipcs -s | grep nobody

If you see a “wall” of these stragglers listed, your problem is solved. Removing these semaphores immediately should solve the problem and allow apache to start.

To do this, simply execute this command:

ipcs -s | grep nobody | perl -e 'while () { @a=split(/\s+/); print `ipcrm sem $a[1]`}'


You will see all of them being removed sequentially, and you can now go ahead and start up your apache service successfully.

PROTIP: Hitting the reset switch is NEVER the solution.

Share on Facebook

“Open Source Tripwire® software is a security and data integrity tool useful for monitoring and alerting on specific file change(s) on a range of systems. The project is based on code originally contributed by Tripwire, Inc. in 2000.”

A crude yet effective intrusion detection system such as Tripwire can alert systems administrators to possible intrusion attempts by periodically verifying the integrity of a server’s file systems. Systems intruders will often use trojan binaries for login, su, ps, and ls, etc. to cover their tracks and keep a low profile on the system. Under normal circumstances even astute systems administrators may not observe the intrusion because the trojan binaries mimic the system binaries so well.

This tool is a MUST have for all security conscious Linux System administrators. For step by step instructions on how to implement this on your server, you can [read entire article here].

Thanks to Linux Security, The central voice for Linux and Open Source security news for this very helpful information. Additionally, Yunliang Yu from Duke University has written a HOWTO on implementing this across an entire network. You can read his article here.

Share on Facebook