when apache simply won’t start, check the semaphores!

I came across this strange issue from a Tier II escalation today. A Virtuozzo based virtual server had a problem with the apache web server refusing to start:

# service httpd restart
[Sat May 15 16:41:13 2010] [warn] NameVirtualHost x.x.x.x:80 has no VirtualHosts
httpd not running, trying to start

Lets see if it’s actually started:

# service httpd status
Looking up localhost
Making HTTP connection to localhost
Alert!: Unable to connect to remote host.

🙁

Lets figure out what’s going on… Here’s some information on the kernel and OS:

-bash-3.00# uname -a
Linux xxxx 2.6.9-023stab046.2-enterprise #1 SMP Mon Dec 10 15:22:33 MSK 2007 i686 i686 i386 GNU/Linux

Normal troubleshooting ensued from there, and you can use this as a basis for determining what’s actually going on with your own server.

1. ALWAYS check the Apache error logs

Take a look at the error logs (usually “/usr/local/apache/logs”) and see if you can find what’s causing the problem.

In this case, the error_log gave me some valuable information and a place to start.

[Sat May 15 17:03:19 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Sat May 15 17:03:19 2010] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
Configuration Failed

On some server environments, you may also see a similar error that says:

[emerg] (28)No space left on device: Couldn’t create accept lock

Seems pretty obvious, yeah? Not so much….

2. Check available disk space.

-bash-3.00# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vzfs 77G 21G 57G 27% /

So we have plenty of disk space. Why can’t apache create the lockfile, then? Next step, (especially on virtual environments):

3. Check your available inodes

-bash-3.00# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vzfs 489K 167K 323K 34% /

**Scratches Head** So, the filesystem and available inodes are fine too. Now this gets interesting. The problem is that apache didn’t shut down properly, and it’s left myriads of semaphore-arrays left, owned by the apache-user (nobody).

A semaphore is a programming concept that is frequently used to solve multi-threading problems. Think of semaphores as bouncers at a nightclub. There are a dedicated number of people that are allowed in the club at once. If the club is full no one is allowed to enter, but as soon as one person leaves another person might enter.

To see if this is your problem, run:

ipcs -s | grep nobody

If you see a “wall” of these stragglers listed, your problem is solved. Removing these semaphores immediately should solve the problem and allow apache to start.

To do this, simply execute this command:

ipcs -s | grep nobody | perl -e 'while () { @a=split(/\s+/); print ipcrm sem $a[1]}'


You will see all of them being removed sequentially, and you can now go ahead and start up your apache service successfully.

PROTIP: Hitting the reset switch is NEVER the solution.

Share on Facebook

Tweet