I came across this strange issue from a Tier II escalation today. A Virtuozzo based virtual server had a problem with the apache web server refusing to start:
# service httpd restart
[Sat May 15 16:41:13 2010] [warn] NameVirtualHost x.x.x.x:80 has no VirtualHosts
httpd not running, trying to start
Lets see if it’s actually started:
# service httpd status
Looking up localhost
Making HTTP connection to localhost
Alert!: Unable to connect to remote host.
🙁
Lets figure out what’s going on… Here’s some information on the kernel and OS:
-bash-3.00# uname -a
Linux xxxx 2.6.9-023stab046.2-enterprise #1 SMP Mon Dec 10 15:22:33 MSK 2007 i686 i686 i386 GNU/Linux
Normal troubleshooting ensued from there, and you can use this as a basis for determining what’s actually going on with your own server.
1. ALWAYS check the Apache error logs
Take a look at the error logs (usually “/usr/local/apache/logs”) and see if you can find what’s causing the problem.
In this case, the error_log gave me some valuable information and a place to start.
[Sat May 15 17:03:19 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Sat May 15 17:03:19 2010] [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
Configuration Failed
On some server environments, you may also see a similar error that says:
[emerg] (28)No space left on device: Couldn’t create accept lock
Seems pretty obvious, yeah? Not so much….
2. Check available disk space.
-bash-3.00# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vzfs 77G 21G 57G 27% /
So we have plenty of disk space. Why can’t apache create the lockfile, then? Next step, (especially on virtual environments):
3. Check your available inodes
-bash-3.00# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vzfs 489K 167K 323K 34% /
**Scratches Head** So, the filesystem and available inodes are fine too. Now this gets interesting. The problem is that apache didn’t shut down properly, and it’s left myriads of semaphore-arrays left, owned by the apache-user (nobody).
A semaphore is a programming concept that is frequently used to solve multi-threading problems. Think of semaphores as bouncers at a nightclub. There are a dedicated number of people that are allowed in the club at once. If the club is full no one is allowed to enter, but as soon as one person leaves another person might enter.
To see if this is your problem, run:
ipcs -s | grep nobody
If you see a “wall” of these stragglers listed, your problem is solved. Removing these semaphores immediately should solve the problem and allow apache to start.
To do this, simply execute this command:
ipcs -s | grep nobody | perl -e 'while () { @a=split(/\s+/); print ipcrm sem $a[1]}'
You will see all of them being removed sequentially, and you can now go ahead and start up your apache service successfully.
Other Symptoms
Apache hangs with semget: No space left on device error message in /usr/local/apache/logs/error_log
About every system semaphore is used. Apache 1.3.27 can leave them used if killed.
Close these semaphores.
# ipcs -s
If there is lots of lines owned by “nobody” it is apache.
# service httpd stop
# /usr/bin/ipcrm sem $(/usr/bin/ipcs -s | grep nobody | awk '{print$2}')
# service httpd start
Semaphore issues in the user’s error_log that look similar to:
PHP Warning: PHP Startup: mm_create(0, /session_mm_cgi-fcgi1005) failed, err mm:core: failed to open semaphore file (Permission denied) …
Fix:
As of 5.2.10 it seems that the semaphore files are being created in /
instead of the usual /tmp. Uncommenting the default session.save_path of
/tmp in the php.ini fixes it.
Reproduce code:
—————
$ php –version
PHP Warning: PHP Startup: mm_create(0, /session_mm_cgi-fcgi1005)
failed, err mm:core: failed to open semaphore file (Permission denied)
in Unknown on line 0
PHP 5.2.10 (cgi-fcgi) (built: Aug 28 2009 10:23:30)
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2009 Zend Technologies
Leave a Reply