Re: ipcCreat: fork: (12) Not enough space (not faq8.8)

Grabmann Martin wrote:
>
> hello squid-users
>
> i'm running squid-2.2STABLE5 on HP-UX 10.20 compiled with gcc-2.8.1. it's
> running well.
>
> but now there is only one dnsserver-process running, although i configured
> five.
>
> if i reconfigure squid with squid -k reconfigure in the cache.log file i get
> the following error:
>
...
> 2000/04/14 11:33:09| helperOpenServers: Starting 5 'dnsserver' processes
> 2000/04/14 11:33:11| ipcCreate: fork: (12) Not enough space
> 2000/04/14 11:33:11| WARNING: Cannot run
> '/opt/squid/squid_22S5/bin/dnsserver' process.
> 2000/04/14 11:33:11| ipcCreate: fork: (12) Not enough space
> 2000/04/14 11:33:11| WARNING: Cannot run
> '/opt/squid/squid_22S5/bin/dnsserver' process.
> 2000/04/14 11:33:11| ipcCreate: fork: (12) Not enough space
> 2000/04/14 11:33:11| WARNING: Cannot run
> '/opt/squid/squid_22S5/bin/dnsserver' process.
> 2000/04/14 11:33:11| ipcCreate: fork: (12) Not enough space
> 2000/04/14 11:33:11| WARNING: Cannot run
> '/opt/squid/squid_22S5/bin/dnsserver' process.
> 2000/04/14 11:33:11| Accepting HTTP connections on port 8000, FD 2.
> 2000/04/14 11:33:11| Ready to serve requests.
>
> i have 256 MB of RAM and 512 MB of swap
...
> TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
> rroot 2099 squid 154 20 154M 148M sleep 297:04 7.73 7.71 (squid)
>
> nothing else is running on this machine, only squid.
>
> i read the faq, but faq 8.8 didn't solve the problem.
> i don't think i have to less memory, what is the problem here????
> please help!

We hit this problem with an earlier Squid version that was leaking memory
badly (so that the Squid process grew to several hundred MB over it's
"normal" size), but that was simply triggering the problem - the underlying
cause wasn't the memory leak.

The problem is that as each dnsserver process is created by fork, the parent
process address space is cloned in full, though it's cleaned up very soon
afterwards when the new process exec()s the much smaller dnsserver process.

All the dnsserver processes will be created over a very short period of
time, so that you can have several proto-dnsserver processes existing at the
same time which temporarily have memory allocations similar to the original
Squid process (though it's the private data space that matters, not e.g.
shared libraries, so the effective size will be less than the total shown by
ps or top).

With 5 processes configured, that means you may need up to 5 copies of 154MB
(minus whatever part of that is shared libraries etc.). That means you could
need up to 750MB (minus shared libraries etc.). On a system with only 768MB
total memory + swap, and a significant part of that already in use, that
does seem likely to cause trouble.

We tried adding an artificial delay between forking dnsserver processes, but
it didn't reliably help in our case - we needed 32 dnsserver processes (the
maximum), so couldn't afford long pauses (even 1 second would delay handling
user requests by half a minute), but 1 or 2 seconds didn't always avoid
problems. That was probably because our Squid's normal size was around 600MB
(on a system with 2GB memory plus some swap) and the problem happened when
it grew to 800-900MB due to memory leaks. I suspect that forking such a
large process pushed the system so close to the limit of available
memory+swap that getting the process started was especially slow - it must
have been taking over a second to start at least some of the dnsserver
processes or the problem would have been avoided.

It's possible that with only 5 dnsserver processes, adding a 1 or 2 second
delay would be sufficient (and the penalty in terms of delaying handling
user requests would be much smaller, 5-10 seconds). What it really needed
was a feedback channel of some sort, so Squid could wait for each process to
exec dnsserver before it forked the next one. The new internal DNS server in
2.3 avoids that problem entirely, of course.

                                John Line

-- 
University of Cambridge WWW manager account (usually John Line)
Send general WWW-related enquiries to webmaster@ucs.cam.ac.uk