Re: [squid-users] FATAL: Too many queued redirector requests

On 09/06/11 20:16, Le Trung Kien wrote:
> Hi, I don't know how to handle memory for squid. Squid automatically
> breaks down itself and restarts:

http://wiki.squid-cache.org/SquidFaq/SquidMemory

>
> In my squid.conf I set:
>
> url_rewrite_children 120 startup=30 idle=100 concurrency=0

You need 3.2 for the dynamic startup capabilities. 3.1 only uses the
first value (120 helpers to start *immediately*).

Given that is actually a reasonable number of helpers I think the memory
consumption elsewhere is probably killing them.

> max_filedesc 7168
> cache_swap_low 60
> cache_swap_high 80

If you are forced to do that with swap values it means you cache
management is badly calculated.

There is no use allocating for example 100GB of disk storage when
cache_swap_low forces 40% of it not to be used. And Squid pausing
occasionally while 20% of it (80%-60%) is erased can also cause visible
slowdown to clients.

The memory FAQ page linked above outlines how to estimate cache sizes.

>
> I notice that just one pending requests queued:
>
> 2011/06/09 11:00:17| WARNING: All redirector processes are busy.
> 2011/06/09 11:00:17| WARNING: 1 pending requests queued
>
> and follow is more details of cache.log:
>
> 2011/06/09 11:00:16| WARNING: Cannot run
> '/opt/squid-3.1.10/urlrewriter_new.pl' process.
> 2011/06/09 11:00:16| ipcCreate: fork: (12) Cannot allocate memory

> 2011/06/09 11:00:16| WARNING: Cannot run
> '/opt/squid-3.1.10/urlrewriter_new.pl' process.
> 2011/06/09 11:00:16| ipcCreate: fork: (12) Cannot allocate memory
> 2011/06/09 11:00:16| WARNING: Cannot run
> '/opt/squid-3.1.10/urlrewriter_new.pl' process.
> 2011/06/09 11:00:16| ipcCreate: fork: (12) Cannot allocate memory
> 2011/06/09 11:00:16| WARNING: Cannot run
> '/opt/squid-3.1.10/urlrewriter_new.pl' process.
> 2011/06/09 11:00:17| WARNING: All redirector processes are busy.
> 2011/06/09 11:00:17| WARNING: 1 pending requests queued
> 2011/06/09 11:00:17| storeDirWriteCleanLogs: Starting...
> 2011/06/09 11:00:17| WARNING: Closing open FD 177
> 2011/06/09 11:00:17| Finished. Wrote 16367 entries.
> 2011/06/09 11:00:17| Took 0.01 seconds (1308941.14 entries/sec).
> FATAL: Too many queued redirector requests
> Squid Cache (Version 3.1.10): Terminated abnormally.

Your machine seems not to be capable of running 120 or these rewriter
processes. You need more available to them RAM or less maximum helpers.

> <---------------------------------- CRASH ***
> CPU Usage: 6631.678 seconds = 4839.266 user + 1792.412 sys
> Maximum Resident Size: 0 KB
> Page faults with physical i/o: 3299

"Page faults" - your system is swapping badly to get that many just on
startup. If this continues it will kill Squid performance later even if
it gets past the helpers.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.12
   Beta testers wanted for 3.2.0.8 and 3.1.12.2