Re: [squid-users] Too many queued redirector requests

Hi,

> I have been following this thread of mails and I
> have a problem with the number of redirectors too.
> I am using squid-2.5.STABLE5 configured to work
> with caching disabled.
Just curious: Any reason to disable caching?

> I am using a redirector that
> does some url filtering using a local database.

What is the purpose? Filtering offending content?
Selfmade redirector? What kind of database?

> I have
> timed the redirector - it takes anywhere between 42
> msecs to 2000 msecs at times to process. I am running
> squid on a system with the configuration -
> Linux-2.4.18-14, P4, 512 MB RAM. We have other
> applications like a web server, etc that are also
> running on this system.

2000msecs? Thats a lot...

> You had mentioned in ur earlier mail that ur
> system is so configured that 80% of the requests are
> handled by the first redirector, 10% by the second and
> so on. Could you kindly elaborate as to how this
> done - or is it the way squid works? If this is the
> case, then adding more redirectors shd not solve my
> problem.

The number of requests handled by each redirector as described in the
posts is more a phenomenon, not a configuration issue.
- Squidguard is very fast and IMHO the request are not distributed round
robin, but to the first idle redirector
- light load - the first redirector is always ready to serve the
request, all the others have never a chance to answer.
- increased load - sometime the first redirector is busy, some requests
are answered by the 2nd redirector
- even higher load - sometime both 1st and 2nd redirectors are busy,
the 3th gets a chance to do some work - and so on with the others if
load increases.
- load peak - all redirectors get something to do, some request are
queued and the warning message occurs, but never a FATAL error as the
queue lenght never gets high (long?) enough.

I think your problem is different as your redirectors are quite slow,
the queues are growing to long which causes the fatal error.
Increasing the number of redirectors should help a bit in your case, but
  probably the database is to slow to handle all the requests. If the
database is the problem, increasing the no of redirector means
increasing the processing time as well - this means the number of
request which can be processed in a given amount of time will not
increase much.

> I tried conducting some simple load tests with
> squid using the redirectors.
> 1 redirector worked fine for 5 simultaneous browser
> clients(that is w/o throwing a FATAL error and
> restarting),
> 2 redirectors worked fine for 14 browser clients
> but subsequent tests showed that even with 5
> redirector clients,20 browser clients cud not be
> handled simultaneously. I don't want to enable
> redirctor bypass though.
> I am failing to understand this behaviour. I would
> be thankful if u cud spare some time to explain what
> cud be happening here and tell me a solution for it.
> Regards and TIA,
> Deepa
>
>
> --- Hendrik_Voigtländer <hendrik@voigtlaenders.net>
> wrote: > Hi,
>
>>E250 with how many processor of what type?
>>Probably you have an performance problem with the
>>sleepycat berkeley-db.
>>If all your processors are busy all the time
>>increasing the number of
>>redirectors won't help.
>>
>>In this case I would switch over to Linux with an
>>Intel machine. We have
>>replaced our old E450 with an HP/Compaq ML370
>>(decent machine, but not
>>high end) with a significant improve in squid
>>perfomance. I gave up on
>>compiling squidguard on solaris, to much hassle and
>>to much load
>>probably as well for the 168MHz(!) processors.
>>With debian no problem at all, 'apt-get install
>>squidguard' and done...
>>
>>I really like the idea of using multiple cheap
>>machine with
>>loadbalancing and failover, but IMHO you need to use
>>automatic proxy
>>configuration to achieve this. I would use server
>>hardware for this, but
>>something cheaper than HP/Compaq, for instance
>>Supermicro.
>>
>>Get cachemgr.cgi running, it is really useful to
>>look at squid &
>>squidguards status.
>>
>># TAG: redirector_bypass
>># When this is 'on', a request will not go
>>through the
>># redirector if all redirectors are busy. If
>>this is 'off'
>># and the redirector queue grows too large,
>>Squid will exit
>># with a FATAL error and ask you to increase
>>the number of
>># redirectors. You should only enable this if
>>the redirectors
>># are not critical to your caching system. If
>>you use
>># redirectors for access control, and you
>>enable this option,
>># then users may have access to pages that
>>they should not
>># be allowed to request.
>>#
>>redirector_bypass on
>>
>>As you may have noticed it is impossible to filter
>>100% of all unwanted
>>stuff, bypassing in high load situations won't make
>>things much worse.
>>
>>Keep an eye on the redirector stats in cachemgr how
>>many requests are
>>actually bypassing squidguard. In our setup it is
>>less than 1%.
>>
>>Regards, Hendrik.
>>
>>
>>Merid Tilahun wrote:
>>
>>>Thanx Hendrik
>>>I am running squid on solaris 8, sun enterprise
>>
>>250
>>
>>>machine. I have more that 500 users connect at
>>
>>peak
>>
>>>hour.
>>>I never got around to configure cachemanager.cgi,
>>
>>I
>>
>>>will look in to that.
>>>I use squidguard to filter porn, and it seems to
>>
>>be
>>
>>>working but it is affecting the servicetime badly.
>>
>>I
>>
>>>run around 20 redirector processes, I have been
>>>increasing constantly.
>>>I deactivated squidguard for a while and I got not
>>>messages, but I need squidGuard to block the porn.
>>>What is redirector bypass and how do i enable it?
>>>
>>>--- Hendrik Voigtlaender
>>
>><hendrik@voigtlaenders.net>
>>
>>>wrote:
>>>
>>>
>>>>Hi,
>>>>
>>>>I just checked my logs:
>>>>We get 40...50req/sec for about 10 hours a day
>>
>>with
>>
>>>>nearly no traffic
>>>>during the night, this levels out to a daily
>>
>>average
>>
>>>>of 1000req/min.
>>>>Number of users are reported with about
>>
>>1000...1500,
>>
>>>>but I guess that we
>>>>have never more than 150 clients connect at the
>>
>>same
>>
>>>>time.
>>>>We have 15 redirectors running, average redirector
>>>>service time is not
>>>>measurable, servicetimes are good. In cachemgr.cgi
>>>>you can check how
>>>>many request are handled by each redirector.
>>>>In our setup, more than 80% of the requests are
>>>>handled by the first
>>>>one. The second gets about 10%, all the other are
>>>>used only during peak
>>>>periods. Redirectors #15 is idle nearly the whole
>>>>time.
>>>>In top only a a few (2...3) squidguards are
>>
>>showing
>>
>>>>enough activity to
>>>>be noticed, but not much. Machine is Intel 2,4GHz,
>>>>2x36GB SCSI
>>>>cache_dir, 2GB RAM, Linux debian woody, squid &
>>>>squidguard out of distro.
>>>>
>>>>I played around with the number of redirectors but
>>>>never got rid of this
>>>>message (all busy, increase blabla). I think it is
>>>>not critical if this
>>>>warning appears on and off, but not to often. 15
>>
>>is
>>
>>>>IMHO more than
>>>>sufficient with this load.
>>>>
>>>>What are you using squidguard for? If you try to
>>>>block pr0n, you can as
>>>>well enable redirector bypass. The blacklist will
>>>>never catch all sites
>>>>anyway, it doesnt matter if a couple of more
>>
>>request
>>
>>>>bypass squidguard.
>>>>This will eliminate the warning for sure.
>>>>
>>>>How many redirectors are you using ? I would not
>>>>install squidguard but
>>>>probably deativate it in squid.conf for testing.
>>>>I can imagine that too many redirectors are eating
>>>>up the ressources
>>>>badly needed by squid, especially on a slow/small
>>>>machine.
>>>>Can you post some specs?
>>>>
>>>>Regards, Hendrik Voigtländer
>>>>
>>>>Merid Tilahun wrote:
>>>>
>>>>
>>>>
>>>>>Hi all:
>>>>>I have istalled and configured squid and it was
>>>>>working
>>>>>fine until now. I installed squidGuard and when
>>>>
>>>>squid
>>>>
>>>>
>>>>>starts working with it I get the user warning
>>>>
>>>>messages
>>>>
>>>>
>>>>>Too many queued redirector requests. I have
>>>>
>>>>increased
>>>>
>>>>
>>>>>the number of redirector processes but I am still
>>>>>getting this messages. Plus the retrieval time
>>
>>for
>>
>>>>>sites is too long. Does any one know a way out?
>>
>>or
>>
>>>>>shall I just
>>>>>uninstall squidguard?
>>>>>My squid get around 850 req/min ( got this from
>>>>
>>>>MRTG).
>>>>
>>>>
>>>>>Thank you in advance
>>>>>Regards
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>__________________________________
>>>>>Do you Yahoo!?
>>>>>Friends. Fun. Try the all-new Yahoo! Messenger.
>>>>>http://messenger.yahoo.com/
>>>>>
>>>>>
>>>>
> === message truncated ===
>
> ________________________________________________________________________
> Yahoo! India Matrimony: Find your partner online. http://yahoo.shaadi.com/india-matrimony/
Received on Mon May 31 2004 - 05:02:38 MDT