Re: [squid-users] Is it possible to ignore parts of or all parts of a query string?

Hi,

A bit old this mail, but since it is still a big issue for me I try to
continue the thread.

On Mon, Sep 13, 2004 at 02:31:38PM +0200, Henrik Nordstrom wrote:
>>1095077099.840 25 195.159.125.115 TCP_MISS/200 15910 GET
>>http://ap.oasfile.aftenposten.no/ap/rosaindex/768x50_aftenposten.swf?clickTAG1=http%3A//primetime.ad.primetime.net/RealMedia/ads/click_lx.ads/www.aftenposten.no/forside/5278/Top1/primetime/ap_rosaindex_768x50_til_des04/rosaindex_2xclicktag.html/63333966376437333430336565663130%3F_RM_REDIR_%3Dhttp%3A//www.rosaindex.no/%3Frsat%3D3&clickTAG2=http%3A//primetime.ad.primetime.net/RealMedia/ads/click_lx.ads/www.aftenposten.no/forside/5278/Top1/primetime/ap_rosaindex_768x50_til_des04/rosaindex_2xclicktag.html/63333966376437333430336565663130%3Fhttp%3A//www.rosaindex.no/%3Fsaction%3DSOK%26fdsc%3DrsCompany%26rsat%3D2
>>- DIRECT/127.0.0.1 application/x-shockwave-flash
>>
>>The actual file is just served statically from a local thttpd that has
>>all the files.
>>
>>I suppose the Shockwave "program" (the swf file) uses the query
>>parameters as a runtime configuration for what to do etc., so if only
>>Squid could pass this on and still cache the file contents, that would
>>be exactly what I want.
>>
>>Anyone have ideas about this? Is it doable? Running the swf's through
>>Squid works, it's just that every file that has query parameters is not
>>cached. :-(
> If you know it is a static file and not dynamically generated you can use
> a redirector helper to strip of the query while Squid forwards the
> request. The query parameters is still available to the plugin, only not
> to the requested server.
>
> The problem is knowing..

I am 100% sure it is a static file. I am hosting all the files from a
local webserver, running Squid in accelerator mode fetching the files off
the local webserver.

I did manage to make Squid cache the files using the Squirm redirector
plugin (http://squirm.foote.com.au/) - yay. One small step for Squid,
one giant leap for Anders. ;-) However, I'm facing a volume of at least
250 hits per second per server, where most (80% or so) is .swf URLS that
needs rewriting/redirect. And I want to be able to handle a lot more
files than that, the reason for forcing Squid to cache the files in the
first place is for performance. So it seems Squirm is not fast enough
for me, it makes Squid go down rapidly. From /var/log/messages:

Jan 12 09:47:12 oasfile (squid): Too many queued redirector requests
(5243 on 15)
Jan 12 09:47:46 oasfile (squid): Too many queued redirector requests
(4943 on 15)
Jan 12 09:48:20 oasfile (squid): Too many queued redirector requests
(5083 on 25)
Jan 12 09:48:54 oasfile (squid): Too many queued redirector requests
(5151 on 25)

And from Squid's cache.log I get a lot of these:

2005/01/12 09:48:54| WARNING: All redirector processes are busy.
2005/01/12 09:48:54| WARNING: up to 5176 pending requests queued
FATAL: Too many queued redirector requests (5151 on 25)
Squid Cache (Version 2.5.STABLE7): Terminated abnormally.
CPU Usage: 2.200 seconds = 1.394 user + 0.805 sys
Maximum Resident Size: 54072 KB
Page faults with physical i/o: 8
2005/01/12 09:48:57| Starting Squid Cache version 2.5.STABLE7 for
i386-portbld-freebsd4.10...

I tried using 5, 10, 15 and 25 redirect_children. No luck changing those
values. How many URLs per seconds can one expect Squirm to be able to
handle, I wonder.

Because of this I'm considering to not use Squid for caching these files
at all. The application vendor says they do not recommend using
non-random URLs for the files, which effectively prevents Squid from
caching them easily. Sigh. Maybe we will try to rewrite the URLs on our
Nortel Alteon load balancer.But hey, it may be the right thing to not use
Squid for this purpose anyway, and drop the recirection.

PS: My OS of choice for Squid is FreeBSD 4.10-RELEASE.

Cheers,

-- 
Anders.