Re: [squid-users] Mixing delay pools and slow aborts

Mike Crowe wrote:
> I'm using Squid (Debian Lenny stable 3.0.STABLE8-3) as a mechanism for
> pre-caching large downloads for a number of client hosts that are
> powered down during the night so that the files they require are ready
> for them to download in the morning. In the late afternoon each of the
> clients connects to Squid, starts downloading each file and then
> disconnects as soon as the data starts flowing. I set "quick_abort_min
> -1" to ensure that Squid continues with the download regardless.
>
> I now need to limit the bandwidth used by Squid when caching the
> files. I initially experimented with Linux traffic control but found
> the server just stopped sending packets on some of the connections
> after a while due to not making any progress. Fiddling with timeouts
> didn't seem to be fully effective. Squid's built in delay pool system
> worked much better and didn't result in any dropped connections even
> when concurrently downloading a hundred large files at 100kbit/s. Even
> the fact that more bandwidth is available during certain hours could
> easily be handled using "acl time"[1].
>
> Here's the interesting parts of my current configuration:
>
> acl nighttime time 00:00-05:00
> delay_pools 2
> delay_class 1 1
> delay_class 2 1
> delay_access 1 allow nighttime
> delay_access 1 deny !nighttime
> delay_access 2 allow !nighttime
> delay_access 2 deny nighttime
> delay_parameters 1 120000/120000
> delay_parameters 2 12000/12000
>
> But, as the FAQ states, once the client disconnects there is no longer
> any link back to the delay pool and the download proceeds at full
> speed. :(
>
> I've been trying to come up with workarounds for this so that I can
> keep both the bandwidth shaping behaviour and slow abort.
>
> My understanding of the Squid code is minimal but looking at
> MemObject::mostBytesAllowed() I wonder whether it might be possible
> for MemObject to store the last DelayId so that when the clients list
> is empty it has something to fall back on? This may be ineffective if
> all clients have disconnected before the first read but perhaps that
> can be fixed by setting the persistent DelayId in
> MemObject::addClient() too.

This may be suitable for your needs, but is not optimal elsewhere. Why
should the last visiting client be penalized for an admin configuration
choice?
Their followup pool will then be reduced by the bandwidth used during
the no longer needed request for the duration of the finishing-up pull.

>
> Alternatively I've wondered whether I could write a redirector or
> external ACL helper which effectively ran wget through the proxy for
> every URL received taking care not to duplicate URLs that it has
> submitted. I believe that this solution could also easily be extended
> to support retrying interrupted downloads too which would be a bonus.

You could. Noting that client requests which test it will hang until the
ACL returns, so its probably better to scan the log periodically and
re-fetch.

>
> Does anyone with greater knowledge than me have any comments on these
> proposals or any better ideas?
>
> Thanks.
>
> Mike.
>
> [1] Although in my testing it appeared that the bandwidth did not
> change at the prescribed time for a download that was already in
> progress.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE6 or 3.0.STABLE15
   Current Beta Squid 3.1.0.8 or 3.0.STABLE16-RC1