Re: Limiting the number of small objects

On Tue, 21 Mar 2000, Bert Driehuis wrote:

> My Squid proxies have too much disk space. The Squid process runs out
> of memory well before the referenceAge kicks in. With disks so cheap
> and memory so expensive, this is a situation that many administrators
> find themselves in.
>
> I wrote the attached diff just to see if it is workable to scan the
> object store at regular intervals to see if small objects can be ditched,
> freeing up memory, while still allowing the referenceAge mechanism to
> get rid of the bigger objects. In polygraph testing it seems to
> work. Where the Squid process would run out of memory it now keeps on
> running (I don't have byte hit rates yet).
>
> I'm posting here and now to get feedback on the conceptual pros and cons.
> The diff is relative to Squid 2.2STABLE5 with Henrik's patches up to
> hno.20000103.
>
> The max_small_object_target puts a cap on the amount of objects kept.
> With max_small_object_target set to 100,000, the overhead in in-core
> store entries is limited to about 10MB. Object larger than
> small_object_max_size are kept until the referenceAge algorithm
> gets rid of them. The algorithm kicks in once per
> minute (the search is relatively expensive compared to the referenceAge
> based algorithm, due to having to skip the entries that are bigger
> than small_object_max_size). Over time, the cache will fill with a
> limited number of big objects filling up the cache, with as much smaller
> objects cached as the overhead can bear.
>
> Doing this can help a lot in the corporate environment, where downstream
> caches are connected through a slow link to a corporate firewall, and
> caching large stuff (like Netscape downloads, virus checker updates
> etcetera) can make a big difference in network responsiveness.
>
> Ideas? Have I just reinvented a wheel? Has it been tried before and found
> to be stupid? I must admit I haven't looked much at the current Squid; I
> needed a solution I can roll out at pretty short notice and didn't want to
> deviate too much from the rock solid 2.2.STABLE5-hno.20000103 I'm running
> right now.

I didn't look at the code yet, but my first reaction is that
this belongs as a replacement policy. An extra scanning event
may be expensive. Wouldn't it be better to have a single policy
that favors large files over small ones, and combines the size
with other parameters such as age, refcount?

Duane W.
Received on Mon Mar 20 2000 - 20:47:55 MST