Limiting the number of small objects

My Squid proxies have too much disk space. The Squid process runs out
of memory well before the referenceAge kicks in. With disks so cheap
and memory so expensive, this is a situation that many administrators
find themselves in.

I wrote the attached diff just to see if it is workable to scan the
object store at regular intervals to see if small objects can be ditched,
freeing up memory, while still allowing the referenceAge mechanism to
get rid of the bigger objects. In polygraph testing it seems to
work. Where the Squid process would run out of memory it now keeps on
running (I don't have byte hit rates yet).

I'm posting here and now to get feedback on the conceptual pros and cons.
The diff is relative to Squid 2.2STABLE5 with Henrik's patches up to
hno.20000103.

The max_small_object_target puts a cap on the amount of objects kept.
With max_small_object_target set to 100,000, the overhead in in-core
store entries is limited to about 10MB. Object larger than
small_object_max_size are kept until the referenceAge algorithm
gets rid of them. The algorithm kicks in once per
minute (the search is relatively expensive compared to the referenceAge
based algorithm, due to having to skip the entries that are bigger
than small_object_max_size). Over time, the cache will fill with a
limited number of big objects filling up the cache, with as much smaller
objects cached as the overhead can bear.

Doing this can help a lot in the corporate environment, where downstream
caches are connected through a slow link to a corporate firewall, and
caching large stuff (like Netscape downloads, virus checker updates
etcetera) can make a big difference in network responsiveness.

Ideas? Have I just reinvented a wheel? Has it been tried before and found
to be stupid? I must admit I haven't looked much at the current Squid; I
needed a solution I can roll out at pretty short notice and didn't want to
deviate too much from the rock solid 2.2.STABLE5-hno.20000103 I'm running
right now.

Cheers,

                                        -- Bert

Bert Driehuis, MIS -- bert_driehuis@nl.compuware.com -- +31-20-3116119
Every nonzero finite dimensional inner product space has an
orthonormal basis. It makes sense, when you don't think about it.