Re: Squid memory footprint (was: MemPools rewrite)

Andres Kroonmaa wrote:

> I think you are underestimating importance of ICP. Yes it has lots of
> problems, yet its the best we have so far if we need to implement a
> tightly coupled (in terms of cache content) cache cluster.

I don't think I am. ICP (or something similar) is very important for a
tightly coupled cluster, but for such cache couplings it is quite
acceptable to have lossy information. It does not matter terribly much
if the false hit/miss ratio in the peering is quite high (0.5% or so),
as long as the speed can be met.

> > With a design like the ReiserFS-raw/butterfly we cannot generate a
> > digest in a practical manner simply due to the fact that doing so would
> > require reading in each and every object in the cache..
>
> So we still have to keep swapstate.

If you want to generate digest for exchange as cache digests yes.

> > What is possible is to have the filesystem generate a digest-like hint
> > map to speed up availability lookups.
>
> what would it possibly look like? besides digests I mean.

Probably quite a lot like a digest, but based on other keys and size
criterias, and with deletes to avoid having to regenerate the digest
every now and then. The purpose is solely to indicate that an object is
most likely not in the cache. And since this digest is used internally a
less dense one can be afforded to make deletes work well without loosing
too much information.

> Indeed. MD5 and some criteria to determine staleness state are the only
> things needed. All else can be pushed to disks and fetched only for
> pending http requests. Even not full MD5 is needed, we can get away
> with a subset of MD5 key, for eg. 4-8 bytes instead of 16bytes would
> give reasonably high correctness.

staleness is only required for ICP, and even then it can probably be
afforded to have some disk access for staleness calculation, as long as
the wast majority of misses are filtered out first.

So what gets left in memory (besides the FS cache) is the above
mentioned digest, using at most a couple of bytes per object.

> That would be very nice. As Alex said we can use incremental digests.
> Or did you mean something else?

Incremental digest does not really make sense here as the digest are
never exchanged. I was more thinking of a gradually decaying digest.

Objects gets added to the digest when they enter the cache, and deleted
from the digest by clearing some bits when they are evicted from the
cache. To make sure bits never get stuck in the digest there is a
function which makes sure that no bit can stay on for ever. A simple
slow scan clearing bits should be sufficient for this.

/Henrik
Received on Mon Nov 06 2000 - 14:19:30 MST