Re: Squid memory footprint (was: MemPools rewrite)

Andres Kroonmaa wrote:
>
> On 2 Nov 2000, at 16:09, Adrian Chadd <adrian@creative.net.au> wrote:
>
> > On Thu, Nov 02, 2000, Andres Kroonmaa wrote:
> > >
> > > I wonder how can you eliminate StoreEntry? IMHO it contains crucial
> > > information that allows squid to skip disk accesses. Moving parts
> > > of this data into squidfs doesn't seem to change much in ram usage.
> > > Moving this crucial information onto disks implies enormous performance
> > > penalty, doesn't it?
> >
> > You are assuming the FS can't handle object reference / lock counts and
> > freshness information itself. :-)
>
> I think I'm assuming that the FS needs to keep freshness data, and
> key->diskobject translations in ram, along with overhead for maintaining
> replacement policys (refcount, locks).
> Sure we can reduce amount of pointers per each object in store db, but
> unless we accept disk access penalty, we need to keep lots stuff per
> object in ram.

I guess it hasn't been made clear that there is actually an
implementation that _does_ drop all of the per object database out of
memory entirely. The reiser_raw/butterfly storeio interface does not
keep anything in memory. It seeks for every request. And the
performance is the best of any Squid to date (it's what we used at the
cacheoff). It can probably be improved by the keeping of a cache_digest
style hash hit/miss db in memory, but even now it performs admirably
well, and recovery from a powerfail is nearly instant.

> I'm simply assuming that we move most of this stuff from StoreEntry db
> into FS metadata.
> Perhaps we can optimise out quite alot along the way, I hope.

For a traditional FS, obviously this idea poses problems. But seeks are
quite cheap in ReiserFS, so it doesn't really cause a big performance
hit, I don't think. Of course, until there is a hashed in core db, we
won't know how much the seeks are costing us.
                                  --
                     Joe Cooper <joe@swelltech.com>
                 Affordable Web Caching Proxy Appliances
                        http://www.swelltech.com
Received on Thu Nov 02 2000 - 04:04:49 MST