Re: Performance tweaks for large caches

Henrik Nordstrom wrote:
>> the administrator to select appropriate ttl's based on the arrival rate
>> and available space such that an lru gc is rarely, if ever, run.
>
> The LRU algorithm will probably be replaced in 1.1, with a more effective
> one that can continously purge objects without any (i.e. very small)
> performance loss.
>
> The slow scan will probably be removed, when IMS is used to refresh
> expired objects (this requires that the LRU replacement is redesigned).
Fixing the slow scan avoids the use of the lru gc (provided you size
the cache appropriately). This is a major win on a big cache.
The squid cache I am refering to is serving more than 300,000 requests per
day (>4G). Arrivals into the cache amount to around 2G per day.
Without these changes squid did't really cope.

>> Our ~9G cache holds 500,000 objects. With only 100 directories, this
> The idea is that the larger cache, the more cache_dir entries
> you should use (each cache_dir contains 100 directories).=20

I briefly considered this. 25 cache_dirs for the current hw configuration.
50+ for the next.

>
>> Background Restore takes too long if there are no events occuring
>> (50 objects read once a second), added a flag to cause squid to block
>> until the restore is complete.
>
> Not quite true. It restores 50 objects, then handles any pending
> network data, restores 50 more objects and so on (no delay). The
> more concurrent activity, the slower it reloads.
Duane corrcectly suggested I was refering to the 1 second delay introduced
in comm_select() when there was no select activity.

> Slow rebuild (after a crash) is a bit trickier, since it requires
> a stat() on each restored file, and therby makes a lot of disk I/O.
> Because of this, the default "speed" for slow rebuilds is 5 objects.
The specific use I had for the -F option I added was after rearranging
the cache and rewritting the log file (for a deeper directory heirarchy).
I wanted everything stat'd, but I didn't want to wait 27 hours for the
consistency check to finish.

 - Mark.
Received on Thu Jun 13 1996 - 17:12:19 MDT