Re: Performance tweaks for large caches

> At Fri, 14 Jun 1996 01:43:11 +1000, Mark Treacy writes:
>>> A summary of the changes follows
>>> Our ~9G cache holds 500,000 objects. With only 100 directories, this
>>> means that each directory holds 5,000 objects. After a week or 2 the
>>> directories develop holes and we had directories 100k in size. This
>>> makes pathname lookups rather expensive, particularly if the activity
>>> hits many such sized directories (flushing various kernel caches).
>>> I have changed the swap area to a 2 level directory heirarchy. The changes
>>> here construct 16 first level directories and 256 second level directories
>>> for a total of 4096 cache directories. This reduces the size of all the
>>> directories to be less than the typical logic block size (8k).
>
> That's a fairly productive change, although we haven't really noticed
> it, partly because our cache systems are configured across more
> filesystems on more spindles to maximise seek bandwidth. That doesn't
> currently help Squid (they were specced for CERN :), but hopefully it
The platform we have switched to are ultra's running solaris. The 12G
filesystem is a concatenated filesystem across several disk drives.
The host also has prestoserve.

> will shortly once I update my asynchronous IO patch for beta17 and
> send it in. (Although Squid tries to use non-blocking IO, we haven't
> found a Unix that actually implements that for disk IO, with the
> result that all disk IO is sequential and causes a bottleneck with
> large numbers of clients --- these patches allow Squid to actually
> kick off multiple IO requests in parallel on a suitable system (eg
> Irix)).
>
>>> Background Restore takes too long if there are no events occuring
>>> (50 objects read once a second), added a flag to cause squid to block
>>> until the restore is complete.
>
> We did notice that with a big cache restores take too long, but
> haven't hada chance to look into it. Our goal is to minimise
> downtime, so blocking until restore is complete is unacceptable. I'll
Likewise, I run a standyby squid and HUP it into action when required.
Because I restart squid this way, it makes sense to have the foreground
flag as the server isn't expected to be doing anything else.

> have a look at the code later and see if we can come up with a reload
> algorithm that basically soaks up all the spare capacity on the
> machine to reload as fast as possible whilst trying not to impact
> active connections.
The pathname lookups, unlink's, and stat's slow down the restore.
You could do the same as what I gather cached 2.0 has done, push
the file operations out to another process. This decouples the restore
rate from the unlink rate, but you have to be careful about when the fileno
can be reused. As far as the stat's go that are as a result of a unclean
shutdown, off the top of my head I can't think how you'd decouple that from
the restore internal process.

 - Mark.
Received on Thu Jun 13 1996 - 10:35:32 MDT