Performance tweaks for large caches

I would like to submit a series of patches for inclusion in the squid
baseline public release.
The company I work for operates a relatively large cache (so it would seem).
In the past couple of weeks, as we have transitioned from a 4G CERN cache
to a 12G squid cache, we have found that squid in some areas does not cope
well with a cache of this size. We are waiting on new hardware to progress
to a 24G cache, so these changes have been done keeping in mind that it
had to scale at least that far.
I have prepared some patches, which although I have made no attempt to tidy
up, accomplish some of the necessary performance improvements we needed.
These patches have primarily been tested on a squid 1.0 beta 11 production
server over the last week+, minimal testing has been performed with squid
1.0 beta 16 in a non production environment (testing on a recent release has
been somewhat problematic due to it being a moving target of late).
I have made the patches available as
        ftp://ftp.aone.net.au/pub/squid/squid-b16.aone.patch.gz

A summary of the changes follows
store.c:
The slow scan routine, storeMaintainSpace(), according to the comments
around the code, should scan a bucket every 20 seconds. The delay between
scans however varies between 20 seconds and 20 minutes, depending upon load.
I have made a slight change such that a bucket is scanned every 10 seconds
based on time rather than number of calls to the routine, a minor change in
main.c is associated with this change. Scanning a bucket every 10 seconds
means that the entire cache is scanned for expired objects once a
day (~20 hours). Because the cache is constantly being cleaned this enables
the administrator to select appropriate ttl's based on the arrival rate
and available space such that an lru gc is rarely, if ever, run.

Our ~9G cache holds 500,000 objects. With only 100 directories, this
means that each directory holds 5,000 objects. After a week or 2 the
directories develop holes and we had directories 100k in size. This
makes pathname lookups rather expensive, particularly if the activity
hits many such sized directories (flushing various kernel caches).
I have changed the swap area to a 2 level directory heirarchy. The changes
here construct 16 first level directories and 256 second level directories
for a total of 4096 cache directories. This reduces the size of all the
directories to be less than the typical logic block size (8k). A side benefit
of this is that it triggers the bsd inode allocation policy into achieving
a more even distribution across the available cylinder groups.
Update squid_curtime after an lru gc in storeGetSwapSpace as it may
have taken more than a second to run (influenced by factors such as the
huge directories described above). This stops storeGetSwapSpace restarting
within the same second it last finished.
Background Restore takes too long if there are no events occuring
(50 objects read once a second), added a flag to cause squid to block
until the restore is complete.
A number of unimportant of debuging changes.

main.c:
Flag for foreground restore.
Reduce max loop timeout to 10 seconds.

squid.h:
foreground flag.

dnsserver.c:
disable search and default domain appending (both of which generate
lots of needless dns traffic).

I also have some scripts for updating a cache to the new structure.
I am not sure whether I will have time to package them up in the next
12 hours.

I hope these changes are of use to others. I will be away from mail until
22 June. Perhaps some other squid users are interested in getting together
for a chat in Montreal about squid etc.

 - Mark.
Received on Thu Jun 13 1996 - 08:43:52 MDT