Re: to RAID, or not to RAID?

Jon,

Obviously, the answer to this "Depends on the RAID level, and what you
want out of it" :-)

I've been told that when we tried using RAID, that we saw no substantial
performance benefits to using RAID 0 just to create a large, striped file
system. (Our group tried using RAID before I joined it) Furthermore with
RAID 0, IMO, you could have some operational problems though. (See below)

If you look at the way squid stores files in a large system - effectively
scattering them in a fairly random manner across all the disks. (as far
as an external observer would view it anyway :-)

The net result is that when the disks are full and the server is under
load, if you have 6 requests coming in simultaneously, and all 6 are hits,
then if you have 6 disks, then there's a chance that all 6 are on
different disks. Obviously in practice the higher the load the more you
see the effects of this, since 6 for 6 would be incredibly lucky :-)

As a result you're unlikely to see any useful effect under when the cache
is under heavy load. That said, you mileage may well vary - they're may be
a good performance boost if the proxy is under low load - ie only a few
concurrent cache hits/cacheable misses per second.

Increasing the number of physical disks gives you a number of benefits -
the SCSI bus stays more active, each disk has it's own caches, thereby
adding extra boosts to the whole system.

Another datapoint is to consider what happens when the whole subsystem is
under continuous heavy load. If sustained for long enough, the whole
system can blow up in your face, - eg a file overrunning due to broken
servers. eg There's an icecast server in eastern europe - I forget the
precise URL :-( , that can happily get a squid to try and cache gigantic
files when the squid is under heavy load.

Also squid + the OS will crash from time to time, again normally on
heavily/overloaded servers. If the machine crashes at the right instant,
this can cause major problems that even an fsck might not be able to
correctly fix - and if it's all one big disk, you lose the entire cache
contents and/or suffer a very long down time...

Disks can and do fail, SCSI subsytems can have errors, problems will
occur. Proxy caching a hefty request stream can take it's toll on the
disks you have, and you should ask yourself what happens to my squid when
a disk dies irretrievably? With RAID 0, you lose the entire file system.

If you use multiple cache_dir lines, your squid will stop functioning
properly until you comment out the appropriate cache_dir line and restart
squid. (An option for squid to stop using the disk and complain bitterly -
eg fire off an arbitrary command - would be useful here...)

You can then fix the disk at your leisure. Incidentally if your ~50GB
cache is made up of (say) 12 * 4.2Gb disks, and a normal fsck fails,
requiring one of the more hefty fscks to be done on just one of the disks,
then what we find to be the quickest way to bring that disk back into
service is:

   * Edit it out of the squid.conf
   * Restart squid.
   * keep a tar file of an empty (and as far as squid is concerned, clean)
     cache disk handy.
   * reformat the disk - normally far faster than a major fsck.
   * untar the cache directories.
   * add the entry back into the squid.conf.
   * Restart squid.

With lots of small disks, this affects only a small proportion of your
cache. With a few big disks, there's less of an incentive to do this since
you'd lose a much greater proportion of your data.

All the above said, if your 50 Gb cache won't be under very heavy load,
then RAID 1, 4 or 5 may well be a very useful option, by reducing most of
the operation problems above.

Michael.

--
National & Local Web Cache Support        R: G117
Manchester Computing                      T: 0161 275 7195
University of Manchester                  F: 0161 275 6040
Manchester UK M13 9PL                     M: Michael.Sparks@wwwcache.ja.net