[squid-users] Issues with squid-2.6.18 on FreeBSD/COSS/ZFS

Dear all,

I'm running several instances of squid at my university CAN and have
recently discovered some annoying things about the cache, so I kindly
ask for a little help on this.

Every time the squid process is cleanly stopped and started again, it
finds the cache dirty and starts to rebuild the COSS storage, which
adds heavy load to the hard drive:

squid2# squid -k shutdown
squid2# tail -f /var/log/squid/cache.log
...
2008/03/27 11:16:40| COSS: /squid/coss: syncing
2008/03/27 11:16:40| Squid Cache (Version 2.6.STABLE16): Exiting normally.

squid2# squid -Nd1
...
2008/03/27 11:17:38| Rebuilding storage in /squid/cache (DIRTY)
2008/03/27 11:17:38| Rebuilding COSS storage in /squid/coss (DIRTY)
...
2008/03/27 11:17:38| Done reading /squid/cache swaplog (255 entries)
2008/03/27 11:17:39| COSS: /squid/coss: Rebuilding (0 % completed -
1/8000 stripes)

after some time:

2008/03/27 11:18:06| storeSwapMetaUnpack: insane length (67305986)!
2008/03/27 11:18:07| storeSwapMetaUnpack: bad type (-1)!

after some more time:
2008/03/27 11:19:48| COSS: /squid/coss: Rebuild Completed

For about two minutes (in the given example - between 11:17 and 11:19)
the cache responds poorly to client requests due to heavy (near 100%)
/dev/da2 load (from gstat). During that time, of course, several
TCP_SWAPFAIL_MISS errors appear on the access.log. After the cache is
rebuilt, the COSS engine significantly reduces the load on the system,
when compared with simple ufs storage, so my preference would be to
use COSS together with ufs. Under current deployment the cache
receives an average of 160 requests/second with peaks of 500req/s.
When additional coss files are introduced to the squid config, the
period of cache rebuilding during startup is greatly extended.

The question is - have I misconfigured anything? If yes, then what
should I change to avoid the cache to be rebuilt at every startup? Any
help or comments will be appreciated.

squid 2.6.18 runs on FreeBSD 7.0-STABLE with the following additional
options in the kernel:

options SHMSEG=16
options SHMMNI=32
options SHMMAX=2097152
options SHMALL=4096
options MAXFILES=8192
options VFS_AIO

Cache configuration:

squid2# grep ^cache squid.conf
cache_mem 512 MB
cache_replacement_policy lru
cache_dir ufs /squid/cache 8000 16 256
cache_dir coss /squid/coss 8000 block-size=512 max-size=131072
cache_swap_log /squid/%s
cache_access_log /var/log/squid/access.log
cache_log /var/log/squid/cache.log
cache_store_log none
cache_effective_user squid

squid2# zfs list
NAME USED AVAIL REFER MOUNTPOINT
squid 8,01G 23,2G 8,01G /squid
squid2# ls -l /squid/
total 8393870
drwxr-xr-x 18 squid squid 18 Kov 27 10:12 cache
-rw-r--r-- 1 squid squid 8589934592 Kov 27 11:04 coss
-rw-r----- 1 squid squid 12688 Kov 27 11:08 squid.cache
-rw-r----- 1 squid squid 4064372 Kov 27 11:08 squid.coss

squid2# grep ^da2 /var/run/dmesg.boot
da2 at ahc0 bus 0 target 4 lun 0
da2: <SEAGATE ST336607LW DS04> Fixed Direct Access SCSI-3 device
da2: 40.000MB/s transfers (20.000MHz, offset 8, 16bit)
da2: Command Queueing Enabled
da2: 34732MB (71132959 512 byte sectors: 255H 63S/T 4427C)

-- 
 Simonas Kareiva