[squid-users] squid tries to rebuild cache but fails...

Hi,

I have here a Netra-machine with Solaris 8 and 1GB RAM. This machine is
serving as mailhost/mailscan-machine and Webproxy. Today,squid stopped
because it claimed not to have enough storage space to store and it tried
to rebuild the store database. But even when there is enough storage space
and inodes available, it still crashes:

[..]
root@host# tail -f /opt/sfw/squid/var/logs/cache.log

2004/04/27 12:30:59| Starting Squid Cache version 2.5.STABLE2-20030411 for
sparc-sun-solaris2.9...
2004/04/27 12:30:59| Process ID 11178
2004/04/27 12:30:59| With 1024 file descriptors available
2004/04/27 12:30:59| DNS Socket created at 0.0.0.0, port 59790, FD 5
2004/04/27 12:30:59| Adding nameserver 212.121.128.10 from /etc/resolv.conf
2004/04/27 12:30:59| Adding nameserver 212.121.128.11 from /etc/resolv.conf
2004/04/27 12:30:59| Unlinkd pipe opened on FD 10
2004/04/27 12:30:59| Swap maxSize 18432000 KB, estimated 1417846 objects
2004/04/27 12:30:59| Target number of buckets: 70892
2004/04/27 12:30:59| Using 131072 Store buckets
2004/04/27 12:30:59| Max Mem size: 131072 KB
2004/04/27 12:30:59| Max Swap size: 18432000 KB
2004/04/27 12:30:59| Store logging disabled
2004/04/27 12:30:59| Rebuilding storage in /cache (DIRTY)
2004/04/27 12:30:59| Using Least Load store dir selection
2004/04/27 12:30:59| Set Current Directory to /opt/sfw/squid/var/logs
2004/04/27 12:30:59| Loaded Icons.
2004/04/27 12:30:59| Accepting HTTP connections at 0.0.0.0, port 8080, FD
11.
2004/04/27 12:30:59| Accepting ICP messages at 0.0.0.0, port 3130, FD 12.
2004/04/27 12:30:59| WCCP Disabled.
2004/04/27 12:30:59| Ready to serve requests.
2004/04/27 12:31:00| Store rebuilding is 0.2% complete
2004/04/27 12:31:15| Store rebuilding is 17.4% complete
2004/04/27 12:31:32| Store rebuilding is 22.0% complete
2004/04/27 12:31:48| Store rebuilding is 23.2% complete
2004/04/27 12:32:03| Store rebuilding is 38.1% complete
Apr 27 12:32:16 boe-stvw-p2 squid[11178]: [ID 702911 local4.alert] Write
failure -- check your disk space and cache.log
2004/04/27 12:32:16| diskHandleWrite: FD 8: disk write error: (28) No space
left on device
FATAL: Write failure -- check your disk space and cache.log
Squid Cache (Version 2.5.STABLE2-20030411): Terminated abnormally.
CPU Usage: 38.110 seconds = 20.550 user + 17.560 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 0
2004/04/27 12:32:19| Starting Squid Cache version 2.5.STABLE2-20030411 for
sparc-sun-solaris2.9...
2004/04/27 12:32:19| Process ID 11194
2004/04/27 12:32:19| With 1024 file descriptors available
2004/04/27 12:32:19| DNS Socket created at 0.0.0.0, port 59791, FD 5
2004/04/27 12:32:19| Adding nameserver 212.121.128.10 from /etc/resolv.conf
2004/04/27 12:32:19| Adding nameserver 212.121.128.11 from /etc/resolv.conf
2004/04/27 12:32:19| Unlinkd pipe opened on FD 10
2004/04/27 12:32:19| Swap maxSize 18432000 KB, estimated 1417846 objects
2004/04/27 12:32:19| Target number of buckets: 70892
2004/04/27 12:32:19| Using 131072 Store buckets
2004/04/27 12:32:19| Max Mem size: 131072 KB
2004/04/27 12:32:19| Max Swap size: 18432000 KB
2004/04/27 12:32:19| Store logging disabled
2004/04/27 12:32:19| Rebuilding storage in /cache (DIRTY)
2004/04/27 12:32:19| Using Least Load store dir selection
2004/04/27 12:32:19| Set Current Directory to /opt/sfw/squid/var/logs
2004/04/27 12:32:19| Loaded Icons.
2004/04/27 12:32:19| Accepting HTTP connections at 0.0.0.0, port 8080, FD
11.
2004/04/27 12:32:19| Accepting ICP messages at 0.0.0.0, port 3130, FD 12.
2004/04/27 12:32:19| WCCP Disabled.
2004/04/27 12:32:19| Ready to serve requests.
2004/04/27 12:32:20| Store rebuilding is 0.2% complete
2004/04/27 12:32:35| Store rebuilding is 19.1% complete
2004/04/27 12:32:39| Preparing for shutdown after 109 requests
2004/04/27 12:32:39| Waiting 30 seconds for active connections to finish
2004/04/27 12:32:39| FD 11 Closing HTTP connection
2004/04/27 12:32:50| Store rebuilding is 36.2% complete
Apr 27 12:32:56 boe-stvw-p2 squid[11194]: [ID 702911 local4.alert] Write
failure -- check your disk space and cache.log
2004/04/27 12:32:56| diskHandleWrite: FD 8: disk write error: (28) No space
left on device
FATAL: Write failure -- check your disk space and cache.log
Squid Cache (Version 2.5.STABLE2-20030411): Terminated abnormally.
CPU Usage: 33.610 seconds = 18.620 user + 14.990 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 0
2004/04/27 12:33:27| Starting Squid Cache version 2.5.STABLE2-20030411 for
sparc-sun-solaris2.9...
2004/04/27 12:33:27| Process ID 11239
2004/04/27 12:33:27| With 1024 file descriptors available
2004/04/27 12:33:27| DNS Socket created at 0.0.0.0, port 59792, FD 5
2004/04/27 12:33:27| Adding nameserver 212.121.128.10 from /etc/resolv.conf
2004/04/27 12:33:27| Adding nameserver 212.121.128.11 from /etc/resolv.conf
2004/04/27 12:33:27| Unlinkd pipe opened on FD 10
2004/04/27 12:33:27| Swap maxSize 18432000 KB, estimated 1417846 objects
2004/04/27 12:33:27| Target number of buckets: 70892
2004/04/27 12:33:27| Using 131072 Store buckets
2004/04/27 12:33:27| Max Mem size: 131072 KB
2004/04/27 12:33:27| Max Swap size: 18432000 KB
2004/04/27 12:33:27| Store logging disabled
2004/04/27 12:33:27| Rebuilding storage in /cache (DIRTY)
2004/04/27 12:33:27| Using Least Load store dir selection
2004/04/27 12:33:27| Set Current Directory to /opt/sfw/squid/var/logs
2004/04/27 12:33:27| Loaded Icons.
2004/04/27 12:33:27| Accepting HTTP connections at 0.0.0.0, port 8080, FD
11.
2004/04/27 12:33:27| Accepting ICP messages at 0.0.0.0, port 3130, FD 12.
2004/04/27 12:33:27| WCCP Disabled.
[..]

as you can see, it tries to restart everytime.

[..]
root@host# uname -a
SunOS boe-stvw-p2 5.8 Generic_108528-27 sun4u sparc SUNW,UltraAX-i2

root@host# df -o i
df: operation not applicable for FSType fd
df: operation not applicable for FSType mntfs
df: operation not applicable for FSType nfs
df: operation not applicable for FSType proc
df: operation not applicable for FSType tmpfs
Filesystem iused ifree %iused Mounted on
/dev/md/dsk/d10 4568 367144 1% /
/dev/md/dsk/d30 45367 466633 9% /usr
/dev/md/dsk/d40 19357 460643 4% /var
/dev/md/dsk/d60 1592287 935713 63% /cache
/dev/md/dsk/d50 25493 318571 7% /opt

root@host# df -k
Filesystem kbytes used avail capacity Mounted on
/dev/md/dsk/d10 1487071 1421933 5656 100% /
/dev/md/dsk/d30 4130982 1009143 3080530 25% /usr
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
/dev/md/dsk/d40 3916782 476513 3401102 13% /var
swap 6841480 24 6841456 1% /var/run
swap 6856384 14928 6841456 1% /tmp
/dev/md/dsk/d60 20587657 16174440 4207341 80% /cache
/dev/md/dsk/d50 2055463 1147450 846350 58% /opt
[..]

as you can see, there is nearly no space anymore on /. This is already
fixed (more than 160Mbyte now free), but squid still crashes at the same
moment (store rebuilding).

I have here the squid.conf with all comments and emtpy lines taken out:

[..]
root@host# grep -v ^# squid.conf|more
http_port 8080
hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
no_cache deny QUERY
cache_mem 128 MB
cache_swap_low 90
cache_swap_high 95
maximum_object_size 8192 KB
minimum_object_size 0 KB
maximum_object_size_in_memory 32 KB
ipcache_size 2048
ipcache_low 90
ipcache_high 95
fqdncache_size 2048
cache_replacement_policy lru
memory_replacement_policy lru
cache_dir ufs /cache 18000 32 256
cache_access_log /opt/sfw/squid/var/logs/access.log
cache_access_log /opt/sfw/squid/var/logs/access.log
cache_log /opt/sfw/squid/var/logs/cache.log
cache_store_log none
pid_filename /opt/sfw/squid/var/logs/squid.pid
debug_options ALL,1
auth_param basic children 5
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern . 60 40% 43200
quick_abort_pct 95
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow all
http_access deny all
http_reply_access allow all
icp_access allow all
logfile_rotate 10
coredump_dir /opt/sfw/squid/var/logs
ie_refresh on
[..]

I first thought that squid tries to write somewhere on the root-Filesystem
and there is not enough space. Unfortunately, I get the same error even
when I freed some space.

Can you give me some hints where to check for enough space or to see what
really is wrong here?

Mit freundlichen Grüßen,

Hanno Wagner

--
Hanno Wagner -  Unix-C/S-Systemadministrator
Boerse Stuttgart AG
Email: Hanno.Wagner@boerse-stuttgart.de Tel: +49-711-222 985-569
Diese E-Mail enthält vertrauliche oder rechtlich geschützte Informationen.
Wenn Sie nicht der beabsichtigte Empfänger sind, informieren Sie bitte
sofort den Absender und löschen Sie diese E-Mail. Das unbefugte Kopieren
dieser E-Mail oder die unbefugte Weitergabe der enthaltenen Informationen
ist nicht gestattet.
The information contained in this message is confidential or protected by
law. If you are not the intended recipient, please contact the sender and
delete this message. Any unauthorised copying of this message or
unauthorised distribution of the information contained herein is
prohibited.