squid hangs after several days running ...

at ~10am this morning, squid hung solid ... after running since Saturday:

demeter# ps aux | grep squid
nobody 297 99.0 40.8 427112 426848 ?? R Sat04PM 520:12.96 squid -NsY
root 290 0.0 0.0 660 76 con- I Sat04PM 0:00.00 /bin/sh /usr/local/squid/bin/RunCache

I have a cron job that runs every hour, on the hour, to rotate the squid
logs via 'squid -k rotate', which is what it looks like 'triggers' the
hang, as the last write to /var/log/messages is at 9:59:

tail -f /var/log/messages
Feb 23 09:00:08 demeter squid[297]: urlParse: Illegal character in hostname 'proxy.acadiau.ca:3128proxy.acadiau.ca'
Feb 23 09:00:44 demeter last message repeated 6 times
Feb 23 09:02:45 demeter last message repeated 20 times
Feb 23 09:12:46 demeter last message repeated 96 times
Feb 23 09:22:51 demeter last message repeated 190 times
Feb 23 09:32:55 demeter last message repeated 192 times
Feb 23 09:43:01 demeter last message repeated 202 times
Feb 23 09:53:05 demeter last message repeated 198 times
Feb 23 09:59:57 demeter last message repeated 137 times
Feb 23 13:05:30 demeter /kernel: pid 15267 (gcore), uid 0 on /: file system full

using gcore to get a core of the runing image, and gdb to see inside of
it, not much appears to be visible?

demeter# gdb /usr/local/squid/bin/squid core.297
GNU gdb 4.18
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...
Core was generated by `squid'.
Reading symbols from /usr/lib/libcrypt.so.2...done.
Reading symbols from /usr/lib/libm.so.2...done.
Reading symbols from /usr/lib/libc_r.so.4...done.
Reading symbols from /usr/libexec/ld-elf.so.1...done.
#0 0x4819aecc in _thread_sys_poll () from /usr/lib/libc_r.so.4
(gdb) where
#0 0x4819aecc in _thread_sys_poll () from /usr/lib/libc_r.so.4
#1 0x4819a1f8 in _thread_kern_scheduler () from /usr/lib/libc_r.so.4
#2 0x48199c35 in _thread_kern_scheduler () from /usr/lib/libc_r.so.4
#3 0x0 in ?? ()
(gdb) quit

I build squid with:

#!/bin/tcsh
setenv CFLAGS "-g"
./configure \
        --prefix=/usr/local/squid \
        --enable-useragent-log \
        --enable-referer-log \
        --enable-snmp \
        --enable-async-io \
        --enable-stacktraces \
        --enable-storeio=ufs,aufs,diskd

and the machine is reasonably up to date ...

demeter# uname -a
FreeBSD demeter.acadiau.ca 4.2-STABLE FreeBSD 4.2-STABLE #2: Wed Feb 14 11:16:35 AST 2001 root@demeter.acadiau.ca:/usr/obj/usr/src/sys/demeter i386

Marc G. Fournier marc.fournier@acadiau.ca
Senior Systems Administrator Acadia University

  "These are my opinions, which are not necessarily shared by my employer"
Received on Fri Feb 23 2001 - 10:20:34 MST