squid hanging after several days running ... follow up ...

Neat thing is that doing a simple 'kill 297' appears to trigger a clean
shutdown ... but the store rebuild fails with:

Feb 23 13:15:25 demeter squid[15317]: Store rebuilding is 1.0% complete
Feb 23 13:15:28 demeter squid[15317]: WARNING: newer swaplog entry for dirno 1, fileno 00005ABB
Feb 23 13:15:28 demeter squid[15317]: assertion failed: store_dir_ufs.c:644: "rb->flags.need_to_validate"

Eventually, it appears to start, but after alot of those 'newer swaplog
entry' messages ...

Oh, squid-head is from Feb 12th ...

On Fri, 23 Feb 2001, Marc G. Fournier wrote:

>
> at ~10am this morning, squid hung solid ... after running since Saturday:
>
> demeter# ps aux | grep squid
> nobody 297 99.0 40.8 427112 426848 ?? R Sat04PM 520:12.96 squid -NsY
> root 290 0.0 0.0 660 76 con- I Sat04PM 0:00.00 /bin/sh /usr/local/squid/bin/RunCache
>
> I have a cron job that runs every hour, on the hour, to rotate the squid
> logs via 'squid -k rotate', which is what it looks like 'triggers' the
> hang, as the last write to /var/log/messages is at 9:59:
>
> tail -f /var/log/messages
> Feb 23 09:00:08 demeter squid[297]: urlParse: Illegal character in hostname 'proxy.acadiau.ca:3128proxy.acadiau.ca'
> Feb 23 09:00:44 demeter last message repeated 6 times
> Feb 23 09:02:45 demeter last message repeated 20 times
> Feb 23 09:12:46 demeter last message repeated 96 times
> Feb 23 09:22:51 demeter last message repeated 190 times
> Feb 23 09:32:55 demeter last message repeated 192 times
> Feb 23 09:43:01 demeter last message repeated 202 times
> Feb 23 09:53:05 demeter last message repeated 198 times
> Feb 23 09:59:57 demeter last message repeated 137 times
> Feb 23 13:05:30 demeter /kernel: pid 15267 (gcore), uid 0 on /: file system full
>
> using gcore to get a core of the runing image, and gdb to see inside of
> it, not much appears to be visible?
>
> demeter# gdb /usr/local/squid/bin/squid core.297
> GNU gdb 4.18
> Copyright 1998 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for details.
> This GDB was configured as "i386-unknown-freebsd"...
> Core was generated by `squid'.
> Reading symbols from /usr/lib/libcrypt.so.2...done.
> Reading symbols from /usr/lib/libm.so.2...done.
> Reading symbols from /usr/lib/libc_r.so.4...done.
> Reading symbols from /usr/libexec/ld-elf.so.1...done.
> #0 0x4819aecc in _thread_sys_poll () from /usr/lib/libc_r.so.4
> (gdb) where
> #0 0x4819aecc in _thread_sys_poll () from /usr/lib/libc_r.so.4
> #1 0x4819a1f8 in _thread_kern_scheduler () from /usr/lib/libc_r.so.4
> #2 0x48199c35 in _thread_kern_scheduler () from /usr/lib/libc_r.so.4
> #3 0x0 in ?? ()
> (gdb) quit
>
> I build squid with:
>
> #!/bin/tcsh
> setenv CFLAGS "-g"
> ./configure \
> --prefix=/usr/local/squid \
> --enable-useragent-log \
> --enable-referer-log \
> --enable-snmp \
> --enable-async-io \
> --enable-stacktraces \
> --enable-storeio=ufs,aufs,diskd
>
> and the machine is reasonably up to date ...
>
> demeter# uname -a
> FreeBSD demeter.acadiau.ca 4.2-STABLE FreeBSD 4.2-STABLE #2: Wed Feb 14 11:16:35 AST 2001 root@demeter.acadiau.ca:/usr/obj/usr/src/sys/demeter i386
>
>
> Marc G. Fournier marc.fournier@acadiau.ca
> Senior Systems Administrator Acadia University
>
> "These are my opinions, which are not necessarily shared by my employer"
>
>

Marc G. Fournier marc.fournier@acadiau.ca
Senior Systems Administrator Acadia University

  "These are my opinions, which are not necessarily shared by my employer"
Received on Fri Feb 23 2001 - 10:22:16 MST