RE: [squid-users] Squid 3.0 stable20 crash

Coredump with debugging, more to come

warning: Can not parse XML syscalls information; XML support was disabled at compile time.
GNU gdb (Gentoo 7.0 p2) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.gentoo.org/>...
Reading symbols from /usr/sbin/squid...done.
[New Thread 27059]
[New Thread 27061]
[New Thread 27060]
[New Thread 27063]
[New Thread 27062]
[New Thread 27065]
[New Thread 27092]
[New Thread 27066]
[New Thread 27067]
[New Thread 27064]
[New Thread 27069]
[New Thread 27074]
[New Thread 27097]
[New Thread 27093]
[New Thread 27071]
[New Thread 27105]
[New Thread 27072]
[New Thread 27068]
[New Thread 27076]
[New Thread 27077]
[New Thread 27102]
[New Thread 27075]
[New Thread 27104]
[New Thread 27106]
[New Thread 27079]
[New Thread 27073]
[New Thread 27101]
[New Thread 27078]
[New Thread 27111]
[New Thread 27081]
[New Thread 27098]
[New Thread 27116]
[New Thread 27086]
[New Thread 27080]
[New Thread 27115]
[New Thread 27083]
[New Thread 27114]
[New Thread 27084]
[New Thread 27120]
[New Thread 27089]
[New Thread 27118]
[New Thread 27122]
[New Thread 27121]
[New Thread 27117]
[New Thread 27087]
[New Thread 27108]
[New Thread 27107]
[New Thread 27109]
[New Thread 27082]
[New Thread 27099]
[New Thread 27119]
[New Thread 27094]
[New Thread 27100]
[New Thread 27113]
[New Thread 27110]
[New Thread 27088]
[New Thread 27103]
[New Thread 27096]
[New Thread 27091]
[New Thread 27112]
[New Thread 27090]
[New Thread 27095]
[New Thread 27085]
[New Thread 27070]

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /usr/lib/libssl.so.0.9.8...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libssl.so.0.9.8
Reading symbols from /usr/lib/libcrypto.so.0.9.8...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libcrypto.so.0.9.8
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/libstdc++.so.6
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_dns.so.2
Core was generated by `/usr/sbin/squid -N -f /etc/squid/squid-fe-8082.conf'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000000004df490 in SplayNode<mem_node*>::start (this=Cannot access memory at address 0x7fffac383ff8
) at ../include/splay.h:123
123 ../include/splay.h: No such file or directory.
        in ../include/splay.h

(gdb) backtrace
#0 0x00000000004df490 in SplayNode<mem_node*>::start (this=Cannot access memory at address 0x7fffac383ff8
) at ../include/splay.h:123
#1 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6a9f0) at ../include/splay.h:126
#2 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6aa10) at ../include/splay.h:126
#3 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6aa30) at ../include/splay.h:126
#4 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6aa50) at ../include/splay.h:126
#5 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6aa70) at ../include/splay.h:126
#6 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6aa90) at ../include/splay.h:126
#7 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6aab0) at ../include/splay.h:126
#8 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6aad0) at ../include/splay.h:126
#9 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6aaf0) at ../include/splay.h:126
#10 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6ab10) at ../include/splay.h:126
#11 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6ab30) at ../include/splay.h:126
#12 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6ac50) at ../include/splay.h:126
#13 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6ac70) at ../include/splay.h:126
#14 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6ac90) at ../include/splay.h:126
#15 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6acb0) at ../include/splay.h:126
#16 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6acd0) at ../include/splay.h:126
#17 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6ac20) at ../include/splay.h:126
#18 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6ab50) at ../include/splay.h:126
#19 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6ab70) at ../include/splay.h:126
#20 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc6ab90) at ../include/splay.h:126
#21 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8bd00) at ../include/splay.h:126
#22 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8bd20) at ../include/splay.h:126
#23 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8bd40) at ../include/splay.h:126
#24 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8bd60) at ../include/splay.h:126
#25 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8bd80) at ../include/splay.h:126
#26 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8bda0) at ../include/splay.h:126
#27 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8bdc0) at ../include/splay.h:126
#28 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8bde0) at ../include/splay.h:126
#29 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8be00) at ../include/splay.h:126
#30 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8be20) at ../include/splay.h:126
#31 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8be40) at ../include/splay.h:126
#32 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8be60) at ../include/splay.h:126
#33 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8be80) at ../include/splay.h:126
#34 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8bea0) at ../include/splay.h:126
#35 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8bec0) at ../include/splay.h:126
#36 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8bee0) at ../include/splay.h:126
#37 0x00000000004df4b5 in SplayNode<mem_node*>::start (this=0x3fc8bf00) at ../include/splay.h:126

-----Original Message-----
From: Kingsley Foreman [mailto:kingsley_at_internode.com.au]
Sent: Friday, 29 January 2010 5:32 AM
To: Amos Jeffries; squid-users_at_squid-cache.org
Subject: RE: [squid-users] Squid 3.0 stable20 crash

I should ask, because I know someone will have a idea how to do it here rather then googling it

How do you add core dumping and debug symbols when building squid?

Kingsley

-----Original Message-----
From: Amos Jeffries [mailto:squid3_at_treenet.co.nz]
Sent: Friday, 29 January 2010 12:13 AM
To: squid-users_at_squid-cache.org
Subject: Re: [squid-users] Squid 3.0 stable20 crash

Kingsley Foreman wrote:
> I've been getting some crashes in squid, anything ranging from 1 per day to about 5
>
> So far I have this information.
>
> From debugging cache.log i've replaced the url's with ......
>
>
<snip>
> 2010/01/28 22:35:08.202| comm_calliocallback: 0
> 2010/01/28 22:35:08.202| commio_call_callback: called for 20
> 2010/01/28 22:35:08.202| client_side.cc(1299) clientWriteBodyComplete schedules clientWriteComplete
> 2010/01/28 22:35:08.202| clientWriteComplete: FD 20, sz 1878, err 0, off 1159773875, len 0x7f7d1d8afa50
> 2010/01/28 22:35:08.202| client_side_reply.cc(944) storeOKTransferDone out.offset=1159773605 objectLen()=1159773870 headers_sz=265
> 2010/01/28 22:35:08.202| clientReplyStatus: transfer is DONE
> 2010/01/28 22:35:08.202| clientReplyStatus: stream was not expected to complete!
> 2010/01/28 22:35:08.202| client_side.cc(1566) initiateClose: closing for STREAM_UNPLANNED_COMPLETE|STREAM_FAILED
> 2010/01/28 22:35:08.202| comm_close: FD 20
> 2010/01/28 22:35:08.202| commSetTimeout: FD 20 timeout -1
> 2010/01/28 22:35:08.202| commio_complete_callback: called for 20 (-10, 11)
> 2010/01/28 22:35:08.202| commio_call_callback: called for 20
> 2010/01/28 22:35:08.202| client_side.cc(2541) clientReadRequest FD 20 size 0
> 2010/01/28 22:35:08.202| commCallCloseHandlers: FD 20
> 2010/01/28 22:35:08.202| commCallCloseHandlers: ch->handler=1 data=0xbcb1c8
> 2010/01/28 22:35:08.202| ConnStateData::close: FD 20
> 2010/01/28 22:35:08.202| clientStreamDetach: Detaching node 0xbdbf68
> 2010/01/28 22:35:08.202| Freeing clientStreamNode 0xbdbf68
> 2010/01/28 22:35:08.202| clientStreamDetach: Calling 1 with cbdata 0x7f7d1efd8620
> 2010/01/28 22:35:08.202| clientStreamDetach: Detaching node 0xbdbee8
> 2010/01/28 22:35:08.202| Freeing clientStreamNode 0xbdbee8
> 2010/01/28 22:35:08.202| storeUnregister: called for '239DA444F4446A2B976F900A9B348824'
> 2010/01/28 22:35:08.202| storePendingNClients: returning 0
> 2010/01/28 22:35:08.202| StoreEntry::unlock: key '239DA444F4446A2B976F900A9B348824' count=1
> 2010/01/28 22:35:08.202| httpRequestFree: http://.....
> 2010/01/28 22:35:08.203| ACLChecklist::~ACLChecklist: destroyed 0xbe7dc8
> 2010/01/28 22:35:08.203| cleaning hdr: 0x2530eae0 owner: 3
> 2010/01/28 22:35:08.203| cleaning hdr: 0x2530eae0 owner: 3
> 2010/01/28 22:35:08.203| StoreEntry::unlock: key '239DA444F4446A2B976F900A9B348824' count=0
> 2010/01/28 22:35:08.203| storePendingNClients: returning 0
> 2010/01/28 22:35:08.203| UFSSwapDir::dereference: referencing 0x7f7d1d8afa50 1/4125
> 2010/01/28 22:35:08.203| HeapKeyGen_StoreEntry_LRU: 239DA444F4446A2B976F900A9B348824 heap_age=1.00 lastref=1264672683.00
> 2010/01/28 22:35:08.203| HeapKeyGen_StoreEntry_LRU: url=http://.........
> 2010/01/28 22:35:08.203| StoreEntry::purgeMem: Freeing memory-copy of 239DA444F4446A2B976F900A9B348824
> 2010/01/28 22:35:08.203| store.cc(378) destroyMemObject 0x1d3a9b10
> 2010/01/28 22:35:08.203| MemObject.cc(97) del MemObject 0x1d3a9b10
>
>
> ________________________________________________________________________________________
> And a coredump
>
>

Yay!

<snip>
> Core was generated by `/usr/sbin/squid -NCd1 -f /etc/squid/squid-fe-8084.conf'.
> Program terminated with signal 11, Segmentation fault.
> #0 0x00000000004b2170 in ?? ()
>
> (gdb) bt
> #0 0x00000000004b2170 in ?? ()
> #1 0x00000000004b2197 in ?? ()
> #2 0x00000000004b2197 in ?? ()
> #3 0x00000000004b2197 in ?? ()
> #4 0x00000000004b2197 in ?? ()
> #5 0x00000000004b2197 in ?? ()
> #6 0x00000000004b2197 in ?? ()
> #7 0x00000000004b2197 in ?? ()
> #8 0x00000000004b2197 in ?? ()
> #9 0x00000000004b2197 in ?? ()
> #10 0x00000000004b2197 in ?? ()
<snip>

Oh fudge. Absolutely useless.
Does look to be a very deep recursion problem somewhere though..

>
> Any suggestions would be great
>
> Kingsley

Step 1) will be to build or get a binary of Squid which includes both
the core dumping and debug symbols. Run that through at least one crash
to get a usable core.

Then the stack trace in the core will provide info to use for step (2).
   http://wiki.squid-cache.org/SquidFaq/BugReporting

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE21
   Current Beta Squid 3.1.0.15