RE: [squid-users] Squid 3.0 stable20 crash

I should ask, because I know someone will have a idea how to do it here rather then googling it

How do you add core dumping and debug symbols when building squid?

Kingsley

-----Original Message-----
From: Amos Jeffries [mailto:squid3_at_treenet.co.nz]
Sent: Friday, 29 January 2010 12:13 AM
To: squid-users_at_squid-cache.org
Subject: Re: [squid-users] Squid 3.0 stable20 crash

Kingsley Foreman wrote:
> I've been getting some crashes in squid, anything ranging from 1 per day to about 5
>
> So far I have this information.
>
> From debugging cache.log i've replaced the url's with ......
>
>
<snip>
> 2010/01/28 22:35:08.202| comm_calliocallback: 0
> 2010/01/28 22:35:08.202| commio_call_callback: called for 20
> 2010/01/28 22:35:08.202| client_side.cc(1299) clientWriteBodyComplete schedules clientWriteComplete
> 2010/01/28 22:35:08.202| clientWriteComplete: FD 20, sz 1878, err 0, off 1159773875, len 0x7f7d1d8afa50
> 2010/01/28 22:35:08.202| client_side_reply.cc(944) storeOKTransferDone out.offset=1159773605 objectLen()=1159773870 headers_sz=265
> 2010/01/28 22:35:08.202| clientReplyStatus: transfer is DONE
> 2010/01/28 22:35:08.202| clientReplyStatus: stream was not expected to complete!
> 2010/01/28 22:35:08.202| client_side.cc(1566) initiateClose: closing for STREAM_UNPLANNED_COMPLETE|STREAM_FAILED
> 2010/01/28 22:35:08.202| comm_close: FD 20
> 2010/01/28 22:35:08.202| commSetTimeout: FD 20 timeout -1
> 2010/01/28 22:35:08.202| commio_complete_callback: called for 20 (-10, 11)
> 2010/01/28 22:35:08.202| commio_call_callback: called for 20
> 2010/01/28 22:35:08.202| client_side.cc(2541) clientReadRequest FD 20 size 0
> 2010/01/28 22:35:08.202| commCallCloseHandlers: FD 20
> 2010/01/28 22:35:08.202| commCallCloseHandlers: ch->handler=1 data=0xbcb1c8
> 2010/01/28 22:35:08.202| ConnStateData::close: FD 20
> 2010/01/28 22:35:08.202| clientStreamDetach: Detaching node 0xbdbf68
> 2010/01/28 22:35:08.202| Freeing clientStreamNode 0xbdbf68
> 2010/01/28 22:35:08.202| clientStreamDetach: Calling 1 with cbdata 0x7f7d1efd8620
> 2010/01/28 22:35:08.202| clientStreamDetach: Detaching node 0xbdbee8
> 2010/01/28 22:35:08.202| Freeing clientStreamNode 0xbdbee8
> 2010/01/28 22:35:08.202| storeUnregister: called for '239DA444F4446A2B976F900A9B348824'
> 2010/01/28 22:35:08.202| storePendingNClients: returning 0
> 2010/01/28 22:35:08.202| StoreEntry::unlock: key '239DA444F4446A2B976F900A9B348824' count=1
> 2010/01/28 22:35:08.202| httpRequestFree: http://.....
> 2010/01/28 22:35:08.203| ACLChecklist::~ACLChecklist: destroyed 0xbe7dc8
> 2010/01/28 22:35:08.203| cleaning hdr: 0x2530eae0 owner: 3
> 2010/01/28 22:35:08.203| cleaning hdr: 0x2530eae0 owner: 3
> 2010/01/28 22:35:08.203| StoreEntry::unlock: key '239DA444F4446A2B976F900A9B348824' count=0
> 2010/01/28 22:35:08.203| storePendingNClients: returning 0
> 2010/01/28 22:35:08.203| UFSSwapDir::dereference: referencing 0x7f7d1d8afa50 1/4125
> 2010/01/28 22:35:08.203| HeapKeyGen_StoreEntry_LRU: 239DA444F4446A2B976F900A9B348824 heap_age=1.00 lastref=1264672683.00
> 2010/01/28 22:35:08.203| HeapKeyGen_StoreEntry_LRU: url=http://.........
> 2010/01/28 22:35:08.203| StoreEntry::purgeMem: Freeing memory-copy of 239DA444F4446A2B976F900A9B348824
> 2010/01/28 22:35:08.203| store.cc(378) destroyMemObject 0x1d3a9b10
> 2010/01/28 22:35:08.203| MemObject.cc(97) del MemObject 0x1d3a9b10
>
>
> ________________________________________________________________________________________
> And a coredump
>
>

Yay!

<snip>
> Core was generated by `/usr/sbin/squid -NCd1 -f /etc/squid/squid-fe-8084.conf'.
> Program terminated with signal 11, Segmentation fault.
> #0 0x00000000004b2170 in ?? ()
>
> (gdb) bt
> #0 0x00000000004b2170 in ?? ()
> #1 0x00000000004b2197 in ?? ()
> #2 0x00000000004b2197 in ?? ()
> #3 0x00000000004b2197 in ?? ()
> #4 0x00000000004b2197 in ?? ()
> #5 0x00000000004b2197 in ?? ()
> #6 0x00000000004b2197 in ?? ()
> #7 0x00000000004b2197 in ?? ()
> #8 0x00000000004b2197 in ?? ()
> #9 0x00000000004b2197 in ?? ()
> #10 0x00000000004b2197 in ?? ()
<snip>

Oh fudge. Absolutely useless.
Does look to be a very deep recursion problem somewhere though..

>
> Any suggestions would be great
>
> Kingsley

Step 1) will be to build or get a binary of Squid which includes both
the core dumping and debug symbols. Run that through at least one crash
to get a usable core.

Then the stack trace in the core will provide info to use for step (2).
   http://wiki.squid-cache.org/SquidFaq/BugReporting

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE21
   Current Beta Squid 3.1.0.15