Re: memory leak associated w/running any ecap adapter

Link to all that you requested is at the bottom of this message. For
completeness, below are the full details of what I did.

I'm sending this message via the nabble web interface (though I am
subscribed to the list) but I don't see Alex's response in the web
interface. I assume people on the list got it since I did, but I don't know
why nabble isn't showing it in the archive.

I built squid (3.2.5) with --disable-optimizations and
--with-valgrind-debug. My full configure command for the build was:

./configure --disable-optimizations --with-valgrind-debug
--prefix=/usr/local/squid3 --sysconfdir=/etc/squid3
--with-default-user=proxy --with-logdir=/var/log
--with-pidfile=/var/run/squid.pid --enable-ecap
EXT_LIBECAP_CFLAGS='-I/usr/local/squid3/include'
EXT_LIBECAP_LIBS='-L/usr/local/squid3/lib -lecap'

Separately I built libecap-0.2.0 and ecap_adapter_passthru from the
ecap_adapter_sample_0.2.0.

I am running squid as a reverse proxy and I used the html from apache's
default index.html (the one that says, "It works!") as my test case. For
what it's worth and in case it helps, I believe the memory leak may be
proportional to the size of the html response because this simple test case
appears to leak less than what I see when hitting normal/larger web pages.
As I mentioned in a separate thread, running the cache manager's "mem"
report shows the largest memory use consumed by the pools identified as "32K
Buffer" followed by "16K Buffer" followed by "8K Buffer". It is only my
guess that these buffers are proportional to the size of the reponse.

I also set "memory_pools off" and "memory_pools_limit 0". I set
debug_options to ALL,9 and cleared out the existing cache.log. The memory
leak does not happen when squid serves up the request from the cache, so I
am running with "cache deny all".

After running valgrind the first time, I saw that it suggested "Rerun with
--leak-check=full to see details of leaked memory" so I then ran under
valgrind control with:

> sudo valgrind --leak-check=full /usr/local/squid3/sbin/squid -N -YC -f
> /etc/squid3/squid.conf.debug

I ran one instance of the test using ab, then saved off the cache.log file
and valgrind output.

As an extra step, I then re-ran the test sending in one-thousand (1000)
requests for the same simple html file. I wanted to see which potential
memory loss grew the most, thinking that might help identify the specific
cause of this problem. I captured the valgrind heap output and have
included that as well. The largest potential loss from my run of 1,000 is:

==9921== 210,120 bytes in 51 blocks are possibly lost in loss record 1,254
of 1,267
==9921== at 0x402A5E6: calloc (in
/usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==9921== by 0x83B0905: xcalloc (xalloc.cc:75)
==9921== by 0x83AB493: MemPoolMalloc::allocate() (MemPoolMalloc.cc:62)
==9921== by 0x83A8FF0: MemImplementingAllocator::alloc() (MemPool.cc:242)
==9921== by 0x83A93B6: MemAllocatorProxy::alloc() (MemPool.cc:355)
==9921== by 0x824E958: mem_node::operator new(unsigned int) (in
/usr/local/squid3/sbin/squid)
==9921== by 0x824DF90: mem_hdr::nodeToRecieve(long long) (stmem.cc:335)
==9921== by 0x824E498: mem_hdr::write(StoreIOBuffer const&)
(stmem.cc:385)
==9921== by 0x8220FCD: MemObject::write(StoreIOBuffer, void (*)(void*,
StoreIOBuffer), void*) (MemObject.cc:163)
==9921== by 0x8252188: StoreEntry::write(StoreIOBuffer) (store.cc:878)
==9921== by 0x82522BC: StoreEntry::append(char const*, int)
(store.cc:897)
==9921== by 0x8252373: storeAppendVPrintf(StoreEntry*, char const*,
char*) (store.cc:918)

10k requests (10x more than this test of 1k requests) with this sample file
leaks about 200MB according to top, so this appears to me to be the culprit.

So in my 162KB squid_debug.tar.gz file is:

cache.log.debug - ALL,9 output from one single request
valgrind.out - full valgrind output from one single request
valgrind1000.out - heap summary from 1,000 requests

I put the file on my server and have created a bit.ly link to it:

http://bit.ly/Ucz8t6 <http://bit.ly/Ucz8t6>

I'll take the file down once we hopefully figure all this out.

Thank you, thank you, thank you!
Tom

--
View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/memory-leak-associated-w-running-any-ecap-adapter-tp4657812p4657841.html
Sent from the Squid - Development mailing list archive at Nabble.com.