Re: [squid-users] Squid getting too big?

On Thu, Aug 09, 2001, BAARDA, Don wrote:
> G'day,
>
> > -----Original Message-----
> > From: Mike Diggins [mailto:diggins@mcmail.cis.mcmaster.ca]
> > Sent: Tuesday, August 07, 2001 5:28 AM
> > To: Florin Andrei
> > Cc: squid-users@squid-cache.org
> > Subject: Re: [squid-users] Squid getting too big?
> [...]
> > On 6 Aug 2001, Florin Andrei wrote:
> >
> > > On 03 Aug 2001 08:56:05 -0400, Mike Diggins wrote:
> > > >
> > > > I'm concerned about how quickly Squid is growing in size.
> [...]
>
> Just wondering, and this is a wild stab in the dark, have any of you guys
> played with the memory_pools option? The whole concept of an application
> doing a better job of managing memory than the OS it runs on seems flawed to
> me, so I turn memory_pools off.
>
> I have a feeling that if cachemgr is not reporting the missing memory, then
> turning memory_pools off will not help. However, if cachemgr is not
> reporting memory reserved by memory_pools, and the memory_pool manager is
> flawed and/or getting fragmented, it could explain the difference.

Right. It sounds crazy, but its not.
See, UNIX is designed to be generally good at everything.
malloc() implementations are a good example - it would be rather
challenging to implement a malloc library that would service
all access types. A few very large blocks of RAM? What about
a lot of small blocks of RAM?

squid does the latter. Unfortunately, most malloc implementations
don't deal with it very well, and some just leak RAM.

the memory pools implementation in squid was designed to combat
this. It keeps a "cache" of allocated RAM, knowing that its going
to be reused at some later date. Since the system malloc hasn't
got any way of knowing (without statistical sampling) that
the application is doing lots of fixed-sized allocations, it has
to use a "generic" algorithm which is going to be comparitively
slow.

What is being worked on right now is a block allocator to replace
the memory pools implementation - this is an allocator tailored
towards squid's memory access patterns. Hopefully it'll cut down
on squid's memory footprint by quite a bit on large caches
(as malloc alignment can waste a _lot_ of RAM with the small
StoreEntry allocations)

HTH,

Adrian
Received on Thu Aug 09 2001 - 05:09:38 MDT