Re: MemPools rewrite

On 21 Oct 2000, at 1:14, Henrik Nordstrom <hno@hem.passagen.se> wrote:

> Andres Kroonmaa wrote:
>
> > In practice idle memory is low. Even on busiest cache I've never seen idle
> > memory ever reach even close to default idle limit of 50MB.
>
> Well, there are still race conditions where the memobject can
> temporarily grow huge, so don't bet on it.

 well, if its temporary, then its released. We can return memory in chunks
 to the system. Probability that every chunk gets locked by few longliving
 items is there, but should be pretty small because of first-fit approach.
 Also, when chunks are large enough to justify mmap, such bursts would not
 bang holes into heap.

> > How can I say? normally its looking good:
> > Memory usage for squid via mallinfo():
> > Total space in arena: 313047 KB
> > Ordinary blocks: 304139 KB 370623 blks
> > Small blocks: 0 KB 0 blks
> > Holding blocks: 20336 KB 17 blks
> > Free Small blocks: 0 KB
> > Free Ordinary blocks: 8907 KB
> > Total in use: 324475 KB 104%
> > Total free: 8907 KB 3%
> > Memory accounted for:
> > Total accounted: 207924 KB
>
> The figures you need to look at for fragmentation is the total space in
> arena (sbrk) and total free.
>
> The percentage values are slightly wrong due to a minor misunderstanding
> on what the four types (arena, ordinary, small, holding) stands for.

 seems that dlmalloc accounts mmaped stuff to that >100%, thats why 104%

 this stuff is looking pretty nasty:

Memory usage for squid via mallinfo():
        Total space in arena: 301352 KB
        Ordinary blocks: 246497 KB 42730 blks
        Small blocks: 0 KB 0 blks
        Holding blocks: 35412 KB 34 blks
        Free Small blocks: 0 KB
        Free Ordinary blocks: 54854 KB
        Total in use: 281909 KB 94%
        Total free: 54854 KB 18%
Memory accounted for:
        Total accounted: 157187 KB

 seems to be quite some fragmentation.

Current memory usage:
Pool Obj Size Allocated
                     (bytes) (#) (KB) high (KB)
Store Mem Buffer 4096 4321 17284 17284
StoreEntry 48 2030050 95159 95159
MD5 digest 16 2030050 31720 31720
 idle: 10920

> > up, I guess I'd be in trouble... Maybe dirty startups, when loading
> > of storedb takes long time, memory gets fragmented alot, as StoreEntry
> > objects are very longliving, and while loading them lots of other
> > stuff flies by. Maybe some kind of spikes in activity cause
> > fragmenting,
>
> In 2.3 there is a accounted memory leak on dirty startups combined with
> a temporary malloc overload condition causing "free ordinary blocks" to
> skyrocket. Patches are available from my patch page, or as part of 2.4

 sounds like I'm hit by this one, but there's been no dirty startups.

> > but sometimes squid size skyrockets, and stays for a long time higher
> > than normal.
>
> Have two patches for this as well.. one where certain request patterns
> could lock up memory, one where certain malformed server replies could
> lock up memory. Both caused the memory usage to grow and grow until the
> connection was closed.
>
> Which Squid version and patches are you running?
 
 2.3.STABLE4-hno.CVS, not sure which date. probably aug.23
 I can't find any patch that could apply from your page.

------------------------------------
 Andres Kroonmaa <andre@online.ee>
 Delfi Online
 Tel: 6501 731, Fax: 6501 708
 Pärnu mnt. 158, Tallinn,
 11317 Estonia
Received on Fri Oct 27 2000 - 17:31:38 MDT