Re: memory watermarks ?

On 27 Oct 2000, at 2:26, Henrik Nordstrom <hno@hem.passagen.se> wrote:

> [linux i386 glibc assumed in this message]
>
> > if I understand right, then VM size and space in arena should match.
> > As they don't then either system or mallinfo is lying.
>
> Not quite.
> VM size is the total address space set up for the process. This includes

 correct, I should have said match approximately. mmap of libs, stack
 and shared mem are all very small for squid compared to its mem usage.
 perhaps accounting for some few MB of vm. Still, VSZ and malloc usage
 should match pretty closely, not be different in times.

> Stack for threads are allocated dynamically as requred by the threads,
> starting with 16K per thread. By default there is an upper bound
> somewhere between 1 to 2MB (not sure exacly where the limit are, but the
> stack segments are starting at 2MB intervals..).

 this probably depends on OS. solaris for eg. mmaps 1M of stack per
 thread into process space, which is really used on demand, by attaching
 real ram pages to reserved ones when need arises. My squid box is using
 actually about 4K per thread, but reserved size is 1M per thread.

 so if something is counting for reserved mmapping space, then it gets
 lots of VM accounted to this.

> Details of the virtual memory allocation can be found in
> /proc/<nn>/maps. The data segment memory is the second entry of the
> process image (the one with write permission). mmap() memory starts at
> 40000000, and the top of the stack is at c0000000. Threads allocates
> their stacks below the normal stack.

 this should give a best clue as to where the vm has gone.

> > Maybe some system libs are using memory in a way that malloc lib
> > cannot account for, and are leaking.
>
> Which should be memory allocated using mmap(), as sbrk is controlled
> exclusively by malloc, and is the value reported in "total space in
> arena" by mallinfo IIRC.
>
> > Squid's own leaks should be reflected in mallinfo, I believe.
>
> As long as we don't use mmap() then yes.

 and we don't, do we? dlmalloc when using mmap is accounting for mmap
 in mallinfo, by difference between space in arena and "total in use",
 which can go over 100% when using mmap.

> The following shows the status of a small program doing malloc() of 1000

 this is very illustrative on how differently top and ps are accounting
 VM. most probably ps is showing total reserved space even if its not
 touched. Then it is quite normal to count shared memory many times.

> $ ps -up 23911
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> henrik 23911 0.0 0.6 1025088 380 pts/0 S 01:50 0:00 ./a.out

> $ top
> PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME
> 23911 henrik 0 0 380 380 312 S 20 0.0 0.6 0:00 a.out

------------------------------------
 Andres Kroonmaa <andre@online.ee>
 Delfi Online
 Tel: 6501 731, Fax: 6501 708
 Pärnu mnt. 158, Tallinn,
 11317 Estonia
Received on Fri Oct 27 2000 - 04:17:21 MDT