Re: [squid-users] memory question and system performance

Then the question becomes: What is cricket measuring when it measures
memory usage?

In cachemgr the "General runtime information" page is a good place to
start. This includes among other things a summary of how much memory
Squid is using.

If you here see that Squid is constantly growing then the "Memory
Utilization" page can be used for determining what inside Squid is
causing the memory usage to grow.

Regards
Henrik

tor 2003-03-27 klockan 17.23 skrev Pat Lendon:
> I apologize for not knowing more about memory usage and squid - but, I
> just started to
> really look at memory and squid since cricket chart shows squid memory
> steadily grows.
>
> I printed the cache manager current memory usage. There is a lot of data
> and numbers there. Can you give me any hints at what I need to look at
> and be concerned with? Or, do I need to look at all entries in current
> memory usage?
>
> Top just reports that squid is using .05% of system memory. Since system
> has 1gig of memory and squid doesn't need that much memroy, then system
> has enough memory.
> How do I measure memory usage?
> Do I need to look at number of clients, avg http requests per minute,
> etc. from cache mgr?
> Listed is cache manager's General Runtime Info and Memory Utilization.
>
> I thank you for replying and for your help.
>
> -Pat
>
>
> Squid Object Cache: Version 2.5.STABLE1-20030102
>
> Start Time: Wed, 05 Mar 2003 20:10:04 GMT
> Current Time: Thu, 27 Mar 2003 16:01:06 GMT
>
> Connection information for squid:
> Number of clients accessing cache: 342
> Number of HTTP requests received: 11201554
> Number of ICP messages received: 0
> Number of ICP messages sent: 0
> Number of queued ICP replies: 0
> Request failure ratio: 0.00%
> Average HTTP requests per minute since start: 356.4
> Average ICP messages per minute since start: 0.0
> Select loop called: 180343978 times, 10.457 ms avg
> Cache information for squid:
> Request Hit Ratios: 5min: 31.8%, 60min: 37.5%
> Byte Hit Ratios: 5min: 21.3%, 60min: 15.0%
> Request Memory Hit Ratios: 5min: 7.9%, 60min: 2.6%
> Request Disk Hit Ratios: 5min: 37.2%, 60min: 37.7%
> Storage Swap size: 1887456 KB
> Storage Mem size: 8248 KB
> Mean Object Size: 10.35 KB
> Requests given to unlinkd: 2480416
> Median Service Times (seconds) 5 min 60 min:
> HTTP Requests (All): 0.09736 0.09219
> Cache Misses: 0.15888 0.18699
> Cache Hits: 0.01164 0.00919
> Near Hits: 0.12783 0.11465
> Not-Modified Replies: 0.00379 0.00767
> DNS Lookups: 0.01535 0.01686
> ICP Queries: 0.00000 0.00000
> Resource usage for squid:
> UP Time: 1885862.149 seconds
> CPU Time: 69315.550 seconds
> CPU Usage: 3.68%
> CPU Usage, 5 minute avg: 7.76%
> CPU Usage, 60 minute avg: 5.93%
> Maximum Resident Size: 0 KB
> Page faults with physical i/o: 2447
> Memory usage for squid via mallinfo():
> Total space in arena: 64956 KB
> Ordinary blocks: 37133 KB 3958 blks
> Small blocks: 0 KB 0 blks
> Holding blocks: 192 KB 1 blks
> Free Small blocks: 0 KB
> Free Ordinary blocks: 27823 KB
> Total in use: 37325 KB 57%
> Total free: 27823 KB 43%
> Total size: 65148 KB
> Memory accounted for:
> Total accounted: 30327 KB
> memPoolAlloc calls: 1588821210
> memPoolFree calls: 1588232248
> File descriptor usage for squid:
> Maximum number of file descriptors: 1024
> Largest file desc currently in use: 265
> Number of file desc currently in use: 254
> Files queued for open: 0
> Available number of file descriptors: 770
> Reserved number of file descriptors: 100
> Store Disk files open: 1
> Internal Data Structures:
> 182373 StoreEntries
> 1713 StoreEntries with MemObjects
> 1700 Hot Object Cache Items
> 182281 on-disk objects
>
> ------------------------------------------------------------------------
>
> Current memory usage:
>
> Pool Obj Size Allocated In Use Idle Allocations Saved Hit Rate
>
> (bytes) (#) (KB) high (KB) high (hrs) impact (%total) (#) (KB) high (KB)
> portion (%alloc) (#) (KB) high (KB) (number) (%num) (%vol) (%num) (number)
> 2K Buffer 2048 47 94 94 211.98 0 3 6 94 6 44 88 88 49347553 3.10 14.24
> 100.00 49347600
> 4K Buffer 4096 24 96 96 326.82 0 0 0 96 0 24 96 96 24315562 1.53 14.03
> 100.00 24315586
> 8K Buffer 8192 191 1528 1528 326.82 5 52 416 1528 27 139 1112 1528
> 7558542 0.48 8.72 100.00 7558733
> 16K Buffer 16384 1 16 16 497.29 0 0 0 16 0 1 16 16 252 0.00 0.00 99.60 253
> Client Socket Buffer 4096 329 1316 1316 168.41 4 275 1100 1316 84 54 216
> 764 49539337 3.12 28.59 100.00 49539666
> acl 48 52 3 3 139.82 0 51 3 3 98 1 1 3 872 0.00 0.00 94.37 924
> acl_ip_data 16 82 2 2 24.39 0 81 2 2 99 1 1 2 1343 0.00 0.00 94.25 1425
> acl_list 12 102 2 2 24.95 0 101 2 2 99 1 1 2 1707 0.00 0.00 94.36 1809
> auth_user_t 60 112 7 7 45.98 0 97 6 7 87 15 1 7 11968 0.00 0.00 99.07 12080
> auth_user_hash_pointer 24 112 3 3 45.98 0 97 3 3 87 15 1 3 10513 0.00
> 0.00 98.95 10625
> acl_proxy_auth_match_cache 20 220 5 5 212.08 0 96 2 5 44 124 3 5 21125
> 0.00 0.00 98.97 21345
> acl_user_data 8 2 1 1 524.01 0 1 1 1 50 1 1 1 24 0.00 0.00 92.31 26
> dread_ctrl 28 2 1 1 524.01 0 0 0 1 0 2 1 1 5394509 0.34 0.02 100.00 5394511
> dwrite_q 24 1 1 1 524.01 0 0 0 1 0 1 1 1 14863213 0.93 0.05 100.00 14863214
> FwdServer 12 222 3 3 233.56 0 13 1 3 6 209 3 3 7748922 0.49 0.01 100.00
> 7749144
> HttpReply 112 2031 223 223 209.36 1 1762 193 223 87 269 30 68 23753353
> 1.49 0.37 99.99 23755384
> HttpHeaderEntry 20 17777 348 348 214.83 1 15013 294 348 84 2764 54 127
> 373852313 23.51 1.05 100.00 373870090
> HttpHdrCc 16 739 12 12 214.86 0 261 5 12 35 478 8 11 20473359 1.29 0.05
> 100.00 20474098
> HttpHdrRangeSpec 8 800 7 7 238.80 0 0 0 7 0 800 7 7 107435 0.01 0.00
> 99.26 108235
> HttpHdrRange 12 7 1 1 497.76 0 0 0 1 0 7 1 1 17301 0.00 0.00 99.96 17308
> HttpHdrContRange 12 13 1 1 210.33 0 0 0 1 0 13 1 1 13293 0.00 0.00 99.90
> 13306
> intlist 8 5 1 1 524.01 0 5 1 1 100 0 0 1 85 0.00 0.00 94.44 90
> MemObject 112 2030 223 223 209.36 1 1762 193 223 87 268 30 68 12827428
> 0.81 0.20 99.98 12829458
> mem_node 4104 2219 8894 8894 326.82 29 2037 8164 8894 92 182 730 1620
> 26129597 1.64 15.11 99.99 26131816
> relist 40 4 1 1 524.01 0 4 1 1 100 0 0 1 68 0.00 0.00 94.44 72
> request_t 724 223 158 158 233.56 1 14 10 158 6 209 148 158 11210205 0.71
> 1.14 100.00 11210428
> StoreEntry 48 193453 9069 9069 97.10 30 182806 8570 9069 94 10647 500
> 1166 9096262 0.57 0.06 97.92 9289715
> wordlist 8 348 3 3 47.13 0 10 1 3 3 338 3 3 75526 0.00 0.00 99.54 75874
> ClientInfo 236 342 79 79 3.65 0 342 79 79 100 0 0 0 0 0.00 0.00 0.00 342
> MD5 digest 16 193453 3023 3023 97.10 10 182806 2857 3023 94 10647 167
> 389 11764020 0.74 0.03 98.38 11957473
> helper_request 12 4 1 1 332.00 0 0 0 1 0 4 1 1 18499 0.00 0.00 99.98 18503
> storeSwapTLV 16 4 1 1 524.01 0 0 0 1 0 4 1 1 18707183 1.18 0.04 100.00
> 18707187
> clientRequestBuffer 4096 318 1272 1272 233.55 4 58 232 1272 18 260 1040
> 1272 3755173 0.24 2.17 99.99 3755491
> storeSwapLogData 48 1 1 1 524.01 0 0 0 1 0 1 1 1 5139418 0.32 0.03
> 100.00 5139419
> Short Strings 36 19850 698 698 214.83 2 16835 592 698 85 3015 106 256
> 405392905 25.50 2.06 100.00 405412755
> Medium Strings 128 1671 209 209 166.45 1 861 108 209 52 810 102 184
> 81918713 5.15 1.48 100.00 81920384
> Long Strings 512 553 277 277 134.91 1 89 45 277 16 464 232 276 9044474
> 0.57 0.65 99.99 9045027
> cbdata acl_access (1) 32 59 2 2 140.13 0 58 2 2 98 1 1 2 990 0.00 0.00
> 94.38 1049
> cbdata aclCheck_t (2) 248 9 3 3 477.01 0 1 1 3 11 8 2 3 227832111 14.33
> 7.96 100.00 227832120
> cbdata clientHttpRequest (3) 540 230 122 122 233.56 0 28 15 122 12 202
> 107 118 11210059 0.71 0.85 100.00 11210289
> cbdata ConnStateData (4) 180 318 56 56 233.55 0 58 11 56 18 260 46 56
> 3755173 0.24 0.10 99.99 3755491
> cbdata ErrorState (5) 96 103 10 10 233.47 0 2 1 10 2 101 10 10 520207
> 0.03 0.01 99.98 520310
> cbdata FwdState (6) 52 199 11 11 233.50 0 9 1 11 5 190 10 11 7161463
> 0.45 0.05 100.00 7161662
> cbdata generic_cbdata (7) 20 30 1 1 337.91 0 1 1 1 3 29 1 1 3728932 0.23
> 0.01 100.00 3728962
> cbdata helper (8) 72 1 1 1 524.01 0 1 1 1 100 0 0 0 0 0.00 0.00 0.00 1
> cbdata helper_server (9) 88 50 5 5 524.01 0 50 5 5 100 0 0 5 1950 0.00
> 0.00 97.50 2000
> cbdata HttpStateData (12) 60 159 10 10 326.83 0 7 1 10 4 152 9 10
> 7013951 0.44 0.06 100.00 7014110
> cbdata ps_state (14) 136 1 1 1 524.01 0 0 0 1 0 1 1 1 7749143 0.49 0.15
> 100.00 7749144
> cbdata RemovalPolicy (15) 56 2 1 1 524.01 0 2 1 1 100 0 0 0 0 0.00 0.00
> 0.00 2
> cbdata RemovalPolicyWalker (16) 32 1 1 1 514.93 0 0 0 1 0 1 1 1 21 0.00
> 0.00 95.45 22
> cbdata RemovalPurgeWalker (17) 44 1 1 1 524.01 0 0 0 1 0 1 1 1 2025112
> 0.13 0.01 100.00 2025113
> cbdata store_client (18) 72 205 15 15 233.50 0 10 1 15 5 195 14 15
> 13114506 0.82 0.13 100.00 13114711
> event 32 23 1 1 405.51 0 8 1 1 35 15 1 1 3645476 0.23 0.02 100.00 3645499
> UFS IO State data 8 36 1 1 336.44 0 1 1 1 3 35 1 1 6213443 0.39 0.01
> 100.00 6213479
> LRU policy node 12 195204 2288 2288 97.12 8 184465 2162 2288 94 10739
> 126 291 4542537 0.29 0.01 95.88 4737741
> cbdata body_size (19) 36 5 1 1 140.13 0 4 1 1 80 1 1 1 70 0.00 0.00
> 93.33 75
> cbdata ConnectStateData (20) 72 172 13 13 235.92 0 2 1 13 1 170 12 13
> 4951373 0.31 0.05 100.00 4951545
> CommWriteStateData 24 76 2 2 326.82 0 1 1 2 1 75 2 2 53669301 3.38 0.18
> 100.00 53669377
> close_handler 12 808 10 10 233.55 0 134 2 10 17 674 8 10 24267474 1.53
> 0.04 100.00 24268282
> ipcache_entry 68 957 64 64 187.64 0 921 62 64 96 36 3 4 1238382 0.08
> 0.01 99.92 1239339
> idns_query 564 26 15 15 336.44 0 0 0 15 0 26 15 15 1239313 0.08 0.10
> 100.00 1239339
> Basic Scheme User Data 20 112 3 3 45.98 0 97 2 3 87 15 1 3 11968 0.00
> 0.00 99.07 12080
> cbdata authenticateStateData (21) 28 4 1 1 332.00 0 0 0 1 0 4 1 1 18499
> 0.00 0.00 99.98 18503
> cbdata RebuildState (24) 616 1 1 1 524.01 0 0 0 1 0 1 1 1 0 0.00 0.00
> 0.00 1
> pconn_data 20 86 2 2 187.63 0 42 1 2 49 44 1 2 2163105 0.14 0.01 100.00
> 2163191
> pconn_fds 32 85 3 3 187.63 0 42 2 3 49 43 2 3 2163106 0.14 0.01 100.00
> 2163191
> Authenticate Request Data 20 230 5 5 233.56 0 27 1 5 12 203 4 5 11118766
> 0.70 0.03 100.00 11118996
> Dlink list nodes 12 230 3 3 233.56 0 27 1 3 12 203 3 3 11118766 0.70
> 0.02 100.00 11118996
> cbdata auth_user_ip_t (26) 36 110 4 4 212.08 0 96 4 4 87 14 1 4 15480
> 0.00 0.00 99.29 15590
> cbdata storeIOState (27) 68 36 3 3 336.44 0 1 1 3 3 35 3 3 6213443 0.39
> 0.06 100.00 6213479
> cbdata SslStateData (28) 72 68 5 5 470.84 0 4 1 5 6 64 5 5 587414 0.04
> 0.01 99.99 587482
> cbdata FtpStateData (29) 8384 15 123 123 498.04 0 0 0 123 0 15 123 123
> 2133 0.00 0.00 99.30 2148
> Total - 636996 30328 30328 3.65 100 591631 25149 27713 83 45365 5179
> 6600 1589405721 99.96 100.00 99.96 1590042717
>
> Cumulative allocated volume: 709.78 GB
> Current overhead: 528714 bytes (2.053%)
> Idle pool limit: 2048.00 MB
> memPoolAlloc calls: 1590042717
> memPoolFree calls: 1589451084
>
> String Pool Impact
>
> (%strings) (%volume)
> Short Strings 95 80
> Medium Strings 5 14
> Long Strings 1 6
> Other Strings 0 0
>
> ------------------------------------------------------------------------
>
>
>
>
>
>
>
>
>
>
>
>
>
> Henrik Nordstrom wrote:
>
> >Pat Lendon wrote:
> >
> >
> >
> >>We are using Redhat Linux OS. System has 1 gig of memory. Cache
> >>Manager and top command show squid memory at 5%.
> >>I've read squid faq 8.11 "How much memory do I need in my squid server?"
> >>cache_dir ufs /d1/squid_cache 2048 16 256
> >>/d1 resides on separate file system (18gig of disk, with 7 gig
> >>available). Squid cache is configured to use 2 gig.
> >>10 MB of ram per 2 GB of total cach_dirs would be 20MB
> >>cache_mem 8 MB
> >>Plus additional/extra 20MB does calculate to the 5% (50 meg) of the 1
> >>gig of memory system contains. System appears to have enough memory.
> >>
> >>
> >
> >The above configuration should be happy with as little as 128MB with a
> >standard bloated full server installation.. or as little as 64MB if the
> >server is stripped down to not run X or any other unneeded memoryhungry
> >services besides Squid..
> >
> >
> >
> >>Looking at chart enclosed, is it normal for squid memory usage to grow
> >>and grow?
> >>
> >>
> >
> >No. The normal is that Squid relatively quickly settles on a stdeady
> >value after it has been in use for a while, and grow a little while the
> >cache is being filled up.
> >
> >If top says Squid is using 5% of the memory, What is it that you measure
> >when you measure the Squid memory usage?
> >
> >
> >
> >>What causes this?
> >>
> >>
> >
> >Good question. If it is Squid who is using this memory additional hints
> >may be provided by looking at the "Memory Utilization" cachemgr page.
> >
> >
> >
> >>Furthermore - I'm also seeing slight delays on system. System has two
> >>AMD Athlon(TM) MP 1800+ processors cpu MHz 1533.431. When system seems
> >>delayed we run top and cpu's are not even at 50%, memory looks fine.
> >>
> >>
> >
> >Keep in mind that Squid is a single application and can only make use of
> >one CPU..
> >
> >Regards
> >Henrik
> >
> >
> >
> >

-- 
Henrik Nordstrom <hno@squid-cache.org>
MARA Systems AB, Sweden