Very high squid 2.1 memory usage during store rebuild.

From: Ivan Debnár <debnar@dont-contact.us>
Date: Wed, 24 Mar 1999 12:59:11 +0100

Hello,
 
I would be very glad, if anyone could help me with following problem:
I'm using Squid 2.1.PATCH2 of FreeBSD 3.1-STABLE for about 2 months. Everything works exceptionaly fine.
 
However, when I reboot the system, or kill squid without shutdown, following cache storage rebuild consumes very much memory. My configuration is attached, from the cachemgr I see very high "Storage MEM space - 2.5 - 3 GB" !!! (it has actualy about 200MB RAM allocated)

The machine has only 128 MB Ram, 3,5GB SQUID disk storage, and I don't thing it's a good idea to store every object in memory for rebuild...
 
After rebuilding the storage, and a clean shutdown-restart (to clean the swapping), the squid reads the clean storage and works without problems with 30Mb RAM allocated.

Is it bug in me ( my configuration) or is it known problem ?
 
Any enligntment gratly appreciated.
 
 
Ivan Debnár
Zoznam s.r.o.
 
http://www.zoznam.sk
tel.://0903 506197
fax://088 4115246
 
General Runtime Information:
 
Cache Manager menu

--------------------------------------------------------------------------------

Squid Object Cache: Version 2.1.PATCH2

Start Time: Wed, 24 Mar 1999 11:21:05 GMT
Current Time: Wed, 24 Mar 1999 11:45:55 GMT

Connection information for squid:
 Number of clients accessing cache: 7
 Number of HTTP requests received: 620
 Number of ICP messages received: 389
 Number of ICP messages sent: 782
 Number of queued ICP replies: 0
 Request failure ratio: 0.00%
 HTTP requests per minute: 25.0
 ICP messages per minute: 47.2
 Select loop called: 56433 times, 26.398 ms avg
Cache information for squid:
 Request Hit Ratios: 5min: 0.0%, 60min: 3.9%
 Byte Hit Ratios: 5min: 3.9%, 60min: 2.5%
 Storage Swap size: 1746 KB
 Storage Mem size: 1540 KB
 Storage LRU Expiration Age: 30.00 days
 Mean Object Size: 4.43 KB
 Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
 HTTP Requests (All): 4.07741 5.06039
 Cache Misses: 3.86308 5.63745
 Cache Hits: 0.00000 0.35832
 Near Hits: 0.00000 2.79397
 Not-Modified Replies: 0.00000 0.03622
 DNS Lookups: 0.00000 2.24458
 ICP Queries: 0.68173 0.47649
Resource usage for squid:
 UP Time: 1489.693 seconds
 CPU Time: 238.789 seconds
 CPU Usage: 16.03%
 CPU Usage, 5 minute avg: 2.41%
 CPU Usage, 60 minute avg: 16.13%
 Maximum Resident Size: 108852 KB
 Page faults with physical i/o: 86569
Memory accounted for:
 Total accounted: 100953 KB
File descriptor usage for squid:
 Maximum number of file descriptors: 2088
 Largest file desc currently in use: 40
 Number of file desc currently in use: 24
 Available number of file descriptors: 2064
 Reserved number of file descriptors: 100
 Disk files open: 3
Internal Data Structures:
 1402425 StoreEntries
    338 StoreEntries with MemObjects
    333 Hot Object Cache Items
 223853 Filemap bits set
    394 on-disk objects

--------------------------------------------------------------------------------

Generated Wed, 24 Mar 1999 11:45:54 GMT, by cachemgr.cgi/2.1.PATCH2@bb.web.sk
 
Configuration:
 
Cache Manager menu

--------------------------------------------------------------------------------

http_port 3128
icp_port 3130
tcp_incoming_address 0.0.0.0
tcp_outgoing_address 255.255.255.255
udp_incoming_address 0.0.0.0
udp_outgoing_address 255.255.255.255
cache_peer cache2.pvt.sk Parent 8080 3130 no-digest default
cache_peer pcb.web.sk Sibling 8080 3130 closest-only
icp_query_timeout 0
mcast_icp_query_timeout 2000
dead_peer_timeout 10 seconds
hierarchy_stoplist cgi-bin
hierarchy_stoplist ?
hierarchy_stoplist \.cgi
hierarchy_stoplist \.fcgi
hierarchy_stoplist \.asp
hierarchy_stoplist \.php
hierarchy_stoplist netaddress
hierarchy_stoplist hotmail
no_cache Deny QUERY
cache_mem 8388608 bytes
cache_swap_low 95
cache_swap_high 98
cache_mem_low 75
cache_mem_high 95
maximum_object_size 8388608 bytes
ipcache_size 1024
ipcache_low 90
ipcache_high 95
fqdncache_size 1024
cache_dir /usr/local/squid/cache 3500 16 256
cache_access_log /usr/local/squid/logs/access.log
cache_log /dev/null
cache_store_log /dev/null
emulate_httpd_log off
mime_table /usr/local/etc/squid/mime.conf
log_mime_hdrs off
pid_filename /var/run/squid.pid
debug_options ALL,1
ident_lookup off
log_fqdn off
client_netmask 255.255.255.255
ftp_user Squid@
ftp_list_width 48
cache_dns_program /usr/local/libexec/dnsserver
dns_children 5
dns_defnames off
unlinkd_program /usr/local/libexec/unlinkd
pinger_program /usr/local/libexec/pinger
redirect_children 5
redirect_rewrites_host_header on
authenticate_children 5
authenticate_ttl 3600
wais_relay_port 0
request_size 102400 bytes
refresh_pattern -i \.(jpe?g|gif|png) 1440 80% 43200
 reload-into-ims
refresh_pattern -i \.(zip|rar|arj|cab|exe) 10080 80% 43200

refresh_pattern -i \.(cgi|asp|php|fcgi) 0 20% 60

refresh_pattern -i \.(htm|hti|htx|js) 60 40% 4320

reference_age 2592000 seconds
quick_abort_min 16 KB
quick_abort_max 16 KB
quick_abort_pct 95
negative_ttl 300 seconds
positive_dns_ttl 21600 seconds
negative_dns_ttl 300 seconds
range_offset_limit 0 bytes
connect_timeout 120 seconds
siteselect_timeout 4 seconds
read_timeout 900 seconds
request_timeout 15 seconds
client_lifetime 86400 seconds
half_closed_clients on
pconn_timeout 120 seconds
shutdown_lifetime 15 seconds
acl QUERY urlpath_regex cgi-bin
acl QUERY urlpath_regex \?
acl QUERY urlpath_regex \.cgi
acl QUERY urlpath_regex \.asp
acl QUERY urlpath_regex \.fcgi
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1
acl SSL_ports port 442
acl SSL_ports port 443
acl SSL_ports port 563
acl Safe_ports port 80
acl Safe_ports port 443
acl Safe_ports port 1025-65535
acl Safe_ports port 21
acl Safe_ports port 563
acl Safe_ports port 70
acl Safe_ports port 210
acl CONNECT method CONNECT
acl klienti src 195.28.74.0/255.255.255.0
acl klienti src 195.28.73.0/255.255.255.0
acl localservers dst 195.28.73.0/255.255.255.0
http_access Allow manager localhost
http_access Deny manager
http_access Deny !Safe_ports
http_access Deny CONNECT !SSL_ports
http_access Allow klienti
http_access Allow localhost
http_access Deny all
icp_access Allow all
miss_access Allow all
proxy_auth_realm Squid proxy-caching web server
cache_mgr cachemaster@web.sk
cache_effective_user nobody
cache_effective_group nogroup
visible_hostname cache.web.sk
announce_period 604800 seconds
announce_host tracker.ircache.net
announce_port 3131
httpd_accel_host virtual
httpd_accel_port 3128
httpd_accel_with_proxy on
httpd_accel_uses_host_header on
dns_testnames www.web.sk
dns_testnames netscape.com
dns_testnames internic.net
dns_testnames nlanr.net
dns_testnames microsoft.com
logfile_rotate 1
append_domain .web.sk
tcp_recv_bufsize 0 bytes
err_html_text
memory_pools on
memory_pools_limit 31457280 bytes
forwarded_for on
log_icp_queries on
icp_hit_stale off
minimum_direct_hops 4
cachemgr_passwd XXXXXXXXXX all
store_avg_object_size 11 KB
store_objects_per_bucket 20
http_anonymizer off
client_db on
netdb_low 900
netdb_high 1000
netdb_ping_period 300 seconds
query_icmp on
test_reachability off
buffered_logs on
reload_into_ims off
always_direct Allow localservers
icon_directory /usr/local/etc/squid/icons
error_directory /usr/local/etc/squid/errors
minimum_retry_timeout 5 seconds
maximum_single_addr_tries 3
as_whois_server whois.ra.net
incoming_icp_average 6
incoming_http_average 4
min_icp_poll_cnt 8
min_http_poll_cnt 8
max_open_disk_fds 0
offline_mode off
uri_whitespace allow
persistent_client_posts on
Received on Wed Mar 24 1999 - 05:26:41 MST

This archive was generated by hypermail pre-2.1.9 : Tue Dec 09 2003 - 16:45:24 MST