Re: [squid-users] Slow memory leak

Hi Eliezer,

Thanks for the feedback. This memory leak causes real-world problems for me, as squid starts to do a lot of swapping when it exhausts physical RAM, and things slow down. Also, the sheer magnitude of the memory numbers is just ridiculous... squid easily grows to in excess of 10GB of resident memory as shown in top. I've got the servers restarting their squids periodically, but that's a pretty bad solution.

Here is my squid.conf. Note two include files... squid_ns5_allowed_ips.conf and squid_blacklist_ips.conf . These are lists of IPs in an ACL. The allowed IPs one has lots of entries (almost 20,000) and that's the only thing I think is really unusual about my setup, so I'm wondering if that is exposing a leak somehwere.

##### BEGIN squid.conf #######
http_port 5000
http_port 5001
http_port 5002
http_port 5003
http_port 5020
http_port 5021
http_port 5022
http_port 5023

cache_mgr [omitted]
visible_hostname [omitted]
max_filedesc 32768

hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
cache deny QUERY
access_log /var/log/squid/access.log squid
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 4320
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 4320
acl demo_sites dstdomain .raynersoftware.com
acl banned_sites dstdomain client.versiontracker.com
acl banned_sites dstdomain .rapidshare.com

# allow yuki2
acl a_ips src 64.62.244.50/32

include /etc/squid/squid_ns5_allowed_ips.conf
include /etc/squid/squid_blacklist_ips.conf

# acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
acl SSL_ports port 443 563
acl Safe_ports port 80
acl Safe_ports port 21
acl Safe_ports port 443 563
acl Safe_ports port 70
acl Safe_ports port 210
acl Safe_ports port 1025-65535
acl Safe_ports port 280
acl Safe_ports port 488
acl Safe_ports port 591
acl Safe_ports port 777
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny to_localhost
http_access deny banned_sites
http_access deny banned_ips
http_access allow demo_sites
# we shorten allowed_ips to a_ips to save space in our include file
http_access allow a_ips
http_access deny all
http_reply_access allow all
#icp_access allow all

pid_filename /var/run/squid.pid

coredump_dir /var/spool/squid

via off
forwarded_for off
##### END squid.conf #######

When I do a mgr:info on one particular box, I get this:

Memory usage for squid via mallinfo():
        Total space in arena: 1101288 KB
        Ordinary blocks: 1098718 KB 287 blks
        Small blocks: 0 KB 0 blks
        Holding blocks: 17636 KB 9 blks
        Free Small blocks: 0 KB
        Free Ordinary blocks: 2569 KB
        Total in use: 1116354 KB 100%
        Total free: 2569 KB 0%
        Total size: 1118924 KB
Memory accounted for:
        Total accounted: 284679 KB 25%
        memPool accounted: 284679 KB 25%
        memPool unaccounted: 834245 KB 75%
        memPoolAlloc calls: 620095300
        memPoolFree calls: 640694955

Yet in ps, I see this:

squid 2576 1.1 32.5 5388508 5314184 ? S Sep05 92:51 (squid) -f /etc/squid/squid.conf

So, squid is using 5GB of RAM yet it only knows about 1 of those GB.

Thanks
-Ty

On Sep 26, 2012, at 11:32 AM, Eliezer Croitoru <eliezer_at_ngtech.co.il> wrote:

> On 9/26/2012 8:11 PM, tcr_at_raynersw.com wrote:
>> Reposting this because I posted the original via the nabble site and it didn't get mailed out to the list....
>>
>>
>> I posted a while ago about this but didn't get any replies. I run squid servers which handle heavy sustained traffic (50-100Mbps per server is not unusual). The squid processes' memory footprints all grow without bounds. I am running various different Squid versions from 2.7 to 3.2.1 on the different servers, and all versions exhibit this problem.
>>
>> I have tried all the debugging steps I've been able to find- playing around with cache sizes, changing the memory directives in the conf files, etc. It just looks like a plain old leak though, because these processes will exhaust 16GB of RAM over the course of a week. When I look at the cache mgr info page, its accounting of memory usage is nowhere near the actual size shown in top.
>>
>> As for my config, the servers are IPv4 only, they grant access based on a lengthy ACL of allowed IPs. This ACL comes from a text file.
>>
>> If anybody has any ideas, I'd really appreciate it. If not, can someone at least tell me how to best go about debugging a leak in squid? I am ok with poking around in valgrind or whatever, but I'm hoping someone can steer me in the right direction to start.
>>
>> Thank you.
>> -Ty
>>
> Post your squid.conf and other data that you have.
> commands you have used?
> You should notice that "top" shows Virt and RES which is a bit different.
> my server is using about 800MB of ram and free+top shows about 1300-1400MB used due to somethings.
> you can try to use HTOP to see a more colorful picture that can give you another angle on the memory usage.
>
> 50-100Mbps is not how you should measure traffic (in most cases) load but more in Requests Per Second.
> I have one server that serves more then 100Mps but with only couple RPS which only strains some memory and cpu for the task.
>
> you can also take a look at your squid proxy used such as proxy using:
> top -u proxy
> which will give you only the squid user process.
> then you can calculate the VIRT RES SHR and %MEM manually to see what and how.
> for example:
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 7765 proxy 20 0 215m 153m 2744 S 0 7.7 3:29.51 squid
> 7768 proxy 20 0 107m 44m 2496 S 0 2.3 0:44.98 squid
> 7794 proxy 20 0 30708 380 156 S 0 0.0 0:00.46 unlinkd
> 8608 proxy 20 0 40600 9328 1236 S 0 0.5 0:00.90 run.rb
> 8610 proxy 20 0 40932 9788 1408 S 0 0.5 0:00.92 run.rb
>
> you see that one squid uses 7.7 % and other one 2.3(of 2GB)
>
> Regards,
> Eliezer
> --
> Eliezer Croitoru
> https://www1.ngtech.co.il
> IT consulting for Nonprofit organizations
> eliezer <at> ngtech.co.il
Received on Wed Sep 26 2012 - 19:09:09 MDT