Re: [squid-users] Squid performance issues

Felipe W Damasio wrote:
> Hi all,
>
> Sorry for the long email.
>
> I'm using squid on a 300Mbps ISP with about 10,000 users.
>
> I have an 8-core I7 Intel processor-machine, with 8GB of RAM and 500
> of HD for the cache. (exclusive Sata HD with xfs). Using aufs as
> storeio.
>
> I'm caching mostly multimedia files (youtube and such).
>
> Squid usually eats around 50-70% of one core.
>
> But always around midnight (when a lot of users browse the internet),
> my squid becomes very slow....I mean, a page that usually takes 0.04s
> to load takes 23seconds to load.
>
> My best guess is that the volume of traffic is making squid slow.
>
> I'm using a 2.6.29.6 vanilla kernel with tproxy enabled for squid.
> And I'm using these /proc configurations:
>
> echo 0 > /proc/sys/net/ipv4/tcp_ecn
> echo 1 > /proc/sys/net/ipv4/tcp_low_latency
> echo 100000 > /proc/sys/net/core/netdev_max_backlog
> echo 409600 > /proc/sys/net/ipv4/tcp_max_syn_backlog
> echo 7 > /proc/sys/net/ipv4/tcp_fin_timeout
> echo 15 > /proc/sys/net/ipv4/tcp_keepalive_intvl
> echo 3 > /proc/sys/net/ipv4/tcp_keepalive_probes
> echo 65536 > /proc/sys/vm/min_free_kbytes
> echo "262144 1024000 4194304" > /proc/sys/net/ipv4/tcp_rmem
> echo "262144 1024000 4194304" > /proc/sys/net/ipv4/tcp_wmem
> echo "1024000" > /proc/sys/net/core/rmem_max
> echo "1024000" > /proc/sys/net/core/wmem_max
> echo "512000" > /proc/sys/net/core/rmem_default
> echo "512000" > /proc/sys/net/core/wmem_default
> echo "524288" > /proc/sys/net/ipv4/netfilter/ip_conntrack_max
> echo "3" > /proc/sys/net/ipv4/tcp_synack_retries
>
> The machine is in bridge-mode.
>
> I wrote a little script that prints:
>
> - The date;
> - The "/usr/bin/time squidclient http://www.amazon.com";
> - The number of ESTABLISHED connections (through netstat -an);
> - The number of TIME_WAIT connections;
> - The total number of netstat connections;
> - The route cache (ip route list cache);
> - The number of clients currently connected in squid (through mgr:info);
> - The number of free memory in MB (free -m);
> - The % used of the squid-running core;
> - The average number of time to respond a request / sec (mgr:info
> also) - 5 minutes avg;
> - The average number of http requests / sec (5 minutes avg) - mgr:info as well.
>
> On any other hour, I have something like:
>
> 2010-01-25 18:48:19 ; 0.04 ; 19383 ; 9902 ; 29865 ; 96972 ; 4677 ; 131
> ; 59 ; 0.24524 ; 476.871718
> 2010-01-25 18:53:29 ; 0.04 ; 18865 ; 8593 ; 30123 ; 179570 ; 4679 ;
> 148 ; 62 ; 0.22004 ; 504.424207
> 2010-01-25 18:58:38 ; 0.04 ; 18377 ; 9056 ; 29283 ; 99038 ; 4680 ; 174
> ; 61 ; 0.22004 ; 466.659336
> 2010-01-25 19:03:49 ; 0.04 ; 18877 ; 9133 ; 28327 ; 181196 ; 4673 ;
> 171 ; 57 ; 0.24524 ; 483.558436
>
> So, it takes around 0.04s to get http://www.amazon.com.
>
> 2010-01-24 23:46:50 ; 2.53 ; 22723 ; 9861 ; 35012 ; 64752 ; 4306 ;
> 166; 70 ; 0.22004 ; 566.364274
> 2010-01-24 23:52:04 ; 3.74 ; 21173 ; 10256 ; 33242 ; 167594 ; 4309 ;
> 169 ; 68 ; 0.20843 ; 537.758601
> 2010-01-24 23:57:20 ; 0.08 ; 18691 ; 9050 ; 29590 ; 65496 ; 4312 ; 138
> ; 71 ; 0.20843 ; 525.119006
> 2010-01-25 00:02:29 ; 15.54 ; 18016 ; 8209 ; 29035 ; 149248 ; 4318 ;
> 160 ; 82 ; 0.25890 ; 491.615241
>
> As I said, it goes from 0.04 to 15.54s(!) to get a single html file.
> Horrible. After 12:30, everything goes back to normal.
>
> From those variables, I can't seem to find any indication of what can
> be causing this appalling slowdown. The number of squid users doesn't
> go up that much, I just see that the avg time squid reports to
> answering a request goes from 0.20s to 0.25, and the number of http
> requests/sec actually goes down from 566 to 491...which is kind of odd
> to me. And the number users using squid stays in aroung 4300.
>
> I talked to Mr. Dave Dykstra, and he thought it could be I/O delay
> issues. So I tried:
>
> cache_dir null /tmp
> cache_access_log none
> cache_store_log none
>
> But no luck, on midnight tonight again things went wild:
>
> 2010-01-25 23:57:03 ; 0.04 ; 24112 ; 11330 ; 37240 ; 74456 ; 3516 ;
> 160 ; 58 ; 0.25890 ; 581.047037
> 2010-01-26 00:02:15 ; 10.82 ; 25638 ; 11695 ; 38537 ; 177198 ; 3533 ;
> 149 ; 78 ; 0.27332 ; 570.312936
> 2010-01-26 00:07:38 ; 42.64 ; 23818 ; 11563 ; 38097 ; 88902 ; 3556 ;
> 171 ; 70 ; 0.30459 ; 585.880418
>
> From 0.04 to 42 seconds to load the main html page of amazon.com. (!)
>
> Do you have any idea or any other data I can collect to try and
> track down this?
>

Check your log rotation schedule. Is it possible that logs are being
rotated at midnight? I think that the swap.state file is rewritten
when "squid -k rotate" is called. Check the beginning of your cache.log
to verify.

> I'm using squid-2.7.stable7, but I'm willing to try squid-3.0 or
> squid-3.1 if you guys think it could help.
>
> I'm using 2 gigabit Marvell Ethernet boards with sky2 driver. Don't
> know if it's relevant, though.
>
> If you guys need any more info to try and help me figure this out, please ask.
>
> I'm willing to test, code or do pretty much anything to make squid
> perform better on my environment Please let me know how can I help you
> help me. :-)
>
> Thanks!
>
> Felipe Damasio
>

Chris
Received on Tue Jan 26 2010 - 19:16:54 MST