Re: [squid-users] Web access through transparent proxy(squid) getting slow.

Igor Luiz Oliveira de Souza wrote:
> Hello folks,
>
> Since a long time I'm experiencing a problem with Squid and after a
> long work, I didn't find the point.
> I believe this must be recurring discussion, but seems to be a kind of
> particular tunning on my enviroments to get my problem solved, so here
> we go with my history:
>
> Around two months ago, I started a patterned configuration of Squid on
> 15 servers over 10 different ISP's which I'm network manager. The
> number of users served by each server varies around 150 to 500 users
> behind them. The bandwidth on each ISP varies around 4 to 30 Mbps. I
> know that this can be a big range of variation, but I'm telling you
> that, only to give an overview about the dimension of enviroments we
> are talking about.
>
> On the first 10 or 15 days, all of then were running perfectly... no
> matter, no issues!
> After that till now, 4 of these(other 11 are still working perfectly)
> servers starts with the same symptom:
> - The browsing becomes lazy till it almost stop's(in some cases get
> back working with no intervention - intermittently)
> The proxies are running in transparent mode, so if I just disable the
> redirect on netfilter, making the traffic goes directly to the
> net(masquerading), the browsing get fast again!
> I'm using almost the same config in all the servers, basically only
> changing the cache_dir based on the size of the disk.
> The main intention of those servers are to save bandwidht usage, but
> keeping the balance with the response time to the users, in a way that
> using proxy must still be faster than directly connect.
> All the servers are running Squid v.2.6.stable22.
>
> I've already checked CPU Load, Memory usage, disk I/O, and in my
> humble analyse, I didn't find the point.
>
> I'll put here some info's about the system of one of those problematic
> servers, so if anyone could find some clue, I'll be very gratefull.

I'm not sure at all what the cause of slowness might be. Might be garage
collection related, or inefficient handling of events under low client
counts.

Regarding garbage collection:
   On your 500GB cache the default 5% garbage reduction could be trying
to erase 25GB worth of ~20KB objects. This is naturally a long slow
process to be happening in the background.
  The cache_swap_high and cache_swap_low settings help reduce this
drain. For a cache over 10GB they need to be set around 1% different or
even identical to get reasonable garbage collection.

> ==================================================================
> => Hardware Configs
> --------------------------------
> CPU: Intel Core 2 Quad - 2.66GHz
> RAM: 8 Gb
>
> => OS
> --------------------------------
> Linux Slackware v.13 - Kernel 2.6.29.6-smp
>
> => SQUID CONF (Squid v.2.6.stable22)
> --------------------------------
> acl all src 0.0.0.0/0.0.0.0
> acl manager proto cache_object
> acl localhost src 127.0.0.1/255.255.255.255
> acl to_localhost dst 127.0.0.0/8
> acl SSL_ports port 443
> acl Safe_ports port 80 # http
> acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 # https
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http
> acl Safe_ports port 591 # filemaker
> acl Safe_ports port 777 # multiling http
> acl CONNECT method CONNECT
>
> http_access allow manager localhost
> http_access deny manager
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
>
> acl our_networks src xxx.xxx.xxx.xxx/16
> acl this_server src yyy.yyy.yyy.yyy/32
> http_access allow our_networks
>
> http_access allow localhost
> http_access deny all
>
> icp_access allow all
>
> http_port 3128 transparent
>
> hierarchy_stoplist cgi-bin ?
>
> cache_mem 512 MB
>
> maximum_object_size_in_memory 384 KB
>
> memory_replacement_policy heap GDSF
>
> cache_replacement_policy heap LFUDA
>
> cache_dir aufs /cache 500000 64 256
>
> maximum_object_size 400 MB
>
> access_log /usr/local/squid/var/logs/access.log squid
>
> cache_log /usr/local/squid/var/logs/cache.log
>
> cache_store_log none
>
> logfile_rotate 3
>
> acl QUERY urlpath_regex cgi-bin \?
> cache deny QUERY

NP: you might get a bit more data stored by removing the two above and
using the new default refresh pattern below.

>
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440

Missing the new:
  refresh_pattern -i (/cgi-bin/|\?) 0 0% 0

> refresh_pattern . 0 20% 4320
>
> acl apache rep_header Server ^Apache
> broken_vary_encoding allow apache
>
> cache_mgr support_at_mydomain.com.br
>
> visible_hostname yyy.yyy.yyy.yyy
>
> snmp_port 3401
> acl snmppublic snmp_community public
> snmp_access allow snmppublic this_server
> snmp_access deny all
> snmp_incoming_address 0.0.0.0
> snmp_outgoing_address 255.255.255.255
>
> coredump_dir /cache
>
> => Squid CACHE MANAGER INFO:
> --------------------------------
> Squid Object Cache: Version 2.6.STABLE22
> Start Time: Sat, 02 Jan 2010 18:08:57 GMT
> Current Time: Sat, 02 Jan 2010 21:10:19 GMT
> Connection information for squid:
> Number of clients accessing cache: 9
> Number of HTTP requests received: 277856
> Number of ICP messages received: 0
> Number of ICP messages sent: 0
> Number of queued ICP replies: 0
> Request failure ratio: 0.00
> Average HTTP requests per minute since start: 1532.1
> Average ICP messages per minute since start: 0.0
> Select loop called: 3581679 times, 3.038 ms avg
> Cache information for squid:
> Request Hit Ratios: 5min: 35.5%, 60min: 39.7%
> Byte Hit Ratios: 5min: 17.0%, 60min: 17.0%
> Request Memory Hit Ratios: 5min: 9.0%, 60min: 5.7%
> Request Disk Hit Ratios: 5min: 52.4%, 60min: 48.6%
> Storage Swap size: 56684352 KB
> Storage Mem size: 513584 KB
> Mean Object Size: 20.48 KB
> Requests given to unlinkd: 0
> Median Service Times (seconds) 5 min 60 min:
> HTTP Requests (All): 0.18699 0.15888
> Cache Misses: 0.30459 0.30459
> Cache Hits: 0.01309 0.00562
> Near Hits: 0.32154 0.08265
> Not-Modified Replies: 0.00286 0.00286
> DNS Lookups: 0.13042 0.12472
> ICP Queries: 0.00000 0.00000
> Resource usage for squid:
> UP Time: 10881.453 seconds
> CPU Time: 127.668 seconds
> CPU Usage: 1.17%
> CPU Usage, 5 minute avg: 1.30%
> CPU Usage, 60 minute avg: 0.91%
> Process Data Segment Size via sbrk(): 851000 KB
> Maximum Resident Size: 0 KB
> Page faults with physical i/o: 0
> Memory usage for squid via mallinfo():
> Total space in arena: 851000 KB
> Ordinary blocks: 850398 KB 31794 blks
> Small blocks: 0 KB 0 blks
> Holding blocks: 27348 KB 5 blks
> Free Small blocks: 0 KB
> Free Ordinary blocks: 601 KB
> Total in use: 877746 KB 100%
> Total free: 601 KB 0%
> Total size: 878348 KB
> Memory accounted for:
> Total accounted: 726322 KB
> memPoolAlloc calls: 51472567
> memPoolFree calls: 44989921
> File descriptor usage for squid:
> Maximum number of file descriptors: 16384
> Largest file desc currently in use: 1153
> Number of file desc currently in use: 972
> Files queued for open: 0
> Available number of file descriptors: 15412
> Reserved number of file descriptors: 100
> Store Disk files open: 69
> IO loop method: epoll
> Internal Data Structures:
> 2767942 StoreEntries
> 36519 StoreEntries with MemObjects
> 36391 Hot Object Cache Items
> 2767222 on-disk objects
>
> => 'top' COMMAND OUTPUT:
> --------------------------------
> top - 18:14:18 up 2 days, 23:34, 1 user, load average: 0.26, 0.17, 0.08
> Tasks: 113 total, 1 running, 112 sleeping, 0 stopped, 0 zombie
> Cpu0 : 0.0%us, 0.3%sy, 0.0%ni, 98.7%id, 1.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu1 : 0.0%us, 0.3%sy, 0.0%ni, 97.7%id, 2.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu2 : 0.3%us, 0.0%sy, 0.0%ni, 99.3%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu3 : 0.3%us, 0.0%sy, 0.0%ni, 99.0%id, 0.3%wa, 0.0%hi, 0.3%si, 0.0%st
> Mem: 8305904k total, 3632340k used, 4673564k free, 525752k buffers
> Swap: 6297440k total, 0k used, 6297440k free, 1980300k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 12561 nobody 20 0 874m 863m 2372 S 0 10.6 2:09.52 squid
>
> => 'free' COMMAND OUTPUT:
> --------------------------------
> total used free shared buffers cached
> Mem: 8305904 3648668 4657236 0 527276 1991908
> -/+ buffers/cache: 1129484 7176420
> Swap: 6297440 0 6297440
>
> => 'vmstat' COMMAND OUTPUT:
> --------------------------------
> root_at_proxy:/usr/local/squid/etc# vmstat 1
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 0 0 0 4655236 527384 1993244 0 0 12 37 8 24 0 0 98 1
> 0 0 0 4654980 527384 1993292 0 0 0 504 4612 368 0 0 99 0
> 0 0 0 4654848 527424 1993524 0 0 40 0 4816 671 0 0 98 2
> 0 0 0 4654724 527436 1993712 0 0 168 0 4740 522 0 0 99 1
> 0 0 0 4654352 527448 1993876 0 0 0 76 4818 487 0 0 100 0
> 0 0 0 4654344 527460 1993976 0 0 4 0 4718 500 0 0 100 0
> 0 0 0 4654220 527468 1994120 0 0 24 588 4579 398 0 0 99 1
> 0 0 0 4653848 527480 1994332 0 0 104 0 4853 627 0 0 99 1
> 0 0 0 4653360 527500 1994532 0 0 0 0 4909 683 0 0 100 0
> 0 0 0 4653212 527516 1994716 0 0 0 40 4877 447 0 0 100 0
> 0 0 0 4652964 527536 1994732 0 0 12 0 4721 518 0 0 100 0
> 0 0 0 4652964 527548 1994800 0 0 16 820 4810 708 0 1 99 1
> 0 0 0 4652716 527572 1994936 0 0 20 0 4696 628 0 0 99 1
>
> ==================================================================
>
> So, with that info's, anyone would suggest something wrong?
>
> Sorry about the long mail, but I'm just trying to be detailed enough
> so you could help me.
>

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE21
   Current Beta Squid 3.1.0.15