Re: [squid-users] High CPU utilization

On Mon, 27 Jul 2009 12:14:29 -0700 (PDT), jotacekm <minuzzo_at_viaip.com.br>
wrote:
> Hello.
> Recently we have added a lot more clientes behind a squid proxy, and now
> cpu
> utilizitaion is usually 70-95%. The processor is a intel dual core 2160
@
> 1.80GHz. Users started complaing about the speed on accessing pages, and
> the
> link is fine.
>
> Here is squidclient mgr:info:
>
> Squid Object Cache: Version 2.6.STABLE5
> Start Time: Fri, 24 Jul 2009 20:19:16 GMT
> Current Time: Mon, 27 Jul 2009 19:09:07 GMT
> Connection information for squid:
> Number of clients accessing cache: 1561
> Number of HTTP requests received: 8590404
> Number of ICP messages received: 0
> Number of ICP messages sent: 0
> Number of queued ICP replies: 0
> Number of HTCP messages received: 0
> Number of HTCP messages sent: 0
> Request failure ratio: 0.00
> Average HTTP requests per minute since start: 2021.3

2k req/min is getting close to what people report for saturation of older
2.6 forward proxies.
First performance improvement will be simply an upgrade to 2.7.

> Average ICP messages per minute since start: 0.0
> Select loop called: 122560206 times, 2.081 ms avg
> Cache information for squid:
> Request Hit Ratios: 5min: 23.0%, 60min: 21.9%
> Byte Hit Ratios: 5min: 13.3%, 60min: 14.7%
> Request Memory Hit Ratios: 5min: 16.3%, 60min: 17.0%
> Request Disk Hit Ratios: 5min: 26.3%, 60min: 29.9%
> Storage Swap size: 4609784 KB
> Storage Mem size: 65692 KB
> Mean Object Size: 15.67 KB
> Requests given to unlinkd: 455490
> Median Service Times (seconds) 5 min 60 min:
> HTTP Requests (All): 1.24267 1.46131
> Cache Misses: 1.54242 1.81376
> Cache Hits: 0.28853 0.30459
> Near Hits: 1.31166 1.62803
> Not-Modified Replies: 0.23230 0.23230
> DNS Lookups: 0.29097 0.31806
> ICP Queries: 0.00000 0.00000
> Resource usage for squid:
> UP Time: 254991.358 seconds
> CPU Time: 56029.670 seconds
> CPU Usage: 21.97%
> CPU Usage, 5 minute avg: 94.38%
> CPU Usage, 60 minute avg: 93.36%
> Process Data Segment Size via sbrk(): 193008 KB
> Maximum Resident Size: 0 KB
> Page faults with physical i/o: 70
> Memory usage for squid via mallinfo():
> Total space in arena: 193008 KB
> Ordinary blocks: 179871 KB 6058 blks
> Small blocks: 0 KB 0 blks
> Holding blocks: 1080 KB 2 blks
> Free Small blocks: 0 KB
> Free Ordinary blocks: 13136 KB
> Total in use: 180951 KB 93%
> Total free: 13136 KB 7%
> Total size: 194088 KB
> Memory accounted for:
> Total accounted: 111416 KB
> memPoolAlloc calls: 973288506
> memPoolFree calls: 972077571
> File descriptor usage for squid:
> Maximum number of file descriptors: 4096
> Largest file desc currently in use: 1604
> Number of file desc currently in use: 1308
> Files queued for open: 0
> Available number of file descriptors: 2788
> Reserved number of file descriptors: 100
> Store Disk files open: 30
> IO loop method: epoll
> Internal Data Structures:
> 300182 StoreEntries
> 14733 StoreEntries with MemObjects
> 14414 Hot Object Cache Items
> 294106 on-disk objects
>
> And here is part of of squid.conf:
>
> http_port 3128
> visible_hostname xxx
> hierarchy_stoplist cgi-bin ?
> acl QUERY urlpath_regex cgi-bin \?
> cache deny QUERY

_after_ upgrading to the most recent 2.6 or a 2.7. Remove the QUERY entries
above.

> acl apache rep_header Server ^Apache
> broken_vary_encoding allow apache
> access_log /var/log/squid/access.log squid
> cache_store_log none
> hosts_file /etc/hosts
>
>
> #
>
----------------------------------------------------------------------------------------------------------
> cache_mem 64 MB
> cache_dir ufs /var/spool/squid 5000 50 256

#1 change that to aufs if using a Linux or variants. diskd if using BSD or
variants. Change to Linux or BSD if using Windows.

#2 up the cache size as much as you can. Local objects are much faster to
retrieve than external ones.
The more local objects you can store the faster Squid responds.
From my experience its around an order of magnitude or more req/sec handled
from cache as from network.

Memory cache is better for speed than disk, but can be lost more easily and
you can get bigger cache total spending RAM on indexes and disk on object
storage.

After the upgrade to 2.7 you have access to a stable COSS system which is
very much more efficient for the small files. It's worth a try.

> cache_replacement_policy heap LFUDA
> maximum_object_size 51200 KB

There is good evidence that the web now has a lot of traffic in the 2MB to
250MB range from multimedia objects.
This will bear investigating in your traffic and see if the limit above is
still good.

> maximum_object_size_in_memory 64 KB
> memory_replacement_policy heap GDSF
> logfile_rotate 3
>
>
> #
>
----------------------------------------------------------------------------------------------------------
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440

Add:
  refresh_pattern -i (/cgi-bin/|\?) 0 0% 0

> refresh_pattern . 0 20% 4320
> acl all src 0.0.0.0/0.0.0.0
> acl manager proto cache_object
> acl localhost src 127.0.0.1/255.255.255.255
> acl to_localhost dst 127.0.0.0/8
>
>
>
> max_open_disk_fds 2046

Definitely raise that. Your stats show that many _new_ requests arriving
every second.
Each single request eats 2-3 FD and may last from seconds to hours.

>
> # timeouts
> connect_timeout 30 seconds
> shutdown_lifetime 5 seconds
> forward_timeout 2 minutes
> pconn_timeout 30 seconds

You _want_ pconn to last a fairly long while. There is no CPU or bandwidth
overhead.
Possibly also enabling client and server persistent connections.

> persistent_request_timeout 1 minute
> request_timeout 2 minute
>
> Is there anything that i can do to lower the cpu utilization, or do i
have
> to upgrade the hardware?

There are a number of other perfromance optimizations possible within the
build options and in other squid.conf settings. You don't say what they are
set at, so we can't suggest any right now.

Amos
Received on Tue Jul 28 2009 - 01:48:17 MDT