Re: [squid-users] High CPU utilization

jotacekm wrote:
>
>
> Amos Jeffries-2 wrote:
>> On Mon, 27 Jul 2009 12:14:29 -0700 (PDT), jotacekm <minuzzo_at_viaip.com.br>
>> wrote:
>>> Hello.
>>> Recently we have added a lot more clientes behind a squid proxy, and now
>>> cpu
>>> utilizitaion is usually 70-95%. The processor is a intel dual core 2160
>> @
>>> 1.80GHz. Users started complaing about the speed on accessing pages, and
>>> the
>>> link is fine.
>>>
>>> Here is squidclient mgr:info:
>>>
>>> Squid Object Cache: Version 2.6.STABLE5
>>> Start Time: Fri, 24 Jul 2009 20:19:16 GMT
>>> Current Time: Mon, 27 Jul 2009 19:09:07 GMT
>>> Connection information for squid:
>>> Number of clients accessing cache: 1561
>>> Number of HTTP requests received: 8590404
>>> Number of ICP messages received: 0
>>> Number of ICP messages sent: 0
>>> Number of queued ICP replies: 0
>>> Number of HTCP messages received: 0
>>> Number of HTCP messages sent: 0
>>> Request failure ratio: 0.00
>>> Average HTTP requests per minute since start: 2021.3
>> 2k req/min is getting close to what people report for saturation of older
>> 2.6 forward proxies.
>> First performance improvement will be simply an upgrade to 2.7.
>>
>>
>>> Average ICP messages per minute since start: 0.0
>>> Select loop called: 122560206 times, 2.081 ms avg
>>> Cache information for squid:
>>> Request Hit Ratios: 5min: 23.0%, 60min: 21.9%
>>> Byte Hit Ratios: 5min: 13.3%, 60min: 14.7%
>>> Request Memory Hit Ratios: 5min: 16.3%, 60min: 17.0%
>>> Request Disk Hit Ratios: 5min: 26.3%, 60min: 29.9%
>>> Storage Swap size: 4609784 KB
>>> Storage Mem size: 65692 KB
>>> Mean Object Size: 15.67 KB
>>> Requests given to unlinkd: 455490
>>> Median Service Times (seconds) 5 min 60 min:
>>> HTTP Requests (All): 1.24267 1.46131
>>> Cache Misses: 1.54242 1.81376
>>> Cache Hits: 0.28853 0.30459
>>> Near Hits: 1.31166 1.62803
>>> Not-Modified Replies: 0.23230 0.23230
>>> DNS Lookups: 0.29097 0.31806
>>> ICP Queries: 0.00000 0.00000
>>> Resource usage for squid:
>>> UP Time: 254991.358 seconds
>>> CPU Time: 56029.670 seconds
>>> CPU Usage: 21.97%
>>> CPU Usage, 5 minute avg: 94.38%
>>> CPU Usage, 60 minute avg: 93.36%
>>> Process Data Segment Size via sbrk(): 193008 KB
>>> Maximum Resident Size: 0 KB
>>> Page faults with physical i/o: 70
>>> Memory usage for squid via mallinfo():
>>> Total space in arena: 193008 KB
>>> Ordinary blocks: 179871 KB 6058 blks
>>> Small blocks: 0 KB 0 blks
>>> Holding blocks: 1080 KB 2 blks
>>> Free Small blocks: 0 KB
>>> Free Ordinary blocks: 13136 KB
>>> Total in use: 180951 KB 93%
>>> Total free: 13136 KB 7%
>>> Total size: 194088 KB
>>> Memory accounted for:
>>> Total accounted: 111416 KB
>>> memPoolAlloc calls: 973288506
>>> memPoolFree calls: 972077571
>>> File descriptor usage for squid:
>>> Maximum number of file descriptors: 4096
>>> Largest file desc currently in use: 1604
>>> Number of file desc currently in use: 1308
>>> Files queued for open: 0
>>> Available number of file descriptors: 2788
>>> Reserved number of file descriptors: 100
>>> Store Disk files open: 30
>>> IO loop method: epoll
>>> Internal Data Structures:
>>> 300182 StoreEntries
>>> 14733 StoreEntries with MemObjects
>>> 14414 Hot Object Cache Items
>>> 294106 on-disk objects
>>>
>>> And here is part of of squid.conf:
>>>
>>> http_port 3128
>>> visible_hostname xxx
>>> hierarchy_stoplist cgi-bin ?
>>> acl QUERY urlpath_regex cgi-bin \?
>>> cache deny QUERY
>> _after_ upgrading to the most recent 2.6 or a 2.7. Remove the QUERY
>> entries
>> above.
>>
>>> acl apache rep_header Server ^Apache
>>> broken_vary_encoding allow apache
>>> access_log /var/log/squid/access.log squid
>>> cache_store_log none
>>> hosts_file /etc/hosts
>>>
>>>
>>> #
>>>
>> ----------------------------------------------------------------------------------------------------------
>>> cache_mem 64 MB
>>> cache_dir ufs /var/spool/squid 5000 50 256
>> #1 change that to aufs if using a Linux or variants. diskd if using BSD
>> or
>> variants. Change to Linux or BSD if using Windows.
>>
>> #2 up the cache size as much as you can. Local objects are much faster to
>> retrieve than external ones.
>> The more local objects you can store the faster Squid responds.
>> From my experience its around an order of magnitude or more req/sec
>> handled
>> from cache as from network.
>>
>> Memory cache is better for speed than disk, but can be lost more easily
>> and
>> you can get bigger cache total spending RAM on indexes and disk on object
>> storage.
>>
>> After the upgrade to 2.7 you have access to a stable COSS system which is
>> very much more efficient for the small files. It's worth a try.
>>
>>> cache_replacement_policy heap LFUDA
>>> maximum_object_size 51200 KB
>> There is good evidence that the web now has a lot of traffic in the 2MB to
>> 250MB range from multimedia objects.
>> This will bear investigating in your traffic and see if the limit above is
>> still good.
>>
>>
>>> maximum_object_size_in_memory 64 KB
>>> memory_replacement_policy heap GDSF
>>> logfile_rotate 3
>>>
>>>
>>> #
>>>
>> ----------------------------------------------------------------------------------------------------------
>>> refresh_pattern ^ftp: 1440 20% 10080
>>> refresh_pattern ^gopher: 1440 0% 1440
>> Add:
>> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
>>
>>> refresh_pattern . 0 20% 4320
>>> acl all src 0.0.0.0/0.0.0.0
>>> acl manager proto cache_object
>>> acl localhost src 127.0.0.1/255.255.255.255
>>> acl to_localhost dst 127.0.0.0/8
>>>
>>>
>>>
>>> max_open_disk_fds 2046
>> Definitely raise that. Your stats show that many _new_ requests arriving
>> every second.
>> Each single request eats 2-3 FD and may last from seconds to hours.
>>
>>> # timeouts
>>> connect_timeout 30 seconds
>>> shutdown_lifetime 5 seconds
>>> forward_timeout 2 minutes
>>> pconn_timeout 30 seconds
>> You _want_ pconn to last a fairly long while. There is no CPU or bandwidth
>> overhead.
>> Possibly also enabling client and server persistent connections.
>>
>>> persistent_request_timeout 1 minute
>>> request_timeout 2 minute
>>>
>>> Is there anything that i can do to lower the cpu utilization, or do i
>> have
>>> to upgrade the hardware?
>> There are a number of other perfromance optimizations possible within the
>> build options and in other squid.conf settings. You don't say what they
>> are
>> set at, so we can't suggest any right now.
>>
>> Amos
>>
>>
>
> Hello,
> Thanks for you help. I'll try the modifications you mentioned and report
> here if they worked. About the build options, this squid was not compiled,
> it was installed via apt-get in a debian etch. And the other squid.conf
> options are just acls and allows / denies.
>

Yes, but... not all ACL are equally fast. The order called matters and
there are tricks with layering types of ACL to prevent slow ones being
tried uselessly.

For pre-packaged build, there is not much help. The Debian settings
aren't bad though.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE6 or 3.0.STABLE17
   Current Beta Squid 3.1.0.12