RE: [squid-users] Cache manager analysis

On Sun, 21 Feb 2010 11:43:12 +0000, "J. Webster"
<webster_jack_at_hotmail.com>
wrote:
> Does this look reasonable?

Mostly. I can just see one operational issue remaining...

> I still have the analysis to start with after this point but will use
some
> linux tools to help with that...
>
> auth_param basic realm P*****r ProxyServer
> auth_param basic credentialsttl 2 hours
> auth_param basic program /usr/lib/squid/ncsa_auth
/etc/squid/squid_passwd
> authenticate_cache_garbage_interval 1 hour
> authenticate_ip_ttl 2 hours
> #acl all src 0.0.0.0/0.0.0.0
> acl all src all
> acl manager proto cache_object
> acl localhost src 127.0.0.1
> acl cacheadmin src 88.xxx.xxx.xxx 127.0.0.1
> acl to_localhost dst 127.0.0.0/8 0.0.0.0/32
> acl SSL_ports port 443
> acl Safe_ports port 80 # http
> acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 # https
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http
> acl Safe_ports port 591 # filemaker
> acl Safe_ports port 777 # multiling http
> acl Safe_ports port 1863 # MSN messenger
> acl ncsa_users proxy_auth REQUIRED
> acl maxuser max_user_ip -s 2
> acl CONNECT method CONNECT
> http_access allow manager cacheadmin
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access deny to_localhost
> http_access deny manager
> http_access allow ncsa_users
> http_access deny maxuser

Um, the maxuser test will not be used yet, because any user who logs in
will be accepted by the ncsa_users line.

What I'd do here is combine the two:

  http_access allow !maxuser ncsa_users

or if that does not work:
 http_access allow ncsa_users !maxuser
 http_access deny !ncsa_users

> http_access deny all
> icp_access allow all
> http_port 8080
> http_port 88.xxx.xxx.xxx:80
> hierarchy_stoplist cgi-bin ?
> cache_mem 256MB
> maximum_object_size_in_memory 50 KB
> cache_replacement_policy heap LFUDA
> cache_dir aufs /var/spool/squid 40000 16 256
> maximum_object_size 50 MB
> cache_swap_low 90
> cache_swap_high 95
> access_log /var/log/squid/access.log squid
> cache_log /var/log/squid/cache.log
> buffered_logs on
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
> refresh_pattern . 0 20% 4320
> quick_abort_min 0 KB
> quick_abort_max 0 KB
> acl apache rep_header Server ^Apache
> broken_vary_encoding allow apache
> half_closed_clients off
> cache_mgr ***'***.com
> cachemgr_passwd ******** all
> visible_hostname P*****r ProxyServer
> log_icp_queries off
> dns_nameservers 208.67.222.222 208.67.220.220
> hosts_file /etc/hosts
> memory_pools off
> forwarded_for off
> client_db off
> coredump_dir /var/spool/squid
>
>> ----------------------------------------
>>> From: webster_jack_at_hotmail.com
>>> To: squid3_at_treenet.co.nz; squid-users_at_squid-cache.org
>>> Date: Sat, 13 Feb 2010 16:35:29 +0000
>>> Subject: RE: [squid-users] Cache manager analysis
>>>
>>>
>>> Thanks.
>>> A few questions on this:
>>> (a) when you said this all src all is that meant to be acl src all?
>>> (b) Hint 2: if possible, define an ACL or the network ranges where you
>>> accept logins. Use it like so
>>> The logins are accepted form IP addresses that I never know, it is an
>>> external proxy server for geo location so not sure I can do this?
logins
>>> will only ever by directed to the 88.xxx.xxx.xxx server though?
>>> (c) cache_mem 100 MB
>>> Bump this up as high as you can go without risking memory swapping.
>>> Objects served from RAM are 100x faster than objects not.
>>> Where can I view if memeory swapping is happening?
>>> (D) maximum_object_size 50 MB
>>> Bump this up too. Holding full ISO CDs and windows service packs can
>>> boost performance when one is used from the cache. 40GB of disk can
>>> store a few.
>>> If I increase this, will the server ever try to store streamed video?
I
>>> had an efficiency problem with the original configuration that came
with
>>> squid, which meant that streamed video was buffering constantly. Not
>>> sure what caused it but with the current config it does not do that.
>>> If I increase the cache_mem and max object size do I also need to
>>> increase this?
>>> maximum_object_size_in_memory 50 KB
>>> (E)
>>> cache_swap_low 90
>>> cache_swap_high 95
>>> access_log /var/log/squid/access.log squid
>>> cache_log /var/log/squid/cache.log
>>> buffered_logs on
>>> acl QUERY urlpath_regex cgi-bin \?
>>> cache deny QUERY
>>>
>>> Drop the QUERY bits above. It's more than halving the things your
Squid
>>> can store.
>>> Remove the acl and the cache deny?
>>> At present, does this stop the cache from storing anything with a ?,
ie
>>> dynamic pages?
>>> What if the same request is made for a dynamic page, will it retrive
it
>>> from the cache (old page) rather then fetch the new dynamic content?
>>>
>>> current conf redone below:
>>> ----------------------------
>>> auth_param basic realm Proxy server
>>> auth_param basic credentialsttl 2 hours
>>> auth_param basic program /usr/lib/squid/ncsa_auth
>>> /etc/squid/squid_passwd
>>> authenticate_cache_garbage_interval 1 hour
>>> authenticate_ip_ttl 2 hours
>>> #acl all src 0.0.0.0/0.0.0.0
>>> acl src all
>>> acl manager proto cache_object
>>> acl localhost src 127.0.0.1
>>> acl cacheadmin src 88.xxx.xxx.xxx
>>> acl to_localhost dst 127.0.0.0/8 0.0.0.0/32
>>> acl SSL_ports port 443
>>> acl Safe_ports port 80 # http
>>> acl Safe_ports port 21 # ftp
>>> acl Safe_ports port 443 # https
>>> acl Safe_ports port 70 # gopher
>>> acl Safe_ports port 210 # wais
>>> acl Safe_ports port 1025-65535 # unregistered ports
>>> acl Safe_ports port 280 # http-mgmt
>>> acl Safe_ports port 488 # gss-http
>>> acl Safe_ports port 591 # filemaker
>>> acl Safe_ports port 777 # multiling http
>>> acl Safe_ports port 1863 # MSN messenger
>>> acl ncsa_users proxy_auth REQUIRED
>>> acl maxuser max_user_ip -s 2
>>> acl CONNECT method CONNECT
>>> http_access allow manager localhost
>>> http_access allow manager cacheadmin
>>> http_access deny !Safe_ports
>>> http_access deny CONNECT !SSL_ports
>>> http_access deny to_localhost
>>> http_access deny manager
>>> http_access allow ncsa_users
>>> http_access deny maxuser
>>> #http_access allow localhost
>>> http_access deny all
>>> icp_access allow all
>>> http_port 8080
>>> http_port 88.xxx.xxx.xxx:80
>>> hierarchy_stoplist cgi-bin ?
>>> cache_mem 100 MB
>>> maximum_object_size_in_memory 50 KB
>>> cache_replacement_policy heap LFUDA
>>> cache_dir aufs /var/spool/squid 40000 16 256
>>> maximum_object_size 50 MB
>>> cache_swap_low 90
>>> cache_swap_high 95
>>> access_log /var/log/squid/access.log squid
>>> cache_log /var/log/squid/cache.log
>>> buffered_logs on
>>> #acl QUERY urlpath_regex cgi-bin \?
>>> #cache deny QUERY
>>> refresh_pattern ^ftp: 1440 20% 10080
>>> refresh_pattern ^gopher: 1440 0% 1440
>>> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
>>> refresh_pattern . 0 20% 4320
>>> quick_abort_min 0 KB
>>> quick_abort_max 0 KB
>>> acl apache rep_header Server ^Apache
>>> broken_vary_encoding allow apache
>>> half_closed_clients off
>>> cache_mgr aaa_at_aaa.com
>>> cachemgr_passwd aaa all
>>> visible_hostname ProxyServer
>>> log_icp_queries off
>>> dns_nameservers 208.67.222.222 208.67.220.220
>>> hosts_file /etc/hosts
>>> memory_pools off
>>> forwarded_for off
>>> client_db off
>>> coredump_dir /var/spool/squid
>>>
>>> ----------------------------------------
>>>> Date: Sat, 13 Feb 2010 18:03:00 +1300
>>>> From: squid3_at_treenet.co.nz
>>>> To: squid-users_at_squid-cache.org
>>>> Subject: Re: [squid-users] Cache manager analysis
>>>>
>>>> J. Webster wrote:
>>>>> What is the best place to start with in cache analysis?
>>>>> Would it be cache size, memory object size, IO, etc.?
>>>>> I'm looking to optimise the settings for my squid server.
>>>>
>>>> Step 0) migrate to the latest Squid 2.7 or 3.1 or if possible 2.HEAD
>>>> (that one is only nominally beta, it's very stable in reality)
>>>>
>>>> 1) Start by defining 'optimize' ... are you going to prioritize...
>>>> Faster service?
>>>> More bandwidth saving?
>>>> More client connections?
>>>>
>>>> 2a) For faster service, look at DNS delays, disk IO delays,
maximizing
>>>> cacheable objects (dynamic objects etc).
>>>>
>>>> 2b) For pure bandwidth savings start with a look at object
cacheablity.
>>>> Check dynamics are being cached, ranges are being fetched in full,
etc
>>>>
>>>> 3) Then profile all the objects stored over a reasonably long period,
>>>> looking at size. compare with the age of objects being discarded.
>>>>
>>>> 3a) tune the storage limits to prioritize the storage locations.
giving
>>>> priority to RAM, then COSS, then AUFS/diskd.
>>>>
>>>> 3b) set the storage limits as high as possible to maximize amount of
>>>> data stored. anywhere.
>>>>
>>>> 4) take a good long look at your access controls and in particular
the
>>>> types speedy/fast/slow. You may get some speed benefits from fixing
up
>>>> the ordering a bit. regex are killers, remote lookups (helpers, or
DNS)
>>>> are second worst.
>>>> (some performance hints below)
>>>>
>>>> 5) repeat from (2b) as often as possible. concentrate traffic which
>>>> seems to logically be storeable but gets a TCP_MISS anyway.
>>>>
>>>> Objects served from cache lead to faster service ties for those
>>>> objects,
>>>> so the speed vs bandwidth are inter-related somewhat. But there is a
>>>> tipping point somewhere where tuning one starts to impact the other.
>>>>
>>>>
>>>>>
>>>>> Server: about 220GB available for the cache, I'm only using 40000 MB
>>>>> at present as in the config below.
>>>>> system D2812-A2
>>>>> /0 bus D2812-A2
>>>>> /0/0 memory 110KiB BIOS
>>>>> /0/4 processor Intel(R) Core(TM)2 Duo CPU E7300 @ 2.66GHz
>>>>> /0/4/5 memory 64KiB L1 cache
>>>>> /0/4/6 memory 3MiB L2 cache
>>>>> /0/4/0.1 processor Logical CPU
>>>>> /0/4/0.2 processor Logical CPU
>>>>> /0/7 memory 3MiB L3 cache
>>>>> /0/2a memory 1GiB System Memory
>>>>> /0/2a/0 memory 1GiB DIMM DDR2 Synchronous 667 MHz (1.5 ns)
>>>>> /0/2a/1 memory DIMM DDR2 Synchronous 667 MHz (1.5 ns) [empty]
>>>>> /0/2a/2 memory DIMM DDR2 Synchronous 667 MHz (1.5 ns) [empty]
>>>>> /0/2a/3 memory DIMM DDR2 Synchronous 667 MHz (1.5 ns) [empty]
>>>>> /0/1 processor
>>>>> /0/1/0.1 processor Logical CPU
>>>>> /0/1/0.2 processor Logical CPU
>>>>>
>>>>>
>>>>> Current squid.conf:
>>>>> ---------------------
>>>>> auth_param basic realm Proxy server
>>>>> auth_param basic credentialsttl 2 hours
>>>>> auth_param basic program /usr/lib/squid/ncsa_auth
>>>>> /etc/squid/squid_passwd
>>>>> authenticate_cache_garbage_interval 1 hour
>>>>> authenticate_ip_ttl 2 hours
>>>>> acl all src 0.0.0.0/0.0.0.0
>>>>
>>>> all src all
>>>>
>>>>> acl manager proto cache_object
>>>>> acl localhost src 127.0.0.1/255.255.255.255
>>>>
>>>> acl localhost src 127.0.0.1
>>>>
>>>>> acl cacheadmin src 88.xxx.xxx.xxx
>>>>> acl to_localhost dst 127.0.0.0/8
>>>>
>>>> acl to_localhost dst 127.0.0.0/8 0.0.0.0/32
>>>>
>>>>> acl SSL_ports port 443
>>>>> acl Safe_ports port 80 # http
>>>>> acl Safe_ports port 21 # ftp
>>>>> acl Safe_ports port 443 # https
>>>>> acl Safe_ports port 70 # gopher
>>>>> acl Safe_ports port 210 # wais
>>>>> acl Safe_ports port 1025-65535 # unregistered ports
>>>>> acl Safe_ports port 280 # http-mgmt
>>>>> acl Safe_ports port 488 # gss-http
>>>>> acl Safe_ports port 591 # filemaker
>>>>> acl Safe_ports port 777 # multiling http
>>>>> acl Safe_ports port 1863 # MSN messenger
>>>>> acl ncsa_users proxy_auth REQUIRED
>>>>> acl maxuser max_user_ip -s 2
>>>>> acl CONNECT method CONNECT
>>>>> http_access allow manager localhost
>>>>> http_access allow manager cacheadmin
>>>>
>>>> Hint: add the localhost IP to the cacheadmin ACL and drop one full
set
>>>> of "allow manager localhost" tests.
>>>>
>>>>> http_access deny manager
>>>>> http_access allow ncsa_users
>>>>
>>>> Hint: drop the authentication down ...
>>>>
>>>>> http_access deny !Safe_ports
>>>>> http_access deny CONNECT !SSL_ports
>>>>> http_access deny to_localhost
>>>>
>>>> ... to here. All the attacks against your proxy for bad ports and
>>>> sources will be dropped quickly by the security blanket settings.
Load
>>>> on your auth server will reduce and may speed up it's response time.
>>>>
>>>> Hint 2: if possible, define an ACL or the network ranges where you
>>>> accept logins. Use it like so:
>>>>
>>>> http_access allow localnet ncsa_users
>>>>
>>>> ... once again that speeds up the rejections, and helps by reducing
>>>> the number of times the slow auth lookup needs checking.
>>>>
>>>>> http_access deny maxuser
>>>>> http_access allow localhost
>>>>
>>>> If localhost really is allowed to do anything, move it up above the
>>>> "to_localhost" one.
>>>> Otherwise drop this completely, having the correct auth login details
>>>> will permit links from localhost just as easily as from anywhere
else.
>>>>
>>>>> http_access deny all
>>>>> icp_access allow all
>>>>
>>>> Define the networks where peer siblings are trusted. Allwo them and
>>>> deny
>>>> everything else.
>>>> That will reduce a fair bit of load on your Squid trying to service
>>>> random ICP requests from the general Internet.
>>>>
>>>>> http_port 8080
>>>>> http_port 88.xxx.xxx.xxx:80
>>>>> hierarchy_stoplist cgi-bin ?
>>>>> cache_mem 100 MB
>>>>
>>>> Bump this up as high as you can go without risking memory swapping.
>>>> Objects served from RAM are 100x faster than objects not.
>>>>
>>>>> maximum_object_size_in_memory 50 KB
>>>>> cache_replacement_policy heap LFUDA
>>>>> cache_dir aufs /var/spool/squid 40000 16 256
>>>>
>>>> If you pick 2.x squid to upgrade to, add a COSS directory as well.
>>>> See the recent threads on optimizing COSS for how to tune that.
>>>>
>>>>> maximum_object_size 50 MB
>>>>
>>>> Bump this up too. Holding full ISO CDs and windows service packs can
>>>> boost performance when one is used from the cache. 40GB of disk can
>>>> store a few.
>>>>
>>>>> cache_swap_low 90
>>>>> cache_swap_high 95
>>>>> access_log /var/log/squid/access.log squid
>>>>> cache_log /var/log/squid/cache.log
>>>>> buffered_logs on
>>>>> acl QUERY urlpath_regex cgi-bin \?
>>>>> cache deny QUERY
>>>>
>>>> Drop the QUERY bits above. It's more than halving the things your
Squid
>>>> can store.
>>>>
>>>>> refresh_pattern ^ftp: 1440 20% 10080
>>>>> refresh_pattern ^gopher: 1440 0% 1440
>>>>
>>>> Add right here:
>>>> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
>>>>
>>>>> refresh_pattern . 0 20% 4320
>>>>> quick_abort_min 0 KB
>>>>> quick_abort_max 0 KB
>>>>> acl apache rep_header Server ^Apache
>>>>> broken_vary_encoding allow apache
>>>>> half_closed_clients off
>>>>> cache_mgr aaa_at_aaa.com
>>>>> cachemgr_passwd aaa all
>>>>> visible_hostname ProxyServer
>>>>> log_icp_queries off
>>>>> dns_nameservers 208.67.222.222 208.67.220.220
>>>>> hosts_file /etc/hosts
>>>>> memory_pools off
>>>>
>>>> Might cause efficiency problems if the underlying malloc is not
>>>> optimized. but oh well, up to you.
>>>>
>>>>> forwarded_for off
>>>>> client_db off
>>>>> coredump_dir /var/spool/squid
>>>>>
>>>>
>>>>
>>>> Amos
>>>> --
>>>> Please be using
>>>> Current Stable Squid 2.7.STABLE7 or 3.0.STABLE23
>>>> Current Beta Squid 3.1.0.16
>>>
>>> _________________________________________________________________
>>> Send us your Hotmail stories and be featured in our newsletter
>>> http://clk.atdmt.com/UKM/go/195013117/direct/01/
>>
>> _________________________________________________________________
>> Send us your Hotmail stories and be featured in our newsletter
>> http://clk.atdmt.com/UKM/go/195013117/direct/01/
>
> _________________________________________________________________
> We want to hear all your funny, exciting and crazy Hotmail stories. Tell
> us now
> http://clk.atdmt.com/UKM/go/195013117/direct/01/
Received on Sun Feb 21 2010 - 23:43:58 MST