Re: [squid-users] Cache manager analysis

J. Webster wrote:
> What is the best place to start with in cache analysis?
> Would it be cache size, memory object size, IO, etc.?
> I'm looking to optimise the settings for my squid server.

Step 0) migrate to the latest Squid 2.7 or 3.1 or if possible 2.HEAD
(that one is only nominally beta, it's very stable in reality)

1) Start by defining 'optimize' ... are you going to prioritize...
  Faster service?
  More bandwidth saving?
  More client connections?

2a) For faster service, look at DNS delays, disk IO delays, maximizing
cacheable objects (dynamic objects etc).

2b) For pure bandwidth savings start with a look at object cacheablity.
Check dynamics are being cached, ranges are being fetched in full, etc

3) Then profile all the objects stored over a reasonably long period,
looking at size. compare with the age of objects being discarded.

3a) tune the storage limits to prioritize the storage locations. giving
priority to RAM, then COSS, then AUFS/diskd.

3b) set the storage limits as high as possible to maximize amount of
data stored. anywhere.

4) take a good long look at your access controls and in particular the
types speedy/fast/slow. You may get some speed benefits from fixing up
the ordering a bit. regex are killers, remote lookups (helpers, or DNS)
are second worst.
   (some performance hints below)

5) repeat from (2b) as often as possible. concentrate traffic which
seems to logically be storeable but gets a TCP_MISS anyway.

Objects served from cache lead to faster service ties for those objects,
so the speed vs bandwidth are inter-related somewhat. But there is a
tipping point somewhere where tuning one starts to impact the other.

>
> Server: about 220GB available for the cache, I'm only using 40000 MB at present as in the config below.
> system D2812-A2
> /0 bus D2812-A2
> /0/0 memory 110KiB BIOS
> /0/4 processor Intel(R) Core(TM)2 Duo CPU E7300 @ 2.66GHz
> /0/4/5 memory 64KiB L1 cache
> /0/4/6 memory 3MiB L2 cache
> /0/4/0.1 processor Logical CPU
> /0/4/0.2 processor Logical CPU
> /0/7 memory 3MiB L3 cache
> /0/2a memory 1GiB System Memory
> /0/2a/0 memory 1GiB DIMM DDR2 Synchronous 667 MHz (1.5 ns)
> /0/2a/1 memory DIMM DDR2 Synchronous 667 MHz (1.5 ns) [empty]
> /0/2a/2 memory DIMM DDR2 Synchronous 667 MHz (1.5 ns) [empty]
> /0/2a/3 memory DIMM DDR2 Synchronous 667 MHz (1.5 ns) [empty]
> /0/1 processor
> /0/1/0.1 processor Logical CPU
> /0/1/0.2 processor Logical CPU
>
>
> Current squid.conf:
> ---------------------
> auth_param basic realm Proxy server
> auth_param basic credentialsttl 2 hours
> auth_param basic program /usr/lib/squid/ncsa_auth /etc/squid/squid_passwd
> authenticate_cache_garbage_interval 1 hour
> authenticate_ip_ttl 2 hours
> acl all src 0.0.0.0/0.0.0.0

all src all

> acl manager proto cache_object
> acl localhost src 127.0.0.1/255.255.255.255

acl localhost src 127.0.0.1

> acl cacheadmin src 88.xxx.xxx.xxx
> acl to_localhost dst 127.0.0.0/8

acl to_localhost dst 127.0.0.0/8 0.0.0.0/32

> acl SSL_ports port 443
> acl Safe_ports port 80 # http
> acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 # https
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http
> acl Safe_ports port 591 # filemaker
> acl Safe_ports port 777 # multiling http
> acl Safe_ports port 1863 # MSN messenger
> acl ncsa_users proxy_auth REQUIRED
> acl maxuser max_user_ip -s 2
> acl CONNECT method CONNECT
> http_access allow manager localhost
> http_access allow manager cacheadmin

Hint: add the localhost IP to the cacheadmin ACL and drop one full set
of "allow manager localhost" tests.

> http_access deny manager
> http_access allow ncsa_users

Hint: drop the authentication down ...

> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access deny to_localhost

... to here. All the attacks against your proxy for bad ports and
sources will be dropped quickly by the security blanket settings. Load
on your auth server will reduce and may speed up it's response time.

Hint 2: if possible, define an ACL or the network ranges where you
accept logins. Use it like so:

   http_access allow localnet ncsa_users

  ... once again that speeds up the rejections, and helps by reducing
the number of times the slow auth lookup needs checking.

> http_access deny maxuser
> http_access allow localhost

If localhost really is allowed to do anything, move it up above the
"to_localhost" one.
  Otherwise drop this completely, having the correct auth login details
will permit links from localhost just as easily as from anywhere else.

> http_access deny all
> icp_access allow all

Define the networks where peer siblings are trusted. Allwo them and deny
everything else.
That will reduce a fair bit of load on your Squid trying to service
random ICP requests from the general Internet.

> http_port 8080
> http_port 88.xxx.xxx.xxx:80
> hierarchy_stoplist cgi-bin ?
> cache_mem 100 MB

Bump this up as high as you can go without risking memory swapping.
Objects served from RAM are 100x faster than objects not.

> maximum_object_size_in_memory 50 KB
> cache_replacement_policy heap LFUDA
> cache_dir aufs /var/spool/squid 40000 16 256

If you pick 2.x squid to upgrade to, add a COSS directory as well.
See the recent threads on optimizing COSS for how to tune that.

> maximum_object_size 50 MB

Bump this up too. Holding full ISO CDs and windows service packs can
boost performance when one is used from the cache. 40GB of disk can
store a few.

> cache_swap_low 90
> cache_swap_high 95
> access_log /var/log/squid/access.log squid
> cache_log /var/log/squid/cache.log
> buffered_logs on
> acl QUERY urlpath_regex cgi-bin \?
> cache deny QUERY

Drop the QUERY bits above. It's more than halving the things your Squid
can store.

> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440

Add right here:
  refresh_pattern -i (/cgi-bin/|\?) 0 0% 0

> refresh_pattern . 0 20% 4320
> quick_abort_min 0 KB
> quick_abort_max 0 KB
> acl apache rep_header Server ^Apache
> broken_vary_encoding allow apache
> half_closed_clients off
> cache_mgr aaa_at_aaa.com
> cachemgr_passwd aaa all
> visible_hostname ProxyServer
> log_icp_queries off
> dns_nameservers 208.67.222.222 208.67.220.220
> hosts_file /etc/hosts
> memory_pools off

Might cause efficiency problems if the underlying malloc is not
optimized. but oh well, up to you.

> forwarded_for off
> client_db off
> coredump_dir /var/spool/squid
>

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE23
   Current Beta Squid 3.1.0.16