Re: [squid-users] Performance Extremely squid configuration advice

On 07/01/11 19:08, Drunkard Zhang wrote:
> In order to get squid server 400M+ traffic, I did these:
> 1. Memory only
> IO bottleneck is too hard to avoid at high traffic, so I did not use
> harddisk, use only memory for HTTP cache. 32GB or 64GB memory per box
> works good.

NP: The problem in squid-2 is large objects in memory. Though the more
objects you have cached the slower the index lookups (very, very minor
impact).

>
> 2. Disable useless acl
> I did not use any acl, even default acls:
> acl SSL_ports port 443
> acl Safe_ports port 80 # http
> acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 # https
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http
> acl Safe_ports port 591 # filemaker
> acl Safe_ports port 777 # multiling http
> acl Safe_ports port 901 # SWAT
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
>
> squid itself do not do any acls, security is ensured by other layers,
> like iptables or acls on routers.

Having the routers etc assemble the packets and parse the HTTP-layer
protocol to find these details may be a larger bottleneck than testing
for them inside Squid where the parsing has to be done a second time
anyway to pass the request on.

Note that the default port and method ACL in Squid are validating on the
HTTP header content URLs not the packet destination port.

>
> 3. refresh_pattern, mainly cache for pictures
> Make squid cache as long as it can, so it looks likes this:
> refresh_pattern -i \.(jpg|jpeg|gif|png|swf|htm|html|bmp)(\?.*)?$
> 21600 100% 21600 reload-into-ims ignore-reload ignore-no-cache
> ignore-auth ignore-private
>
> 4. multi-instance
> I can't get single squid process runs over 200M, so multi-instance
> make perfect sense.

Congratulations, most can't get Squid to go over 50MBps per instance.

> Both CARP frontend and backend (for store HTTP files) need to be
> multi-instanced. Frontend configuration is here:
> http://wiki.squid-cache.org/ConfigExamples/ExtremeCarpFrontend
>
> I heard that squid is still can't process "huge" memory properly, so I
> splited big memory into 6-8GB per instance, which listens at ports
> lower than 80. And on a box with 32GB memory CARP frontend configs
> like this:
>
> cache_peer 192.168.1.73 parent 76 0 carp name=73-76 proxy-only
> cache_peer 192.168.1.73 parent 77 0 carp name=73-77 proxy-only
> cache_peer 192.168.1.73 parent 78 0 carp name=73-78 proxy-only
> cache_peer 192.168.1.73 parent 79 0 carp name=73-79 proxy-only
>
> 5. CARP frontend - cache_mem 0 MB
> I used to use "cache_mem 0 MB", time flies, I think that files smaller
> than 1.5KB would be waste if GET from CARP backend, am I right? I use
> these now:
>
> cache_mem 5 MB
> maximum_object_size_in_memory 1.5 KB

The best value here differs on every network so we can't answer your
question with details.

Log analysis of live traffic will show you the amount of objects your
Squid are handling in each size bracket. That will determine where the
best place to set this limit at to reduce the lag on small items versus
your available cache_mem memory.

>
> 6. LAN, WAN seperates
> Again, to split load on NIC. Use LAN for clients and CARP interaction,
> WAN to fetch content from internet.
>
> 7. Using official NIC driver.
> Sometimes chip vender's official driver acts better behavior than
> builtin driver, so it's worth to try.
>
> 8. Based on gentoo
> Using gentoo, we can strip useless function as much as possible, make
> the cache system thinner, and faster.
>
> 9. Strip useless compile options and runtime options
> Proper CFLAGS and LDFLAGS are needed, here's one good doc:
> http://en.gentoo-wiki.com/wiki/Safe_Cflags
>
> ~ # squid -v
> Squid Cache: Version 2.7.STABLE9
> configure options: '--prefix=/usr' '--build=x86_64-pc-linux-gnu'
> '--host=x86_64-pc-linux-gnu' '--mandir=/usr/share/man'
> '--infodir=/usr/share/info' '--datadir=/usr/share' '--sysconfdir=/etc'
> '--localstatedir=/var/lib' '--libdir=/usr/lib64'
> '--sysconfdir=/etc/squid' '--libexecdir=/usr/libexec/squid'
> '--localstatedir=/var' '--datadir=/usr/share/squid' '--disable-auth'
> '--disable-delay-pools' '--enable-removal-policies=lru,heap'
> '--enable-ident-lookups' '--enable-useragent-log'
> '--enable-cache-digests' '--enable-referer-log'
> '--enable-http-violations' '--with-pthreads' '--with-large-files'
> '--enable-wccpv2' '--enable-htcp' '--enable-carp' '--enable-icmp'
> '--enable-follow-x-forwarded-for' '--enable-x-accelerator-vary'
> '--enable-kill-parent-hack' '--enable-cachemgr-hostname=squid37'
> '--enable-err-languages=English'
> '--enable-default-err-language=English' '--with-maxfd=65535'
> '--without-libcap' '--disable-snmp' '--disable-ssl'
> '--enable-storeio=ufs,diskd,coss,aufs,null' '--enable-async-io'
> '--enable-linux-netfilter' '--disable-linux-tproxy' '--enable-epoll'
> 'build_alias=x86_64-pc-linux-gnu' 'host_alias=x86_64-pc-linux-gnu'
> 'CC=x86_64-pc-linux-gnu-gcc' 'CFLAGS=-march=barcelona -mtune=barcelona
> -O2 -pipe' 'LDFLAGS=-Wl,-O1 -Wl,--as-needed'
>
> 10. sysctl tune
> net.ipv4.ip_forward = 0
> net.ipv4.conf.default.rp_filter = 1
> net.ipv4.conf.default.accept_source_route = 0
> kernel.sysrq = 0
> kernel.core_uses_pid = 1
> net.ipv4.tcp_syncookies = 1
> net.ipv4.tcp_syn_retries = 3
> net.ipv4.tcp_synack_retries = 3
> net.ipv4.tcp_max_syn_backlog = 4096
> net.core.netdev_max_backlog = 4096
> net.ipv4.ip_local_port_range = 1024 65534
> net.netfilter.nf_conntrack_max = 1048576
> net.netfilter.nf_conntrack_tcp_timeout_established = 1000
> net.ipv4.tcp_timestamps = 0
> net.ipv4.tcp_sack = 0
> net.ipv4.tcp_low_latency = 1
> net.ipv4.tcp_fin_timeout = 15
> net.ipv4.tcp_keepalive_intvl = 30
> net.ipv4.tcp_keepalive_probes = 3
> net.ipv4.tcp_keepalive_time = 1800
> net.ipv4.tcp_max_orphans = 16384
> net.ipv4.tcp_orphan_retries = 1
> net.ipv4.ipfrag_high_thresh = 524288
> net.ipv4.ipfrag_low_thresh = 262144
> kernel.pid_max = 65535
> vm.swappiness = 1
> net.ipv4.tcp_mem = 6085248 8113664 12170496
> net.ipv4.tcp_wmem = 4096 65536 8388608
> net.ipv4.tcp_rmem = 4096 87380 8388608
> net.core.rmem_default = 8388608
> net.core.rmem_max = 8388608
> net.core.wmem_default = 8388608
> net.core.wmem_max = 8388608
> net.core.somaxconn = 512
> net.ipv4.udp_mem = 6194688 8259584 12389376
> net.ipv4.udp_rmem_min = 8192
> net.ipv4.udp_wmem_min = 8192
> net.ipv4.tcp_tw_reuse = 1
> net.ipv4.tcp_tw_recycle = 1
>
>
> This is all I did to get high performance, what should I do to get
> even better performance, any more advice?

You mentioned a backend storing files. If that is using disk storage the
use of COSS there (or the RockStore squid-3 branch) will speed up the
disk IO by reducing the number of small ops.

I can't comment on the sysctl settings other than to say swapping is BAD
for Squid performance.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.10
   Beta testers wanted for 3.2.0.4