[squid-users] Performance Extremely squid configuration advice

In order to get squid server 400M+ traffic, I did these:
1. Memory only
IO bottleneck is too hard to avoid at high traffic, so I did not use
harddisk, use only memory for HTTP cache. 32GB or 64GB memory per box
works good.

2. Disable useless acl
I did not use any acl, even default acls:
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 901 # SWAT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

squid itself do not do any acls, security is ensured by other layers,
like iptables or acls on routers.

3. refresh_pattern, mainly cache for pictures
Make squid cache as long as it can, so it looks likes this:
refresh_pattern -i \.(jpg|jpeg|gif|png|swf|htm|html|bmp)(\?.*)?$
21600 100% 21600 reload-into-ims ignore-reload ignore-no-cache
ignore-auth ignore-private

4. multi-instance
I can't get single squid process runs over 200M, so multi-instance
make perfect sense.
Both CARP frontend and backend (for store HTTP files) need to be
multi-instanced. Frontend configuration is here:
http://wiki.squid-cache.org/ConfigExamples/ExtremeCarpFrontend

I heard that squid is still can't process "huge" memory properly, so I
splited big memory into 6-8GB per instance, which listens at ports
lower than 80. And on a box with 32GB memory CARP frontend configs
like this:

cache_peer 192.168.1.73 parent 76 0 carp name=73-76 proxy-only
cache_peer 192.168.1.73 parent 77 0 carp name=73-77 proxy-only
cache_peer 192.168.1.73 parent 78 0 carp name=73-78 proxy-only
cache_peer 192.168.1.73 parent 79 0 carp name=73-79 proxy-only

5. CARP frontend - cache_mem 0 MB
I used to use "cache_mem 0 MB", time flies, I think that files smaller
than 1.5KB would be waste if GET from CARP backend, am I right? I use
these now:

cache_mem 5 MB
maximum_object_size_in_memory 1.5 KB

6. LAN, WAN seperates
Again, to split load on NIC. Use LAN for clients and CARP interaction,
WAN to fetch content from internet.

7. Using official NIC driver.
Sometimes chip vender's official driver acts better behavior than
builtin driver, so it's worth to try.

8. Based on gentoo
Using gentoo, we can strip useless function as much as possible, make
the cache system thinner, and faster.

9. Strip useless compile options and runtime options
Proper CFLAGS and LDFLAGS are needed, here's one good doc:
http://en.gentoo-wiki.com/wiki/Safe_Cflags

~ # squid -v
Squid Cache: Version 2.7.STABLE9
configure options: '--prefix=/usr' '--build=x86_64-pc-linux-gnu'
'--host=x86_64-pc-linux-gnu' '--mandir=/usr/share/man'
'--infodir=/usr/share/info' '--datadir=/usr/share' '--sysconfdir=/etc'
'--localstatedir=/var/lib' '--libdir=/usr/lib64'
'--sysconfdir=/etc/squid' '--libexecdir=/usr/libexec/squid'
'--localstatedir=/var' '--datadir=/usr/share/squid' '--disable-auth'
'--disable-delay-pools' '--enable-removal-policies=lru,heap'
'--enable-ident-lookups' '--enable-useragent-log'
'--enable-cache-digests' '--enable-referer-log'
'--enable-http-violations' '--with-pthreads' '--with-large-files'
'--enable-wccpv2' '--enable-htcp' '--enable-carp' '--enable-icmp'
'--enable-follow-x-forwarded-for' '--enable-x-accelerator-vary'
'--enable-kill-parent-hack' '--enable-cachemgr-hostname=squid37'
'--enable-err-languages=English'
'--enable-default-err-language=English' '--with-maxfd=65535'
'--without-libcap' '--disable-snmp' '--disable-ssl'
'--enable-storeio=ufs,diskd,coss,aufs,null' '--enable-async-io'
'--enable-linux-netfilter' '--disable-linux-tproxy' '--enable-epoll'
'build_alias=x86_64-pc-linux-gnu' 'host_alias=x86_64-pc-linux-gnu'
'CC=x86_64-pc-linux-gnu-gcc' 'CFLAGS=-march=barcelona -mtune=barcelona
-O2 -pipe' 'LDFLAGS=-Wl,-O1 -Wl,--as-needed'

10. sysctl tune
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_syn_retries = 3
net.ipv4.tcp_synack_retries = 3
net.ipv4.tcp_max_syn_backlog = 4096
net.core.netdev_max_backlog = 4096
net.ipv4.ip_local_port_range = 1024 65534
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 1000
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_sack = 0
net.ipv4.tcp_low_latency = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_max_orphans = 16384
net.ipv4.tcp_orphan_retries = 1
net.ipv4.ipfrag_high_thresh = 524288
net.ipv4.ipfrag_low_thresh = 262144
kernel.pid_max = 65535
vm.swappiness = 1
net.ipv4.tcp_mem = 6085248 8113664 12170496
net.ipv4.tcp_wmem = 4096 65536 8388608
net.ipv4.tcp_rmem = 4096 87380 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 8388608
net.core.wmem_default = 8388608
net.core.wmem_max = 8388608
net.core.somaxconn = 512
net.ipv4.udp_mem = 6194688 8259584 12389376
net.ipv4.udp_rmem_min = 8192
net.ipv4.udp_wmem_min = 8192
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1

This is all I did to get high performance, what should I do to get
even better performance, any more advice?
Received on Fri Jan 07 2011 - 03:40:56 MST