[squid-users] Performanceproblem Squid with one URL - strange behaviour ...

Hi folks,

we have a very special problem with our proxy environment. It concerns
only ONE uri http://www.mediaassetbox.com/client/escada. Other uris are
working properly.

Unfortunately this is a very bad uri because it works only with flash.
Nevertheless our customer is working with it and we have a performance
issue.

If page starts to load it need approx. 60-70 seconds until the blue
progress bar under the login field disappears.

If I use another proxy product - eg. IWSS - the page loads in about 30
seconds. Also with direct internet connection we have this value ...

So far so good - strange behaviour starts after working on the problem.
Starting strace on the squid process - the performance increases to direct
internet connection speed.

Next we started debugging in squid itself - ALL,3 - without strace - the
performance increases again. Starting with debug section 0 we found out
that 'debug_options 5,3' (or 5,5 ...) increases the performance as fast
as a direct connection.

What we already did without success
- disable ipv6 in os
- strip configuration to minimum
- using a cache_peer parent configuration (the IWSS proxy)
- tried to find out, which systemcalls 'increases' the squid (see
  statistics below)

Now some details about the system:
- OS - Debian Squeeze - Linux xxx 2.6.32-5-amd64 #1 SMP Thu Nov 3 03:41:26 UTC 2011 x86_64 GNU/Linux
- Squid - 3.1.6-1.2+squeeze2
  Squid Cache: Version 3.1.6
  configure options: '--build=x86_64-linux-gnu' '--prefix=/usr'
  '--includedir=${prefix}/include' '--mandir=${prefix}/share/man'
  '--infodir=${prefix}/share/info' '--sysconfdir=/etc'
  '--localstatedir=/var' '--libexecdir=${prefix}/lib/squid3'
  '--disable-maintainer-mode' '--disable-dependency-tracking'
  '--disable-silent-rules' '--srcdir=.' '--datadir=/usr/share/squid3'
  '--sysconfdir=/etc/squid3' '--mandir=/usr/share/man'
  '--with-cppunit-basedir=/usr' '--enable-inline' '--enable-async-io=8'
  '--enable-storeio=ufs,aufs,diskd' '--enable-removal-policies=lru,heap'
  '--enable-delay-pools' '--enable-cache-digests' '--enable-underscores'
  '--enable-icap-client' '--enable-follow-x-forwarded-for'
  '--enable-auth=basic,digest,ntlm,negotiate'
  '--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SASL,SMB,YP,DB,POP3,getpwnam,squid_radius_auth,multi-domain-NTLM'
  '--enable-ntlm-auth-helpers=smb_lm,'
  '--enable-digest-auth-helpers=ldap,password'
  '--enable-negotiate-auth-helpers=squid_kerb_auth'
  '--enable-external-acl-helpers=ip_user,ldap_group,session,unix_group,wbinfo_group'
  '--enable-arp-acl' '--enable-esi' '--disable-translation'
  '--with-logdir=/var/log/squid3' '--with-pidfile=/var/run/squid3.pid'
  '--with-filedescriptors=65536' '--with-large-files'
  '--with-default-user=proxy' '--enable-linux-netfilter'
  'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -g -Wall -O2' 'LDFLAGS='
  'CPPFLAGS=' 'CXXFLAGS=-g -O2 -g -Wall -O2'
  --with-squid=/tmp/buildd/squid3-3.1.6

We can also provide HTTPFox (Firefox extension) lines for fast and slow
connections.

We searched the mailing list and found http://www.mail-archive.com/squid-users@squid-cache.org/msg33267.html -
but there was no really helpful information. Other entries doesn't
match.

We collected the strace statistics only for this session:

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
 97.66 0.004015 1 3209 epoll_wait
  1.24 0.000051 0 145 145 connect
  0.71 0.000029 0 870 recvmsg
  0.22 0.000009 0 912 epoll_ctl
  0.17 0.000007 0 299 getsockname
  0.00 0.000000 0 484 2 read
  0.00 0.000000 0 494 write
  0.00 0.000000 0 444 close
  0.00 0.000000 0 435 socket
  0.00 0.000000 0 16 7 accept
  0.00 0.000000 0 290 sendto
  0.00 0.000000 0 290 bind
  0.00 0.000000 0 290 setsockopt
  0.00 0.000000 0 145 getsockopt
  0.00 0.000000 0 616 fcntl
  0.00 0.000000 0 1 getrusage
------ ----------- ----------- --------- --------- ----------------
100.00 0.004111 8940 154 total

Our squid config:

***********
pid_filename /var/run/squid3-special.pid
http_port 8081

hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
no_cache deny QUERY

refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 4320

#acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/32

acl to_localhost dst 127.0.0.0/8

acl SSL_ports port 443
acl SSL_ports port 8443
acl SSL_ports port 4643
acl Safe_ports port 80
acl Safe_ports port 443
acl Safe_ports port 21
acl Safe_ports port 11371 # PGP Keyserver
acl Safe_ports port 8080
acl Safe_ports port 8443
acl purge method PURGE
acl CONNECT method CONNECT

http_access allow manager localhost
http_access deny manager

http_access allow purge localhost
http_access deny purge

http_access deny CONNECT !SSL_ports
http_access deny !CONNECT !Safe_ports

http_access allow all
icp_access deny all

#debug_options 5,5
***********

network layout is:

client -> firewall -> proxy -> firewall -> internet

Does anyone has an idea what could be the cause for this strange
behaviour?

--
Andreas Schulz