[squid-users] squidaio_queue_request: WARNING - Queue congestion

Squid 2.7Stable7
F12
AUFS on a ext3 FS
6gigs ram
dual proc
cache_dir aufs /cache 32000 16 256

Filesystem Size Used Avail Use% Mounted on
/dev/vda2 49G 3.8G 42G 9% /cache

configure options: '--host=i686-pc-linux-gnu'
'--build=i686-pc-linux-gnu' '--target=i386-redhat-linux'
'--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr'
'--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc'
'--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib'
'--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib'
'--mandir=/usr/share/man' '--infodir=/usr/share/info'
'--exec_prefix=/usr' '--libexecdir=/usr/lib/squid'
'--localstatedir=/var' '--datadir=/usr/share/squid'
'--sysconfdir=/etc/squid' '--with-logdir=$(localstatedir)/log/squid'
'--with-pidfile=$(localstatedir)/run/squid.pid'
'--disable-dependency-tracking' '--enable-arp-acl'
'--enable-follow-x-forwarded-for'
'--enable-auth=basic,digest,ntlm,negotiate'
'--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,getpwnam,multi-domain-NTLM,SASL,squid_radius_auth'
'--enable-ntlm-auth-helpers=no_check,fakeauth'
'--enable-digest-auth-helpers=password,ldap'
'--enable-negotiate-auth-helpers=squid_kerb_auth'
'--enable-external-acl-helpers=ip_user,ldap_group,session,unix_group,wbinfo_group'
'--enable-cache-digests' '--enable-cachemgr-hostname=localhost'
'--enable-delay-pools' '--enable-epoll' '--enable-icap-client'
'--enable-ident-lookups' '--with-large-files'
'--enable-linux-netfilter' '--enable-referer-log'
'--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl'
'--enable-storeio=aufs,diskd,ufs' '--enable-useragent-log'
'--enable-wccpv2' '--enable-esi' '--with-aio'
'--with-default-user=squid' '--with-filedescriptors=16384' '--with-dl'
'--with-openssl' '--with-pthreads' 'build_alias=i686-pc-linux-gnu'
'host_alias=i686-pc-linux-gnu' 'target_alias=i386-redhat-linux'
'CFLAGS=-fPIE -Os -g -pipe -fsigned-char -O2 -g -march=i386
-mtune=i686' 'LDFLAGS=-pie'

No load to speak of, very little iowait. Threads were configured as the default.

This is running on a striped pair of SSD's and is only a test script
that (ya it's hitting it a bit hard), but nothing that squid nor my
hardware should have an issue with.

I've searched and really there does not appear to be a solid answer,
except running out of cpu or running out of iops, neither "appears" to
be the case here. Figured if it was a thread issue, I would see a
bottleneck on my server? (ya?). Also the if it only happens a couple
of times ignore it. This is just some testing and I believe this
congestion is possibly causing the 500 errors I'm seeing while running
my script.

Any pointers, where to look etc? (2.7stable6 on fc6/xen kernel) had no
such issues (yes, the SSD's are a new variable (but otherwise
identical hardware).

Thanks
Tory

avg-cpu: %user %nice %system %iowait %steal %idle
          13.13 0.00 40.66 4.80 0.00 41.41

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
vda 1539.50 11516.00 6604.00 23032 13208

avg-cpu: %user %nice %system %iowait %steal %idle
          13.60 0.00 38.29 11.08 0.00 37.03

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
vda 1385.07 11080.60 0.00 22272 0

[@cache01 ~]$ free
             total used free shared buffers cached
Mem: 5039312 421768 4617544 0 50372 183900
-/+ buffers/cache: 187496 4851816
Swap: 7143416 0 7143416

Totals since cache startup:
sample_time = 1266055655.813016 (Sat, 13 Feb 2010 10:07:35 GMT)
client_http.requests = 810672
client_http.hits = 683067
client_http.errors = 0
client_http.kbytes_in = 171682
client_http.kbytes_out = 2145472
client_http.hit_kbytes_out = 1809060
server.all.requests = 127606
server.all.errors = 0
server.all.kbytes_in = 321960
server.all.kbytes_out = 38104
server.http.requests = 127606
server.http.errors = 0
server.http.kbytes_in = 321960
server.http.kbytes_out = 38104
server.ftp.requests = 0
server.ftp.errors = 0
server.ftp.kbytes_in = 0
server.ftp.kbytes_out = 0
server.other.requests = 0
server.other.errors = 0
server.other.kbytes_in = 0
server.other.kbytes_out = 0
icp.pkts_sent = 0
icp.pkts_recv = 0
icp.queries_sent = 0
icp.replies_sent = 0
icp.queries_recv = 0
icp.replies_recv = 0
icp.query_timeouts = 0
icp.replies_queued = 0
icp.kbytes_sent = 0
icp.kbytes_recv = 0
icp.q_kbytes_sent = 0
icp.r_kbytes_sent = 0
icp.q_kbytes_recv = 0
icp.r_kbytes_recv = 0
icp.times_used = 0
cd.times_used = 0
cd.msgs_sent = 0
cd.msgs_recv = 0
cd.memory = 0
cd.local_memory = 487
cd.kbytes_sent = 0
cd.kbytes_recv = 0
unlink.requests = 0
page_faults = 1
select_loops = 467112
cpu_time = 681.173445
wall_time = -40015.496720
swap.outs = 126078
swap.ins = 1366134
swap.files_cleaned = 0
aborted_requests = 0
Received on Fri Feb 12 2010 - 23:04:39 MST