Re: [squid-users] CLOSE_WAIT

On 09/01/13 21:07, Amos Jeffries wrote:

> Does the CONNECT request contain Connection:close or
> Connection:keep-alive? Squid supports keep-alive on CONNECT requests in
> these situations where the CONNECT size is known and may be waiting for
> another client request.

The client sends "Proxy-Connection: keep-alive", so it would indeed be
possible for the connection to be reused. I should add that I'm seeing
this with a non-transparent configuration (i.e. without Tproxy and
without ssl_bump).

> Please upgrade to 3.2.6 (should anyway for the CVE resolved there) and
> see if this issue is gone there.

I've now upgraded my test server and the problem remains.

The "403 Forbidden" being sent back to the client didn't have a Content
Length header, which of course caused the client to drop the connection.
  If I add a Content Length, the connection stays alive and I think it's
a bit more obvious whats happening:

1. Client connects to Squid and sends a CONNECT.
2. Squid passes the request to the ICAP REQMOD service.
3. ICAP rewrites the request.
4. Squid returns a "403 Forbidden" to the client in the clear. At this
point, the connection is still alive with empty queues. Squid's
cache.log shows:
2013/01/10 17:52:18 kid1| abandoning local=[2a00:1a90:5::9]:3128
remote=[2001:4d48:ad51:501:226:bbff:fe18:f3ff]:49926 FD 19 flags=1

Now, the user retries the connection:
5. Client sends CONNECT over the existing connection.
6. Squid does not read the socket (netstat shows the rx queue on the
socket is 220 octets long, which is the whole request the client just sent).
7. The client sits there spinning.
8. Eventually the client times out and drops the connection.
The connection is now in CLOSE_WAIT on the Squid server and will remain
like that indefinitely.

So it seems apparent that after Squid delivers the clear-text response,
it abandons the socket but never closes it. From looking in the source,
this is client_side.cc, and it has a comment:
     // XXX: Can this happen? CONNECT tunnels have deferredRequest set.
It looks to me as if the (conn->flags.readMore) section above should be
the bit being executed, although I don't quite understand deferred
requests. In either case, it seems like we should close the socket if
it ever gets abandoned?

A few notes on the REQMOD rewrite:
My ICAP service can generate a "403 Forbidden" response during REQMOD,
or rewrite the request to a GET to a local web server. In the latter
case, the local webserver generates a "403 Forbidden" response. In both
cases the client sees a similar response to its request, and in both
cases this bug manifests.

Interestingly, if the ICAP service closes the ICAP connection instead of
returning a response, Squid generates a "500 Internal Server Error" and
does not abandon the socket (the client then drops the connection, which
squid handles correctly, and therefore doesn't end in CLOSE_WAIT).

-- 
  - Steve Hill
    Technical Director
    Opendium Limited     http://www.opendium.com
Direct contacts:
    Instant messager: xmpp:steve_at_opendium.com
    Email:            steve_at_opendium.com
    Phone:            sip:steve_at_opendium.com
Sales / enquiries contacts:
    Email:            sales_at_opendium.com
    Phone:            +44-844-9791439 / sip:sales_at_opendium.com
Support contacts:
    Email:            support_at_opendium.com
    Phone:            +44-844-4844916 / sip:support_at_opendium.com