Re: [squid-users] CLOSE_WAIT

On 11/01/2013 7:11 a.m., Steve Hill wrote:
> On 09/01/13 21:07, Amos Jeffries wrote:
>
>> Does the CONNECT request contain Connection:close or
>> Connection:keep-alive? Squid supports keep-alive on CONNECT requests in
>> these situations where the CONNECT size is known and may be waiting for
>> another client request.
>
> The client sends "Proxy-Connection: keep-alive", so it would indeed be
> possible for the connection to be reused. I should add that I'm
> seeing this with a non-transparent configuration (i.e. without Tproxy
> and without ssl_bump).
>
>> Please upgrade to 3.2.6 (should anyway for the CVE resolved there) and
>> see if this issue is gone there.
>
> I've now upgraded my test server and the problem remains.
>
> The "403 Forbidden" being sent back to the client didn't have a
> Content Length header, which of course caused the client to drop the
> connection. If I add a Content Length, the connection stays alive and
> I think it's a bit more obvious whats happening:
>
> 1. Client connects to Squid and sends a CONNECT.
> 2. Squid passes the request to the ICAP REQMOD service.
> 3. ICAP rewrites the request.
> 4. Squid returns a "403 Forbidden" to the client in the clear. At
> this point, the connection is still alive with empty queues. Squid's
> cache.log shows:
> 2013/01/10 17:52:18 kid1| abandoning local=[2a00:1a90:5::9]:3128
> remote=[2001:4d48:ad51:501:226:bbff:fe18:f3ff]:49926 FD 19 flags=1
>
> Now, the user retries the connection:
> 5. Client sends CONNECT over the existing connection.
> 6. Squid does not read the socket (netstat shows the rx queue on the
> socket is 220 octets long, which is the whole request the client just
> sent).
> 7. The client sits there spinning.
> 8. Eventually the client times out and drops the connection.
> The connection is now in CLOSE_WAIT on the Squid server and will
> remain like that indefinitely.
>
> So it seems apparent that after Squid delivers the clear-text
> response, it abandons the socket but never closes it. From looking in
> the source, this is client_side.cc, and it has a comment:
> // XXX: Can this happen? CONNECT tunnels have deferredRequest set.
> It looks to me as if the (conn->flags.readMore) section above should
> be the bit being executed, although I don't quite understand deferred
> requests. In either case, it seems like we should close the socket if
> it ever gets abandoned?
>
>
> A few notes on the REQMOD rewrite:
> My ICAP service can generate a "403 Forbidden" response during REQMOD,
> or rewrite the request to a GET to a local web server. In the latter
> case, the local webserver generates a "403 Forbidden" response. In
> both cases the client sees a similar response to its request, and in
> both cases this bug manifests.
>
> Interestingly, if the ICAP service closes the ICAP connection instead
> of returning a response, Squid generates a "500 Internal Server Error"
> and does not abandon the socket (the client then drops the connection,
> which squid handles correctly, and therefore doesn't end in CLOSE_WAIT).
>

Okay. So the source of the problem is that Squid thinks there is
something using the socket, but it never got into the tunnel code which
would have closed it?
  Is the 403 message being generated inside Squid or by ICAP?

In the case where the server generates the 403 and things hang it will
be because Squid is expecting a close to arrive from server or client.
Once a tunnel is open and transmitting data it is up to those endpoints
to ensure keep-alive and other such details, although Squid applies a
timeout since last byte transferred.

Amos
Received on Fri Jan 11 2013 - 00:06:39 MST