Re: [squid-users] Arrrrrrrgh! Yahoo mail, Gmail, Hotmail, MSN troubles on Squid2.6STABLE4 transparent

Gonzalo,

This issue has been RESOLVED.

You were correct, the problem was indeed related to WCCP. Our 7206
router has only 256MB memory and when we added a second BGP peer some
time ago, we were able to receive the full routing table and still had
some memory left over. The cache was working fine for some time after
that, so I did not suspect the router to be the cause of the problem.

However when we attempted to upgrade the IOS to a newer version in
hopes of resolving the cache issue, the router could not load the full
routing table from both peers and ended up in an out-of-memory 'loop'
behaviour.

When we downgraded to the previous IOS version, to my dismay the same
out-of-memory problem occurred! We had to block the receiving of
routes from one peer in order to have any connectivity at all.

I believe what happened was that somehow there was just enough memory
to load the BGP routing table when we originally brought up the second
peer. Over time as memory was used it became fragmented or suffered
from a memory leak and caused issues with our cache. Then when we
rebooted with the same IOS version, it couldn't pull off the same
trick (loading the routing table from both peers) a second time.

Now we are buying an NPE400 so that we can go to 512MB memory and be
able to run a newer, more featureful IOS image and still have memory
to spare. I hope that others will find this post and not have to go
through the week of hell I went through! Thanks so much for all your
help.

Regards,

Tom

On 10/12/06, Gonzalo Arana <gonzalo.arana@gmail.com> wrote:
> On 10/11/06, Tom Warren <funkknob@gmail.com> wrote:
> > Hi Gonzalo!
> >
> > On 10/11/06, Gonzalo Arana <gonzalo.arana@gmail.com> wrote:
> > > Hi,
> > >
> > > What kinds of errors are you getting? Timeouts?
> >
> > It depends on which browser and which site, but with MSN we're getting
> > 'connection reset' on Firefox and 'page cannot be displayed' on IE.
> > With gmail we can log in but when we click on an inbox message using
> > Firefox, 'loading...' is displayed but it just hangs there; with IE we
> > get 'page cannot be displayed.' With Yahoo mail we can log in and read
> > and compose messages, but get a blank page after pressing 'send.'
>
> the http status of the response is quite useful. Try searching in
> access logs for the lines of the requests that fail.
>
> > > Anything in your cache.log?
> >
> > There's nothing in cache.log related to the problem.
> >
> > > Try to get the access.log lines that represent the problem.
> >
> > Here are some examples from Gmail:
> >
> > 11/Oct/2006:16:23:07 +0700.740 212 xxx.yyy.24.34 TCP_MISS/204 240
> > GET http://www.google.com/setgmail? - DIRECT/64.233.189.104 text/html
> > 11/Oct/2006:16:23:08 +0700.687 1038 xxx.yyy.24.34 TCP_MISS/200 385
> > GET http://chatenabled.mail.google.com/mail/images/cleardot.gif? -
> > DIRECT/64.233.163.189 image/gif
> >
> > Those are the only lines that show up when we click on a message in the inbox.
>
> 204 No content
> and
> 200 OK
> they seem ok to me.
>
> >
> > >
> > > Here is a minimal troubleshooting guide:
> > >
> > > 1) Try to access them without squid, using any web browser (explorer, firefox).
> >
> > This always works with both browsers. Also works if we set the
> > browser's proxy settings to point directly to Squid.
>
> this suggests to look into wccp. Any wccp guru around?
>
> > > 2) Try to access using elinks (text browser with some javascript &
> > > frames support) from a squid server without wccp.
> >
> > When signing into gmail using elinks from the cache, I get 'The page
> > you requested is invalid.'
> >
> > Hotmail gives a message about javascript; I couldn't figure out how to
> > enable it in elinks.
> >
> > > 3) Try to access them using plain old telnet without wccp (just to
> > > test your TCP stack, perhaps you may have a broken tcp timestamps
> > > implementation, for instance).
> >
>
> > Cache-Control: private
> > Content-Encoding: gzip
> > Content-Length: 12254
> >
> > ³QtpU?uP??7v?w qQ?Q0?.)?.?w?))°?///?7?/J?
> > ???e?¤(??????·UJ?--VR@??nhii
> > 1dPjb?MnjI"?%º(c)?¥?e¶J?y%(c)y%º!?(c)J
>
> The "garbage" you see is the content of the response compressed with
> gzip (as instructed in 'Content-Encoding: gzip' response header). The
> web server sends the response compressed because you instructed it
> that it is ok to send gzipped responses by sending 'Accept-Encoding:
> gzip, deflate' in request.
>
> > ........Lots of garbage follows. BTW how can I reset my terminal to
> > get rid of the garbage text without logging out?
>
> try with
> echo -ne '\033c'
>
> > > 4) Try to access them using telnet from your Cisco 7206 using
> > > different ip source address (to discard routing problems).
> >
> > Similar result:
>
> so routing problems may be discarded.
>
> > Content-Length: 12253
>
> > But it looks like the content length is one byte off?
>
> most likely the content have changed.
>
> > Also here's some tcpdump output when trying to connect to MSN:
>
> >
>
> > 17:29:28.653892 IP xxx.yyy.24.34.1700 > 207.68.173.76.http: S
> > 1196943964:1196943964(0) win 65535 <mss 1460,nop,nop,sackOK>
> > 17:29:28.653923 IP 207.68.173.76.http > xxx.yyy.24.34.1700: S
> > 3187314926:3187314926(0) ack 1196943965 win 5840 <mss
> > 1460,nop,nop,sackOK>
> > 17:29:28.654640 IP xxx.yyy.24.34.1700 > 207.68.173.76.http: . ack
> > 3187314927 win 65535
>
> the above 3 packets are TCP connection establishment
>
> > 17:29:28.673760 IP xxx.yyy.24.34.1700 > 207.68.173.76.http: P
> > 1460:1730(270) ack 1 win 65535
>
> this is the request (270 bytes long)
>
> > 17:29:28.673777 IP 207.68.173.76.http > xxx.yyy.24.34.1700: . ack 1
> > win 5840 <nop,nop,sack sack 1 {1461:1731} >
>
> This is the web server ACK of the request, but the response is missing.
>
> My guess is that this is a wccp related problem. Unfortunately, I've
> never used wccp and I am not aware of how it works.
>
> Hope someone else can help you.
>
> Regards,
>
> --
> Gonzalo A. Arana
>
Received on Mon Oct 16 2006 - 22:45:49 MDT