Re: [squid-users] Arrrrrrrgh! Yahoo mail, Gmail, Hotmail, MSN troubles on Squid2.6STABLE4 transparent

Hi Gonzalo!

On 10/11/06, Gonzalo Arana <gonzalo.arana@gmail.com> wrote:
> Hi,
>
> What kinds of errors are you getting? Timeouts?

It depends on which browser and which site, but with MSN we're getting
'connection reset' on Firefox and 'page cannot be displayed' on IE.
With gmail we can log in but when we click on an inbox message using
Firefox, 'loading...' is displayed but it just hangs there; with IE we
get 'page cannot be displayed.' With Yahoo mail we can log in and read
and compose messages, but get a blank page after pressing 'send.'

> Anything in your cache.log?

There's nothing in cache.log related to the problem.

> Try to get the access.log lines that represent the problem.

Here are some examples from Gmail:

 11/Oct/2006:16:23:07 +0700.740 212 xxx.yyy.24.34 TCP_MISS/204 240
GET http://www.google.com/setgmail? - DIRECT/64.233.189.104 text/html
 11/Oct/2006:16:23:08 +0700.687 1038 xxx.yyy.24.34 TCP_MISS/200 385
GET http://chatenabled.mail.google.com/mail/images/cleardot.gif? -
DIRECT/64.233.163.189 image/gif

Those are the only lines that show up when we click on a message in the inbox.

>
> Here is a minimal troubleshooting guide:
>
> 1) Try to access them without squid, using any web browser (explorer, firefox).

This always works with both browsers. Also works if we set the
browser's proxy settings to point directly to Squid.

> 2) Try to access using elinks (text browser with some javascript &
> frames support) from a squid server without wccp.

When signing into gmail using elinks from the cache, I get 'The page
you requested is invalid.'

Hotmail gives a message about javascript; I couldn't figure out how to
enable it in elinks.

> 3) Try to access them using plain old telnet without wccp (just to
> test your TCP stack, perhaps you may have a broken tcp timestamps
> implementation, for instance).

I tried:

*******************************************************************************************

[root@cache ~]# telnet mail.google.com 80
Trying 64.233.163.19...
Connected to mail.google.com (64.233.163.19).
Escape character is '^]'.
GET /mail HTTP/1.1
Accept: */*\r\n
Accept-Language: en-us\r\n
Referer: http://mail.google.com/mail
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: mail.google.com

HTTP/1.1 302 Moved Temporarily
Set-Cookie: GMAIL_HELP=EXPIRED;Path=/;Expires=Mon, 01-Jan-1990 00:00:00 GMT
Set-Cookie: GV=EXPIRED;Domain=mail.google.com;Path=/;Expires=Mon,
01-Jan-1990 00:00:00 GMT
Set-Cookie: GV=EXPIRED;Domain=mail.google.com;Path=/mail;Expires=Mon,
01-Jan-1990 00:00:00 GMT
Location: https://www.google.com/accounts/ServiceLogin?service=mail&passive=true&rm=false&continue=http%3A%2F%2Fmail.google.com%2Fmail%3Fui%3Dhtml%26zy%3Dl&ltmpl=cm_blanco&ltmplcache=2
Cache-control: private
Content-Length: 0
Content-Type: text/html
Server: GFE/1.3
Date: Thu, 12 Oct 2006 02:16:30 GMT

*******************************************************************************************

Unfortunately I have no way to test further since it redirects to https.

But here's MSN:

*******************************************************************************************

[root@cache ~]# telnet www.msn.com 80
Trying 207.68.183.35...
Connected to www.msn.com (207.68.183.35).
Escape character is '^]'.
GET / HTTP/1.1
Accept: */*\r\n
Accept-Language: en-us\r\n
Referer: http://mail.google.com/mail
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host:www.msn.com

HTTP/1.1 200 OK
Date: Thu, 12 Oct 2006 02:21:11 GMT
Server: Microsoft-IIS/6.0
P3P: CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"
S: appB32
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Pragma: no-cache
Set-Cookie: MC1=V=3&GUID=b3968637065047c5865107321e491848;
domain=.msn.com; expires=Mon, 04-Oct-2021 19:00:00 GMT; path=/
Set-Cookie: mh=MSFT; domain=.msn.com; expires=Sun, 09-Oct-2016
02:21:11 GMT; path=/
Set-Cookie: CULTURE=EN-US; domain=.msn.com; expires=Thu, 19-Oct-2006
02:21:11 GMT; path=/
Set-Cookie: ushpsvr=M:5|F:5|T:5|E:5|D:blu|W:F; domain=.msn.com;
expires=Sun, 09-Oct-2016 02:21:11 GMT; path=/
Set-Cookie: ushpcli=0|H.0.1|G.0.1|Z.0.1|R.0.1.cap|C.0.1.lg:newyorkny|L.0.1.LN:WNBC;
domain=.msn.com; expires=Sun, 09-Oct-2016 02:21:11 GMT; path=/
Set-Cookie: ushpwea=wc:USNY0996; domain=.msn.com; expires=Sun,
09-Oct-2016 02:21:11 GMT; path=/
Cache-Control: no-cache
Content-Type: text/html; charset=utf-8
Cache-Control: private
Content-Encoding: gzip
Content-Length: 12254

³QtpU?uP??7v?w qQ?Q0?.)?.?w?))°?///?7?/J?
???e?¤(??????·UJ?--VR@??nhii
1dPjb?MnjI"?%º(c)?¥?e¶J?y%(c)y%º!?(c)J

*******************************************************************************************

........Lots of garbage follows. BTW how can I reset my terminal to
get rid of the garbage text without logging out?

> 4) Try to access them using telnet from your Cisco 7206 using
> different ip source address (to discard routing problems).

Similar result:

*******************************************************************************************

core#telnet www.msn.com 80
Translating "www.msn.com"...domain server (202.79.24.152) [OK]
Trying www.msn.com.nsatc.net (207.68.183.35, 80)... Open
GET / HTTP/1.1
Accept: */*\r\n
Accept-Language: en-us\r\n
Referer: http://mail.google.com/mail
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host:www.msn.com

HTTP/1.1 200 OK
Date: Thu, 12 Oct 2006 02:24:30 GMT
Server: Microsoft-IIS/6.0
P3P: CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"
S: appB32
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Pragma: no-cache
Set-Cookie: MC1=V=3&GUID=0643d1d0d09941b9849828c494dd7d7b;
domain=.msn.com; expires=Mon, 04-Oct-2021 19:00:00 GMT; path=/
Set-Cookie: mh=MSFT; domain=.msn.com; expires=Sun, 09-Oct-2016
02:24:30 GMT; path=/
Set-Cookie: CULTURE=EN-US; domain=.msn.com; expires=Thu, 19-Oct-2006
02:24:30 GMT; path=/
Set-Cookie: ushpsvr=M:5|F:5|T:5|E:5|D:blu|W:F; domain=.msn.com;
expires=Sun, 09-Oct-2016 02:24:30 GMT; path=/
Set-Cookie: ushpcli=0|H.0.1|G.0.1|Z.0.1|R.0.1.cap|C.0.1.lg:newyorkny|L.0.1.LN:WNBC;
domain=.msn.com; expires=Sun, 09-Oct-2016 02:24:30 GMT; path=/
Set-Cookie: ushpwea=wc:USNY0996; domain=.msn.com; expires=Sun,
09-Oct-2016 02:24:30 GMT; path=/
Cache-Control: no-cache
Content-Type: text/html; charset=utf-8
Cache-Control: private
Content-Encoding: gzip
Content-Length: 12253

³QtpU?uP??7v?w qQQ0?.)?.?w?))°?///?7?/J?
???e?¤(??????·UJ?--VR@??nhii 1dPjb?MnjI"?%º(c)

*******************************************************************************************

But it looks like the content length is one byte off?

Also here's some tcpdump output when trying to connect to MSN:

*******************************************************************************************

17:29:28.653892 IP xxx.yyy.24.34.1700 > 207.68.173.76.http: S
1196943964:1196943964(0) win 65535 <mss 1460,nop,nop,sackOK>
17:29:28.653923 IP 207.68.173.76.http > xxx.yyy.24.34.1700: S
3187314926:3187314926(0) ack 1196943965 win 5840 <mss
1460,nop,nop,sackOK>
17:29:28.654640 IP xxx.yyy.24.34.1700 > 207.68.173.76.http: . ack
3187314927 win 65535
17:29:28.673760 IP xxx.yyy.24.34.1700 > 207.68.173.76.http: P
1460:1730(270) ack 1 win 65535
17:29:28.673777 IP 207.68.173.76.http > xxx.yyy.24.34.1700: . ack 1
win 5840 <nop,nop,sack sack 1 {1461:1731} >

*******************************************************************************************

>
> Hope this helps,

Thanks!

>
> On 10/11/06, Tom Warren <funkknob@gmail.com> wrote:
> > A few days ago we suddenly began experiencing hangs and getting
> > (browser) error pages for many webmail and other popular sites. First
> > complaint was about Yahoo mail on Monday, then Hotmail yesterday, and
> > today MSN won't load either! The problem affects both IE6 and Firefox
> > 1.5.0.7 and can be worked-around by excluding the user's IP from wccp
> > on the router or by pointing directly to the cache in the browser's
> > proxy settings.
> >
> > We have tried a multitude of suggestions parsed from this and other
> > forums, including the MSS iptables fix, 'Accept-Encoding deny',
> > no_cache, disabling various TCP parameters in Linux and also removing
> > things that break http like 'reload_into_ims on'. But the problem
> > persists and seems to be getting worse over time. No changes were made
> > to the cache configuration prior to the start of the problem.
> >
> > Yesterday we thought we were able to solve the issue by clearing
> > cookies at one customer site, but the problem returned and now
> > clearing cookies does nothing. We thought it may be due to NAT but
> > today it is affecting workstations in our NOC on public IPs directly
> > connected to our core router.
> >
> > I saw a similar complaint regarding Yahoo mail recently and I'm
> > curious if anyone else is having similar problems. I'll be happy to
> > post any pertinent information if I can get an idea of where to start
> > troubleshooting and what information is required. The basics are
> > Fedora 4, Squid2.6STABLE4 using WCCP from a Cisco 7206.
> >
> > Regards,
> >
> > Tom
> >
>
>
> --
> Gonzalo A. Arana
Received on Wed Oct 11 2006 - 20:36:35 MDT