Re: [squid-users] Blank pages off www.msn.com

wrh_at_wootsie.com wrote:
> Hello,
>
> This just started happening -
>
> We are getting blank pages returned from our Squid 2.6.STABLE4 proxy
> from the search results pages off www.msn.com. And MSN's
> maps&directions site comes up blank as well. The www.msn.com home
> page is fine.
>
> Happening with IE7 and FF3.
>
> No explanation for it on the squid server, nothing in the logs.
>
> If I don't use squid, the pages appear.
>
> Our squid has been unchanged for a long while.
>
> Is anyone else experiencing this issue?
>
>
>
> Thanks,
>
> -Bill
>
I may be wrong, but it looks like MSN explicitly refuses proxied
connections. If I hide the proxy with "forwarded_for off" it suddenly works!

The following shows radically different headers depending on whether the
X-Forwarded-For header is present or not.

Trying the URL
http://search.msn.com/results.aspx?q=debian+lenny+calamaris&FORM=MSNH11
using wget -S

First with standard proxy settings
==================================
--2009-03-27 14:10:40--
http://search.msn.com/results.aspx?q=debian+lenny+calamaris&FORM=MSNH11
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:8080... connected.
Proxy request sent, awaiting response...
  HTTP/1.1 200 OK
  X-Powered-By: ASP.NET
  Content-Length: 0
  Expires: Fri, 27 Mar 2009 14:10:40 GMT
  Cache-Control: max-age=0, no-cache, no-store
  Pragma: no-cache
  Date: Fri, 27 Mar 2009 14:10:40 GMT
  X-Cache: MISS from cpc3-staf1-0-0-cust235.sol2.cable.ntl.com
  Via: 1.1 cpc3-staf1-0-0-cust235.sol2.cable.ntl.com:8080
(squid/2.7.STABLE6)
  Connection: close
Length: 0
Saving to: `results.aspx?q=debian+lenny+calamaris&FORM=MSNH11'

    [
<=>

] 0 --.-K/s in 0s

2009-03-27 14:10:40 (0.00 B/s) -
`results.aspx?q=debian+lenny+calamaris&FORM=MSNH11' saved [0/0]

Without the proxy
=================
--2009-03-27 14:10:56--
http://search.msn.com/results.aspx?q=debian+lenny+calamaris&FORM=MSNH11
Resolving search.msn.com... 84.53.178.209, 84.53.178.218
Connecting to search.msn.com|84.53.178.209|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.0 200 OK
  Content-Length: 31048
  Content-Type: text/html; charset=utf-8
  X-Powered-By: ASP.NET
  P3P: CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND",
policyref="http://privacy.msn.com/w3c/p3p.xml"
  Date: Fri, 27 Mar 2009 14:10:56 GMT
  Connection: keep-alive
  Set-Cookie: AFORM=MSNH11; expires=Mon, 20-Jul-2015 23:59:59 GMT; path=/
  Set-Cookie: mkt1=norm=-GB; domain=.msn.com; path=/
  Set-Cookie: mkt2=ui=en-GB; domain=search.msn.com; path=/
  Set-Cookie: MUID=95278E835C21487EB0C3B5390C5735BE; expires=Mon,
20-Jul-2015 23:59:59 GMT; domain=.msn.com; path=/
  Set-Cookie: OrigMUID=95278E835C21487EB0C3B5390C5735BE; expires=Mon,
20-Jul-2015 23:59:59 GMT; domain=.msn.com; path=/
  Set-Cookie:
OVR=flt=0&PerfTracking=0&DomainVertical=0&Cashback=0&MSCorp=kiev;
domain=.msn.com; path=/
  Set-Cookie: RMS=SERP=320&T=48; path=/
  Set-Cookie: SRCHD=D=650290&AF=MSNH11; expires=Mon, 20-Jul-2015
23:59:59 GMT; domain=.msn.com; path=/
  Set-Cookie:
SRCHSESS=GUID=659B63EAB2A64E9597884654B10FF49E&TS=1238163056;
expires=Fri, 27-Mar-2009 14:30:56 GMT; path=/
  Set-Cookie: SRCHUID=V=2&GUID=83EF0003D92A4C429D5D6DBCA44B86C8;
expires=Mon, 20-Jul-2015 23:59:59 GMT; path=/
  Set-Cookie: SRCHUSR=AUTOREDIR=0&GEOVAR=&DOB=20090327; expires=Sun,
27-Mar-2011 14:10:56 GMT; path=/
  Cache-Control: private
Length: 31048 (30K) [text/html]
Saving to: `results.aspx?q=debian+lenny+calamaris&FORM=MSNH11.1'

100%[==========================================================================================>]
31,048 199K/s in 0.2s

2009-03-27 14:10:57 (199 KB/s) -
`results.aspx?q=debian+lenny+calamaris&FORM=MSNH11.1' saved [31048/31048]

Now set "forwarded_for off" in squid.conf
=========================================
--2009-03-27 14:27:51--
http://search.msn.com/results.aspx?q=debian+lenny+calamaris&FORM=MSNH11
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:8080... connected.
Proxy request sent, awaiting response...
  HTTP/1.1 200 OK
  Content-Length: 31048
  Content-Type: text/html; charset=utf-8
  X-Powered-By: ASP.NET
  P3P: CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND",
policyref="http://privacy.msn.com/w3c/p3p.xml"
  Date: Fri, 27 Mar 2009 14:27:51 GMT
  Set-Cookie: AFORM=MSNH11; expires=Mon, 20-Jul-2015 23:59:59 GMT; path=/
  Set-Cookie: mkt1=norm=-GB; domain=.msn.com; path=/
  Set-Cookie: mkt2=ui=en-GB; domain=search.msn.com; path=/
  Set-Cookie: MUID=71DF4074D8BE456381CDA9945EE5AE96; expires=Mon,
20-Jul-2015 23:59:59 GMT; domain=.msn.com; path=/
  Set-Cookie: OrigMUID=71DF4074D8BE456381CDA9945EE5AE96; expires=Mon,
20-Jul-2015 23:59:59 GMT; domain=.msn.com; path=/
  Set-Cookie:
OVR=flt=0&PerfTracking=0&DomainVertical=0&Cashback=0&MSCorp=kiev;
domain=.msn.com; path=/
  Set-Cookie: RMS=SERP=320&T=48; path=/
  Set-Cookie: SRCHD=D=650307&AF=MSNH11; expires=Mon, 20-Jul-2015
23:59:59 GMT; domain=.msn.com; path=/
  Set-Cookie:
SRCHSESS=GUID=E8E98A7C50064824A06EC246214D405D&TS=1238164071;
expires=Fri, 27-Mar-2009 14:47:51 GMT; path=/
  Set-Cookie: SRCHUID=V=2&GUID=BE07F5751A95407FB6E94683FB41EDBB;
expires=Mon, 20-Jul-2015 23:59:59 GMT; path=/
  Set-Cookie: SRCHUSR=AUTOREDIR=0&GEOVAR=&DOB=20090327; expires=Sun,
27-Mar-2011 14:27:51 GMT; path=/
  Cache-Control: private
  X-Cache: MISS from cpc3-staf1-0-0-cust235.sol2.cable.ntl.com
  Via: 1.1 cpc3-staf1-0-0-cust235.sol2.cable.ntl.com:8080
(squid/2.7.STABLE6)
  Connection: close
Length: 31048 (30K) [text/html]
Saving to: `results.aspx?q=debian+lenny+calamaris&FORM=MSNH11.2'

100%[==========================================================================================>]
31,048 --.-K/s in 0.1s

2009-03-27 14:27:51 (214 KB/s) -
`results.aspx?q=debian+lenny+calamaris&FORM=MSNH11.2' saved [31048/31048]
Received on Fri Mar 27 2009 - 14:34:27 MDT