RE: [squid-users] Streaming asx files through squid

> -----Original Message-----
> From: Amos Jeffries [mailto:squid3@treenet.co.nz]
> Sent: 16 January 2008 21:12
> To: Tris Hoar
> Cc: squid-users@squid-cache.org
> Subject: Re: [squid-users] Streaming asx files through squid
>
> > Hi List,
> >
> > We are fairly new to using squid and are in the process of migrating 400
> > odd
> > proxy servers over to it. This is going fairly smoothly at the moment.
> > However some of the customers have reported that
> http://www.heartfm.co.uk
> > is
> > no longer streaming radio for them.
> >
> > The server that the audio streams from firstly sends this HTML file to
> the
> > client
> >
> > <asx version = "3.0">
> > <title>Heart West Midlands 100.7 Live Stream</title>
> > <entry>
> >
> > <title>LIVE :
> > Heart1007</title>
> >
> > <ref
> > href="http://wm-global.london.as34763.net/Heart1007" />
> >
> > </entry>
> >
> > </asx>
> >
> >
> > Then http://wm-global.london.as34763.net/Heart1007
> >
> >
> >
> > Points to
> >
> >
> >
> > [Reference]
> >
> > Ref1=http://wm-global.london.as34763.net/Heart1007?MSWMExt=.asf
> >
> > Ref2=http://192.168.97.221:80/Heart1007?MSWMExt=.asf
> >
> >
> >
> > If we use http://wm-global.london.as34763.net/Heart1007?MSWMExt=.asf
> > directly in Media player 11 this works through squid
> >
> >
> >
> > Our setup is a local squid server at each site > to DNS round robin of
> > parent servers in the core > Internet
> >
> > The squid servers are built on Xeon 3.0 GHz servers with 2GB ram and
> > ~500GB
> > disk space of which 400GB is cache. The OS is REHL 5.1 and running Squid
> > 2.6.STABLE6 (we are using this version due to redhat offering support
> for
> > it)
>
> For that size disk cache the RAM is very low. A rule of thumb for general
> principles is to dedicate 10MB-RAM per 1 GB-disk just to hold the cache
> index. If your primary business is streaming media you can get away with
> less (very-large files need less index per disk byte).

The customers are Schools and many of them have subscriptions to services
like Espresso and Knowledgebox. These services are web based and have a lot
of video content on them. The plan is to pre populate most of the cache
space with the video content using an application called cachestream. Thus
the low memory should not be a problem.

>
> >
> > Using the parents directly or the old proxy server's this streams fine
> > though windows media player 11.
> >
> >
> >
> > Our squid config is
> >
> > ------------------------------------------------------------------------
> ----
> >
> > http_port 80
>
> Is this meant to be a website accelerator or customer proxy?
>
> Accelerators need accel or vhost and could do with defaultsite= in their
> http_port.
>

The servers are intended to function as a standard web proxy's, not as
accelerators.

>
> >
> > hierarchy_stoplist cgi-bin ?
> >
> > acl QUERY urlpath_regex cgi-bin \?
> >
> > cache deny QUERY
> >
>
> Are you sure you don't want to cache anything with '?' in the URI?
> The dev team are now recommending killing this and using a refresh_pattern
> instead to allow caching of properly working dynamic pages/files.
>

We were just using the example included with the default squid.conf for
this. Do you have an example of a refresh_pattern we could try?

>
> >
> > acl apache rep_header Server ^Apache
> >
> > broken_vary_encoding allow apache
> >
> >
> > cache_replacement_policy heap LRU
> >
> > memory_replacement_policy heap GDSF
> >
> > cache_dir aufs /var/spool/squid 400000 64 256
> >
> > access_log /var/log/squid/access.log
> >
> > access_log none
> >
>
> This second access_log line is not correct. 'none' is not a proper
> filename for the log, did you mean 'cache_log none'?.
> I would recommend having a cache_log with 'debug_options ALL,0' so that
> you get a log of even just the critical errors squid has.
>

Opps! We have a duplicated line
access_log /var/log/squid/access.log <- should not have been there
We monitor this from our core servers rather than the edge devices. Though
while testing I will have the log on.

> >
> > refresh_pattern ^ftp: 1440 20% 10080
> >
> > refresh_pattern ^gopher: 1440 0% 1440
> >
> > refresh_pattern . 0 20% 4320
> >
> >
> >
> >
> >
> > acl all src 0.0.0.0/0.0.0.0
> >
> > acl manager proto cache_object
> >
> > acl localhost src 127.0.0.1/255.255.255.255
> >
> > acl to_localhost dst 127.0.0.0/8
> >
> > acl SSL_ports port 443 563 8443
> >
> > acl Safe_ports port 80 81 21 443 563 70 210 280 554 488 591 777 889
> > 1025-65535
> >
> > acl CONNECT method CONNECT
> >
> > acl Proxy_bypass dstdomain .misportal.net .bgfl.org
> >
> >
> >
> >
> >
> > #Access Control Rules
> >
> > http_access allow manager localhost
> >
> > http_access deny manager
> >
> > http_access deny !Safe_ports
> >
> > http_access deny CONNECT !SSL_ports
> >
> > http_access allow localhost
> >
> > http_access allow all
>
> Oh yay! An open web proxy on a high-speed server.
>
> If this is actually a site accelerator note the comment after http_port
> above, and you will need some ACL restricting the destination sites by
> name or rDNS. Probably listing some cache_peer's as preferred information
> sources.
>

The proxy sits out at the customer's site which connects back to our servers
over a 10MB MPLS connection. This connection is firewalled from the internet
and restricted with ACL's on the routers at each site so that only people at
that site or at our office can connect to the server. Every one at the site
MUST use the proxy for internet access however we do not wish to use a
transparent proxy. Rather than having a custom config for each site we allow
any one to connect to the server as only people at that site will be able
to.

> >
> > http_reply_access allow all
> >
> > icp_access allow all
> >
> >
> >
> > always_direct allow Proxy_bypass
> >
> > never_direct allow all
> >
> >
> >
> > nonhierarchical_direct off
> >
> >
> >
> > coredump_dir /var/spool/squid
> >
> > visible_hostname eduproxy.bgfl.org
> >
> >
> >
> > cache_effective_user squid
> >
> > cache_effective_group squid
> >
> > cachemgr_passwd xxxxxx info stats/objects
> >
> >
> >
> > ------------------------------------------------------------------------
> ----
> >
> >
> >
> > If you can shed any light on how to resolve this that would be great!
> >
>
> Firstly check that the clients software is even using the proxy.
> This should show up as entries in access.log for the stream URI.
>

This is showing up in the access log fine.
TCP_MISS/200 6310 GET http://meta.as34763.net/content/9.asx -
FIRST_UP_PARENT/parentproxy.bgfl.org video/x-ms-asf

> Then enable a cache_log and see if there are any serious problems
> happening in squid about that time.

Nothing showing up in the cache_log that seams wrong

>
> Then check the request headers going to/from squid for one of the failing
> requests. Also, does it work with any other software? and what does that
> show?
>

Looking at TCPDump I think the problem is due to the streaming server (which
we do not control) trying to talk in HTTP1.1 and Squid not understanding it.
Unless we can find a better solution what we are planning to do is set the
core servers to just rewrite the url from
http://meta.as34763.net/content/9.asx
Into
http://wm-global.london.as34763.net/Heart1007?MSWMExt=.asf

> If that does not lead you to a solution, we will need to know the squid
> release (including the stable part) and what you have managed to find out
> from trying the above.
>
>

2.6.Stable6 - this is the RedHat maintained release.

> Amos
>
>

Kind Regards

Tris

>
>
> *************************************************************
> This message has been checked for viruses by the
> Birmingham Grid for Learning. For guidance on good
> e-mail practice, e-mail viruses and hoaxes please visit:
> http://www.bgfl.org/connectivity/e-mail.htm
> *************************************************************

*************************************************************
This email and any files transmitted with it are confidential
and intended solely for the use of the individual or entity
to whom they are addressed. If you have received this email
in error please notify postmaster@bgfl.org

The views expressed within this email are those of the
individual, and not necessarily those of the organisation
*************************************************************
Received on Thu Jan 17 2008 - 08:38:16 MST