Re: [squid-users] caching googlevideo.com with squid

On Tuesday 31 March 2009 13:32:08 Gavin McCullagh wrote:
> Hi,
>
> running squid in transparent mode for our network, we're trying to
> maximised our byte cache hit ratio. Isn't everyone ;-)
>
> I've been looking over reports generated by SRG and I guess shouldn't be
> surprised to find that youtube/goodlevideo (when you add up the 250
> different mirror servers) accounts for around 15% of our bandwidth usage
> (c. 28GB per day in c. 300 requests) and has a 0% hit rate. This seems
> worthy of some effort.
>
> So, I've been looking at these pages:
>
> http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube
> http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube/Discussio
>n
>
> I gather I'd need to upgrade to squid 2.7 (for storeurl) which I can do.
> The remainder looks doable, if a little complex.
>
> Is there a big performance hit involved in spawning the perl script? I
> guess those ACLs will get hit relatively rarely so perhaps not?
>
> I guess the perl script needs to be modified every time google/youtube
> change subdomain names. Do I presume this doesn't happen too often?
>
> Does anyone else have any experience or advice on this? What sort of hit
> rate do you tend to get in practice? There's a lot of video on
> goodlevideo, but I guess certain videos tend to become highly popular for a
> few days.
>
> Many thanks in advance for any suggestions,
> Gavin

Hi Gavin,

I've tried a lot of methods of caching youtube/googlevideos.

I also tried the method you mentioned. I've decided against it, mostly because
of performance issues.
As squid cannot utilize SMP machines, you will be forced to use sibling
relationships if you plan on exceeding 600-700~ mbit per second, and even
then be prepared for a lot of performance issues.

I also tried videocache (http://cachevideos.com/) which is more scaleable, and
easier to maintain, but suffered from stability issues (mainly with the RPC
component - it tends to hang at high loads).

I ended up rewriting videocache (using perl), and I am now pushing 1.5 Gbit of
youtube traffic on a single HP DL 360 G5 machine (2 Intel E5410 quad core
CPUs).

To summarize, I would recommend checking videocache out, as I know the
developer is aware of the RPC problem and he is working to solve it.
Received on Tue Mar 31 2009 - 12:51:08 MDT