Re: [squid-users] Problems with squid in a campus setup

On 27/03/2012 12:25, Christian Loth wrote:
> Hello,
>
> first of all, thank you for your recommendations.
>
> On Monday 26 March 2012 16:34:21 Marcus Kool wrote:
>> Youtube may be hogging your pipe but it is better to know than to guess.
>
> Of course, before we decided for the proxy setup we investigated bandwidth
> usage. HTTP Traffic was about 60-70% of our traffic, and a good chunk of that was
> youtube. That's why we decided, that we would try a squid setup with youtube
> caching. However it took a while before we found a solution for caching
> youtube, as 3.1 hasn't implemented the necessary features yet.
>
>> The access.log shows content sizes so with a simple awk script it should
>> be easy to find out.
>>
>> I have also seen many sites where advertisements and trackers consume 15%
>> bandwidth. This may vary. So blocking ads and trackers is a thing to
>> consider.
>
> Thanks for this insight! This would of course be a welcome saving of bandwidth
> in my personal opinion. I'm just not sure if we're allowed to do this, as the
> patron of the proxy is a public-law institution and as such bound to anti-
> censorship laws. Need to check with a Legalese translator.
>
>>
>> Do not expect too much from web caching. More and more websites
>> intentionally make their sites not cacheable. Look at the percentage of
>> TCP_MISS in access.log or use a second awk script to find out more about
>> cacheability.
>
> Every bit counts. Before we apply for an increase of (expensive) uplink
> bandwidth we want to play every trick we have up our sleeve. At the moment our
> cache is still cold, because for getting the proxy running again I had to
> completely wipe the cache. At the moment we have a hit:miss ratio of about
> 1:5. For youtube caching we have a saved bandwidth around 100 GB for the 27th
> of march (one video in particular had a size of 768 MB and was watched 19
> times). Online lectures are currently en vogue.
>
>>
>> I recommend going for a newer Squid: 3.1.19 is stable and fixes issues that
>> 3.1.10 has.
>
> Will do so.
>
>> On Linux, aufs has a better performance than diskd
>
> Thanks again for this tip!
>
>>
>>> Additional memory for storing objects is 2048 MB:
>>>
>>> cache_mem 2048 MB
>>
>> Seems right. But you also need virtual memory for Squid being able to
>> fork processes without issues. Do have have 8 GB swap ?
>
> Yes. 10 GB actually.
>
>>
>> But read the FAQ about memory usage and a large disk cache:
>> http://wiki.squid-cache.org/SquidFaq/SquidMemory
>> Squid uses an additional 512*14 MB = 7.1 GB for the index of the disk
>> cache. I suggest to downsize to 1 GB in-memory index which implies to
>> use only 73 GB disk cache.
>
> Ah okay, here's one of my initial mistakes. I used only 10 MB for my
> calculation but of course we use a 64bit squid. Out of curiosity and because I
> want to learn: the reasoning for shrinking the disk cache from 512 GB to 73 GB
> is because a big cache as we have it at the moment only leads to lots of stale
> objects in cache which additionally burden the CPU and RAM because of in-
> memory metadata?
mostly because for a squid cache (not nginx) of 512GB will consume a lot
of memory on index while you will prefer to serve other stuff from
memory and because of the stale objects.
if you are using nginx for youtube caching remember to eliminate the
caching on youtube videos.
if you wont what will happen is that nginx will create new headers for
the cached objects and then squid will cache them but will never use
them again.
(if you didnt changed the basic not cache for objects contains "?" and
"cgi-bin" you are safe).
by the way if you have some time to analyze the proxy logs you can
manage to find some other sites you can use nginx to serve some stale files.
i have used it to cache also windows updates and some other video sites
but the pattern was similar so i had a huge list i could serv most of
the stale content from nginx what was leaving a lot of ram for squid index.

p.s.
forgot to say that you always must not get into a point that your proxy
ram is at the limit or else swapiness will happen and will slow the
server down.

Regards,
Eliezer
>
>
> Best regards,
> - Christian Loth
>

-- 
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il