Re: large proxies

In article <cistron.200002211134.MAA10631@xs4.xs4all.nl>,
Cor Bosman <cor@xs4all.net> wrote:
>Hi all, we've been running a few squid caches for a few years. Ive been
>looking at new proxies (current ones are nearing end of life).
>Squid is currently the prefered option (although Netcache is also in the
>picture).
>
>Is anyone here running a large cache on squid? Say 100GB in disk capacity.
>If so, what kind of system is it? It seems you need like 2gb memory for such
>large caches.

Hello Cor ;)

holodeck% df
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/sda1 189387 15215 164394 8% /
/dev/sda5 1922044 441008 1383400 24% /usr
/dev/sda6 1922044 1070888 753520 59% /var
/dev/sda7 4618160 2930308 1453256 67% /extra
/dev/sdb1 17656612 7922180 8837512 47% /cache/disk1
/dev/sdc1 17656612 7936552 8823140 47% /cache/disk2
/dev/sdd1 17656612 7896088 8863604 47% /cache/disk3
/dev/sde1 17656612 7898724 8860968 47% /cache/disk4

About 70 GB of disk. This is a dual Pentium III @ 450 Mhz,
with 512 MB or RAM. Running squid-2.2STABLE5 on a Linux 2.2.15pre5
kernel. The dual CPU is only useful if you have an OS with kernel-
level threading and compile squid with async-io, otherwise squid
would only use one CPU.

The cache is not extremely busy though - about 10-20 connections/sec

Load is reasonable. # uptime
 11:38pm up 17 days, 22:06, 1 user, load average: 0.22, 0.26, 0.19

As for how much memory you need - well, the average size of an object
in the cache is rumoured to be 13 Kbytes. So, with 70 GB of disk
you can store 5384615 objects. Each object needs 72 bytes in-memory.
So we need 369 MB of memory. The 512 MB we have suffices. For
a 100 GB cache 512 MB would be on the edge, 768 MB would do nicely.

>Is there any research available as to if a 100gb cache is even worth it?
>Should we remain with something smaller like 25gb (currectly).
>We serve several tens of thousands of users.

Have a look for yourself:

http://stats.cistron.nl/cachemgr.html

For example:

Cache information for squid:
        Request Hit Ratios: 5min: 33.4%, 60min: 39.0%
        Byte Hit Ratios: 5min: 13.4%, 60min: 18.0%
        Storage Swap size: 26643777 KB
        Storage Mem size: 10072 KB
        Storage LRU Expiration Age: 365.00 days
        Mean Object Size: 11.03 KB

You don't really save a lot of bandwidth with a cache anymore, a lot
of content is dynamic nowadays. But it does help for slow far-away
sites that you visit reasonably often.

Mike.

-- 
"quote me on this one - overgaan op Exchange is een wijs besluit" -- MarcoH.
-- 
The From: and Reply-To: addresses are internal news2mail gateway addresses.
Reply to the list or to miquels@cistron.nl (Miquel van Smoorenburg)