Re: [squid-users] squid on 32-bit system with PAE and 8GB RAM

Hi,

I don't mean to labour this, I'm just keen to understand better and
obviously you guys are the experts on squid.

On Mon, 16 Mar 2009, Marcello Romani wrote:

>> Really? I would have thought the linux kernel's disk caching would be far
>> less optimised for this than using a large squid cache_mem (whatever about
>> a ramdisk).
>
> As others have pointed out, squid's cache_mem is not used to serve
> on-disk cache objects, while os's disk cache will hold those objects in
> RAM after squid requests them for the first time.

Agreed. I would have thought though that a large cache_mem would be a
better way to increase the data served from RAM, compared to the OS disk
caching.

I imagine, perhaps incorrectly, that squid uses the mem_cache first for
data, then when it's removed (by LRU or whatever), pushes it out to the
disk cache. This sounds like it should lead to a pretty good
mem_cache:disk_cache serving ratio. I don't have much to back this up, but
the ratio in my own case is pretty high so squid appears not to just treat
all caches (memory and disk) equally.

http://deathcab.gcd.ie/munin/gcd.ie/watcher.gcd.ie-squid_cache_hit_breakdown.html

By comparison, I would expect linux's disk caching, which has no
understanding of the fact that this is a web proxy cache, to be less smart.
Perhaps that's incorrect though, I'm not sure what mechanism linux uses.

> So if you leave most of RAM to OS for disk cache you'll end up having
> many on-disk object loaded from RAM, i.e. very quickly.

Some, but I would imagine not as many as with mem_cache.

> Also, squid needs memory besides cache_mem, for its own internal
> structures and for managing the on-disk repository. If its address space
> is already almost filled up by cache_mem alone, it might have problems
> allocating its own memory structures.

Absolutely agreed and the crashes I've seen appear to be caused by this,
though dropping to around 1.7GB mem_cache appears to cure this.

The question then is, which would be better, an extra cache based on a
ramdisk, or just leaving it up to the kernel's disk caching.

> OS's disk cache, on the other hand, is not allcated from squid's process
> memory space and has also a variable size, automatically adjusted by the
> OS when app memory needs grow or shrink.

Right. A ramdisk is also not allocated from squid's process space either,
but it doesn't shrink in the way linux disk caching would and that might
cause swapping in a bad situation. That's a clear advantage for linux's
caching. Simplicity is another clear advantage.

The question I'm left with is, which of the two would better optimise the
amount of data served from ram (thus lowering iowait), linux's caching or
the ramdisk?

I guess it's not a very normal setup, so maybe nobody has done this.

Thanks for all the feedback,
Gavin
Received on Tue Mar 17 2009 - 21:38:04 MDT