Re: [squid-users] squid on 32-bit system with PAE and 8GB RAM

Gavin McCullagh wrote:
> Hi,
>
> thanks for the reply.
>
> On Tue, 17 Mar 2009, Amos Jeffries wrote:
>
>> FYI: The latest Intrepid or Jaunty package should work just as well in
>> Hardy.
>
> I'll look into this. I tried to build the intrepid debian package from
> source, but I came across a build dependency which was apparently not
> available on hardy: libgssglue-dev. I'll look into installing the
> pre-built package, but I would've thought it would need newer version of
> libraries.
>
> In general, I'm looking for simple maintenance and patching, but not at the
> expense of too much performance. Would we benefit much from a hand-built
> squid install? In what way?

I'm not aware of any major benefits. Just some minor gains from reduced
binary size when unused components are build-disabled. Someone reported
a major reduction when building with LVM recently. Thats about it.

>
>>> Of course, I forgot that the squid process can't address anything like
>>> that much RAM on a 32-bit system. I think the limit is about 3GB,
>>> right?
>> For 32-bit I think it is yes. You can rebuild squid as 64-bit or check
>> the distro for a 64-bit build.
>
> The server hardware isn't 64-bit so surely I can't run a 64-bit squid
> build, can I?

Ah, no I believe thats a problem. I kind of assumed that since your
system could take >2GB of RAM it was 64-bit enabled hardware.

>
>> However keep this in mind: rule-of-thumb is 10MB index per GB of cache.
>>
>> So your 600 GB disk cache is likely to use ~6GB of RAM for index +
>> whatever cache_mem you allocate for RAM-cache + index for RAM-cache + OS
>> and application memory.
>
> Ouch. That's not a rule of thumb I'd seen anywhere. I'm really not
> observing it either. Squid runs stabley for days with a 1.7GB cache_mem
> and a 600GB disk cache.
>
> It may help that we're allowing large objects into the cache and using
> "heap lfuda". We plot the average object size with munin and it's about
> 90KB. Presumably the 10MB per 1GB is strongly a function of average object
> size.
> http://deathcab.gcd.ie/munin/gcd.ie/watcher.gcd.ie.html#Squid

(sites restricted, but never mind)
Yes the rule-of-thumb was from past measures mediated by object size
(averages between 64KB and 128KB). I'm surprised you are seeing such a
low index size.

>
> The drops in RAM usage are all due to squid restarting. As long as I keep
> the cache_mem below about 1.8-2GB

Maybe the large-file changes in 2.7 will help then.

>
>>> I have two questions. Whenever I up the cache_mem beyond about 2GB, I
>>> notice squid terminates with signal 6 and restarts as the cache_mem fills.
>>> I presume this is squid hitting the 3GB-odd limit? Could squid not behave
>>> a little more politely in this situation -- either not attempting to
>>> allocate the extra RAM, giving a warning or an error?
>> cache.log should contain a FATAL: message and possibly a line or two
>> beforehand about why and where the crash occured.
>> Please can you post that info here.
>
> My apologies, there is a useful error, though in syslog not cache.log.
>
> Mar 15 22:50:24 watcher squid[6751]: httpReadReply: Excess data from "POST http://im.studivz.net/webx/re"
> Mar 15 22:52:50 watcher squid[6748]: Squid Parent: child process 6751 exited due to signal 6
> Mar 15 22:52:53 watcher squid[4206]: Starting Squid Cache version 2.6.STABLE18 for i386-debian-linux-gnu...
> Mar 15 22:52:53 watcher squid[4206]: Store logging disabled
> Mar 15 22:52:53 watcher squid[4206]: Rebuilding storage in /var/spool/squid/cache2 (DIRTY)
> Mar 15 22:54:29 watcher squid[4206]: 262144 Entries Validated so far.
> Mar 15 22:54:29 watcher squid[4206]: 524288 Entries Validated so far.
>
> I read this before and missed the "out of memory" error which appears in
> the syslog:
>
> Mar 15 22:52:50 watcher out of memory [6751]
>
> this seems to happen every time:
>
> Mar 10 11:58:12 watcher out of memory [22646]
> Mar 10 17:52:03 watcher out of memory [24620]
> Mar 11 00:57:52 watcher out of memory [31626]
>
>>> My main question is, is there a sensible way for me to use the extra RAM?
>>> I know the OS does disk caching with it but with a 600GB cache, I doubt
>>> that'll be much help.
>> RAM swapping (disk caching by the OS) is one major performance killer.
>> Squid needs direct access to all its memory for fast index searches and
>> in-transit processing.
>
> Of course. We definitely don't see any swapping to disk. I watch our
> munin memory graphs carefully for this. What I mean is that the linux OS
> does the opposite where RAM is unused -- it caches data in RAM, reads ahead
> open files, etc. but this probably won't help much where the amount of data
> on disk is very large.
>
> http://deathcab.gcd.ie/munin/gcd.ie/watcher.gcd.ie.html#System
>
>>> I thought of creating a 3-4GB ramdisk and using it
>>> as a volatile cache for squid which gets re-created (either by squid -z or
>>> by dd of an fs image) each time the machine reboots. The things is, I
>>> don't know how squid addresses multiple caches. If one cache is _much_
>>> faster but smaller than the other, can squid prioritise using it for the
>>> most regularly hit data or does it simply treat each cache as equal? Are
>>> there docs on these sorts of issues?
>> No need that is already built into Squid. cache_mem defines the amount
>> of RAM-cache Squid uses.
>
> Right, but if the squid process is hitting its 32-bit memory limit, I
> can't increase this any more, can I? This is why I'm suggesting a ramdisk
> cache as that won't expand squid's internal memory usage.
>
>> Squid allocates the disk space based on free space and attempts to
>> spread the load evenly over all dirs to minimize disk access/seek times.
>> cache_mem is used for the hottest objects to minimize delays even
>> further.
>
> I suppose one way to artificially "prioritise" the ramdisk caches might be
> to have n smaller ramdisk caches? Squid would then split the load as
> roughly 1/(n+1) between the disk cache and the n ramdisk caches?
>
> Sorry, some of you may be scratching your heads and wondering why one would
> do something so crazy. I've just got 4GB RAM sitting moreorless idle,
> a really busy disk and would like to use one to help the other :-)
>
> Gavin
>

Aha, in that case maybe. It would be an interesting setup anyhow.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE6 or 3.0.STABLE13
   Current Beta Squid 3.1.0.6