[squid-users] xmalloc failure, yet FAQ items check out ok

Hello,

I continue to have a problem where the squid process crashes but
recovers on it's own. The error is FAQ 8.7's "xmalloc unable to
allocate..." error. It started back when system only had 300MB of
swap. So I recreated a new filesystem swap of 1.5GB. At that time I
also added an extra cache_dir (on separate disk). The system is a Sun
Ultra 60 running Solaris 8 with 1GB RAM. Squid is 2.5STABLE2.

The effect of the changes I made (larger swap and more cache_dirs) was
that the squid process is staying less than 300MB when before (with
one cache_dir and a small swap) the squid process got as large as
600+MB. I am assuming that this is because now it can write that
extra memory stuff out to swap, but if that is not the case please
enlighten me.

Anyhow, once swap is eliminated as a problem, then FAQ 8.7 says it
must be a data segment size problem. However "ulimit -aHd" shows
unlimited and I checked with Sun and they said there is no per process
limit in the kernel, process limits are only a shell thing. It's not
like filedescriptors which can be bumped up in the kernel. For the
kernel, the data segment size is (essentially) unlimited and is a
shell thing, not a kernel thing. Furthermore, prior to my changes,
squid's process size was regularly larger than 600MB and I haven't
made any kernel type changes.

So have I mistweaked something in the squid.conf file? I haven't
changed the system (e.g. in the /etc/system file), just added more
filesystem swap. Below are my squid compile options, the cache.log
error, an extract of my squid.conf file, and before/after's of top and
ps.

Side question: for Sun Solaris 8, is dlmalloc the recommended malloc
to compile against? I've read that I should avoid Sun's own malloc
(though that may only have applied to x86) but I am wondering if the
problem is the dlmalloc library interacting with Solaris...

thanks in advance for any corrections or advice.

thanks,

Adam

Squid Cache: Version 2.5.STABLE2
configure options: --enable-dlmalloc --enable-async-io
--enable-storeio=aufs,diskd,ufs --enable-removal-policies=heap,lru
--enable-delay-pools --disable-icmp
--enable-cachemgr-hostname=aocproxy --enable-snmp
--disable-ident-lookups --with-pthreads

From the cache.log file:
  [ usual lines like comm_accept and http_accept deleted ]
2003/06/05 10:47:25| httpAccept: FD 16: accept failure: (130) Software
caused connection abort
FATAL: xmalloc: Unable to allocate 24576 bytes!

Squid Cache (Version 2.5.STABLE2): Terminated abnormally.
 CPU Usage: 2627.620 seconds = 1421.140 user + 1206.480 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 57391
Memory usage for squid via mallinfo():
        total space in arena: 218032 KB
        Ordinary blocks: 214550 KB 17911 blks
        Small blocks: 0 KB 0 blks
        Holding blocks: 7128 KB 7 blks
        Free Small blocks: 0 KB
        Free Ordinary blocks: 3482 KB
        Total in use: 221678 KB 102%
        Total free: 3482 KB 2%
2003/06/05 10:47:39| storeDirWriteCleanLogs: Starting...
 [ snip ]

Main items from squid.conf:
cache_mem 64 MB
cache_swap_low 95
cache_swap_high 95
maximum_object_size_in_memory 16 KB
cache_replacement_policy heap GDSF
memory_replacement_policy heap GDSF
cache_dir aufs /cache 7256 16 256
cache_dir aufs /cache3 6800 16 256

TOP and PS before crash:
top -b:
Memory: 1024M real, 541M free, 237M swap in use, 1994M swap free
  PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
29604 root 36 43 0 216M 213M sleep 43:06 10.35% squid
29600 root 3 18 0 4216K 1464K sleep 0:00 0.00% squid
29605 squid 4 28 0 1256K 912K sleep 0:00 0.00% unlinkd
          
/usr/ucb/ps -auxwww:
USER PID %CPU %MEM SZ RSS TT S START TIME COMMAND
squid 29604 10.5 21.8221040218360 ? O 21:15:25 43:06 (squid)
-F -f /usr/local/squid/etc/squid.conf -s
root 29600 0.0 0.2 4216 1464 ? S 21:15:25 0:00
/usr/local/squid/sbin/squid -F -f /usr/local/squid/etc/squid.conf -s
squid 29605 0.0 0.1 1256 912 ? S 21:15:25 0:00 (unlinkd)
          
TOP and PS after crash:
Memory: 1024M real, 640M free, 156M swap in use, 2076M swap free
  PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
 3268 root 36 32 0 135M 132M sleep 1:13 15.43% squid
29600 root 4 38 0 4224K 1472K sleep 0:00 0.00% squid
 3269 squid 4 38 0 1256K 912K sleep 0:00 0.00% unlinkd
          
/usr/ucb/ps -auxwww:
USER PID %CPU %MEM SZ RSS TT S START TIME COMMAND
squid 3268 15.4 13.5138616135616 ? R 10:47:42 1:13 (squid)
-F -f /usr/local/squid/etc/squid.conf -s
squid 3269 0.0 0.1 1256 912 ? S 10:47:43 0:00 (unlinkd)
root 29600 0.0 0.2 4224 1472 ? S 21:15:25 0:00
/usr/local/squid/sbin/squid -F -f /usr/local/squid/etc/squid.conf -s

from sysdef -i:
* Tunable Parameters
*
20815872 maximum memory allowed in buffer cache (bufhwm)
   15882 maximum number of processes (v.v_proc)
      99 maximum global priority in sys class (MAXCLSYSPRI)
   15877 maximum processes per user id (v.v_maxup)
      30 auto update time limit in seconds (NAUTOUP)
      25 page stealing low water mark (GPGSLO)
       5 fsflush run rate (FSFLUSHR)
      25 minimum resident memory for avoiding deadlock
(MINARMEM)
      25 minimum swapable memory for avoiding deadlock
(MINASMEM)
*
* Process Resource Limit Tunables (Current:Maximum)
*
          Infinity:Infinity cpu time
          Infinity:Infinity file size
          Infinity:Infinity heap size
0x0000000000800000:Infinity stack size
          Infinity:Infinity core file size
0x0000000000000800:0x0000000000002000 file descriptors
          Infinity:Infinity mapped memory
Received on Mon Jun 09 2003 - 21:01:12 MDT