[squid-users] Squid 2.7STABLE7 randomly crashes

Greetings,

A Squid server (RHEL-5.2 64-bit) is serving around 1000 users with
NTLM (winbindd),
Basic, wbinfo, and URL rewriter (Websense). Squid has randomly crashed with
the following messages:

FATAL: Received Segment Violation...dying.
2010/01/20 15:53:29| storeDirWriteCleanLogs: Starting...
2010/01/20 15:53:29| WARNING: Closing open FD 183
2010/01/20 15:53:29| commSetEvents: epoll_ctl(EPOLL_CTL_DEL): failed
on fd=183: (1) Operation not permitted
2010/01/20 15:53:29| 65536 entries written so far.
2010/01/20 15:53:29| 131072 entries written so far.
...
...
2010/01/20 15:53:35| Finished. Wrote 10511684 entries.
2010/01/20 15:53:35| Took 6.2 seconds (1702503.8 entries/sec).
CPU Usage: 3406.086 seconds = 2431.073 user + 975.013 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 3
Memory usage for squid via mallinfo():
        total space in arena: -1092812 KB
        Ordinary blocks: -1096748 KB 3986 blks
        Small blocks: 0 KB 24 blks
        Holding blocks: 24108 KB 8 blks
        Free Small blocks: 1 KB
        Free Ordinary blocks: 3935 KB
        Total in use: -1072640 KB 100%
        Total free: 3936 KB 0%
2010/01/20 15:54:48| Starting Squid Cache version 2.7.STABLE7 for
x86_64-unknown-linux-gnu...

Ran gdb over the core dump file and got this result:

# gdb /usr/local/squid/sbin/squid core.30068
GNU gdb Red Hat Linux (6.5-37.el5rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db
library "/lib64/libthread_db.so.1".

Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libnsl.so.1...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/libnss_dns.so.2...done.
Loaded symbols for /lib64/libnss_dns.so.2
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Core was generated by `(squid)'.
Program terminated with signal 6, Aborted.
#0 0x000000376b630215 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x000000376b630215 in raise () from /lib64/libc.so.6
#1 0x000000376b631cc0 in abort () from /lib64/libc.so.6
#2 0x0000000000474515 in death (sig=<value optimized out>) at tools.c:327
#3 <signal handler called>
#4 0x000000000044271a in httpHeaderGetEntry (hdr=0x141868f8,
    pos=0x7fff2c609774) at HttpHeader.c:595
#5 0x00000000004427fe in httpHeaderFindEntry (hdr=0x141868f8,
    id=HDR_CONTENT_TYPE) at HttpHeader.c:619
#6 0x0000000000443037 in httpHeaderGetStr (hdr=0x141868f8,
    id=HDR_CONTENT_TYPE) at HttpHeader.c:1086
#7 0x00000000004075b9 in aclMatchAclList (list=0x2aaac246fa40,
    checklist=0x2aaad3524738) at acl.c:2010
#8 0x0000000000407d82 in aclCheck (checklist=0x2aaad3524738) at acl.c:2178
#9 0x000000000042daae in externalAclHandleReply (data=0x2aaab26015d0,
    reply=0x2aaac6951e38 "") at external_acl.c:985
#10 0x000000000043cd65 in helperHandleRead (fd=<value optimized out>,
    data=<value optimized out>) at helper.c:769
#11 0x0000000000427511 in comm_select (msec=<value optimized out>)
    at comm_generic.c:264
#12 0x000000000044f7a6 in main (argc=<value optimized out>,
    argv=0x7fff2c609a58) at main.c:863

The configure options read

./configure --prefix=/usr/local/squid --enable-async-io --enable-snmp --enab
le-epoll --enable-storeio=aufs null --enable-removal-policies=heap lru
--enable-
large-cache-files --with-large-files --disable-poll --disable-select
--disable-c
arp --disable-wccp --disable-ident-lookups --with-maxfd=32768
--enable-ntlm-auth
-helpers=SMB --enable-external-acl-helpers=wbinfo_group ldap_group
--enable-auth
=basic ntlm --enable-basic-auth-helpers=LDAP --enable-err-languages=English

Snippet of squid.conf

emulate_httpd_log on
buffered_logs on
debug_options ALL,1

logfile_daemon /usr/local/squid/libexec/logfile-daemon
access_log daemon:/var/log/squid/access.log

shutdown_lifetime 10 seconds

cache_dir aufs /var/spool/squid/cache01 62500 32 256
cache_dir aufs /var/spool/squid/cache02 62500 32 256
cache_dir aufs /var/spool/squid/cache03 62500 32 256
cache_dir aufs /var/spool/squid/cache04 62500 32 256

cache_replacement_policy heap GDSF
memory_replacement_policy heap GDSF

cache_log /var/log/squid/cache.log
cache_store_log none
pid_filename /var/run/squid.pid
coredump_dir /var/spool/squid/crash
log_icp_queries off
client_db on
half_closed_clients off

cache_mem 512 MB
maximum_object_size 768000 KB
maximum_object_size_in_memory 96 KB
memory_pools off

forwarded_for off

snmp_port 1601
snmp_incoming_address 0.0.0.0
snmp_outgoing_address 255.255.255.255

auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
auth_param ntlm children 30
auth_param ntlm keep_alive on

auth_param basic program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-basic
auth_param basic children 5
auth_param basic realm User Authentication

external_acl_type ads-group children=20 %LOGIN
/usr/local/squid/libexec/wbinfo_group.pl

url_rewrite_children 50
redirector_bypass off
url_rewrite_program /opt/Websense/bin/WsRedtor

Does anyone have any idea how to fix this problem?

Many Thanks

myOcella
Received on Wed Jan 20 2010 - 06:18:01 MST