Re: [squid-users] Squid 2.7STABLE7 randomly crashes

Can you paste in your acl-related configuration? That's where the crash is...

On 27/01/2010, at 10:50 AM, myocella wrote:

> Anyone?
>
> On Wed, Jan 20, 2010 at 4:17 PM, myocella <myocella_at_gmail.com> wrote:
>> Greetings,
>>
>> A Squid server (RHEL-5.2 64-bit) is serving around 1000 users with
>> NTLM (winbindd),
>> Basic, wbinfo, and URL rewriter (Websense). Squid has randomly crashed with
>> the following messages:
>>
>> FATAL: Received Segment Violation...dying.
>> 2010/01/20 15:53:29| storeDirWriteCleanLogs: Starting...
>> 2010/01/20 15:53:29| WARNING: Closing open FD 183
>> 2010/01/20 15:53:29| commSetEvents: epoll_ctl(EPOLL_CTL_DEL): failed
>> on fd=183: (1) Operation not permitted
>> 2010/01/20 15:53:29| 65536 entries written so far.
>> 2010/01/20 15:53:29| 131072 entries written so far.
>> ...
>> ...
>> 2010/01/20 15:53:35| Finished. Wrote 10511684 entries.
>> 2010/01/20 15:53:35| Took 6.2 seconds (1702503.8 entries/sec).
>> CPU Usage: 3406.086 seconds = 2431.073 user + 975.013 sys
>> Maximum Resident Size: 0 KB
>> Page faults with physical i/o: 3
>> Memory usage for squid via mallinfo():
>> total space in arena: -1092812 KB
>> Ordinary blocks: -1096748 KB 3986 blks
>> Small blocks: 0 KB 24 blks
>> Holding blocks: 24108 KB 8 blks
>> Free Small blocks: 1 KB
>> Free Ordinary blocks: 3935 KB
>> Total in use: -1072640 KB 100%
>> Total free: 3936 KB 0%
>> 2010/01/20 15:54:48| Starting Squid Cache version 2.7.STABLE7 for
>> x86_64-unknown-linux-gnu...
>>
>>
>>
>> Ran gdb over the core dump file and got this result:
>>
>> # gdb /usr/local/squid/sbin/squid core.30068
>> GNU gdb Red Hat Linux (6.5-37.el5rh)
>> Copyright (C) 2006 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and you are
>> welcome to change it and/or distribute copies of it under certain conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB. Type "show warranty" for details.
>> This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db
>> library "/lib64/libthread_db.so.1".
>>
>> Reading symbols from /lib64/libcrypt.so.1...done.
>> Loaded symbols for /lib64/libcrypt.so.1
>> Reading symbols from /lib64/libpthread.so.0...done.
>> Loaded symbols for /lib64/libpthread.so.0
>> Reading symbols from /lib64/libm.so.6...done.
>> Loaded symbols for /lib64/libm.so.6
>> Reading symbols from /lib64/libnsl.so.1...done.
>> Loaded symbols for /lib64/libnsl.so.1
>> Reading symbols from /lib64/libc.so.6...done.
>> Loaded symbols for /lib64/libc.so.6
>> Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
>> Loaded symbols for /lib64/ld-linux-x86-64.so.2
>> Reading symbols from /lib64/libnss_files.so.2...done.
>> Loaded symbols for /lib64/libnss_files.so.2
>> Reading symbols from /lib64/libnss_dns.so.2...done.
>> Loaded symbols for /lib64/libnss_dns.so.2
>> Reading symbols from /lib64/libresolv.so.2...done.
>> Loaded symbols for /lib64/libresolv.so.2
>> Core was generated by `(squid)'.
>> Program terminated with signal 6, Aborted.
>> #0 0x000000376b630215 in raise () from /lib64/libc.so.6
>> (gdb) bt
>> #0 0x000000376b630215 in raise () from /lib64/libc.so.6
>> #1 0x000000376b631cc0 in abort () from /lib64/libc.so.6
>> #2 0x0000000000474515 in death (sig=<value optimized out>) at tools.c:327
>> #3 <signal handler called>
>> #4 0x000000000044271a in httpHeaderGetEntry (hdr=0x141868f8,
>> pos=0x7fff2c609774) at HttpHeader.c:595
>> #5 0x00000000004427fe in httpHeaderFindEntry (hdr=0x141868f8,
>> id=HDR_CONTENT_TYPE) at HttpHeader.c:619
>> #6 0x0000000000443037 in httpHeaderGetStr (hdr=0x141868f8,
>> id=HDR_CONTENT_TYPE) at HttpHeader.c:1086
>> #7 0x00000000004075b9 in aclMatchAclList (list=0x2aaac246fa40,
>> checklist=0x2aaad3524738) at acl.c:2010
>> #8 0x0000000000407d82 in aclCheck (checklist=0x2aaad3524738) at acl.c:2178
>> #9 0x000000000042daae in externalAclHandleReply (data=0x2aaab26015d0,
>> reply=0x2aaac6951e38 "") at external_acl.c:985
>> #10 0x000000000043cd65 in helperHandleRead (fd=<value optimized out>,
>> data=<value optimized out>) at helper.c:769
>> #11 0x0000000000427511 in comm_select (msec=<value optimized out>)
>> at comm_generic.c:264
>> #12 0x000000000044f7a6 in main (argc=<value optimized out>,
>> argv=0x7fff2c609a58) at main.c:863
>>
>> The configure options read
>>
>> ./configure --prefix=/usr/local/squid --enable-async-io --enable-snmp --enab
>> le-epoll --enable-storeio=aufs null --enable-removal-policies=heap lru
>> --enable-
>> large-cache-files --with-large-files --disable-poll --disable-select
>> --disable-c
>> arp --disable-wccp --disable-ident-lookups --with-maxfd=32768
>> --enable-ntlm-auth
>> -helpers=SMB --enable-external-acl-helpers=wbinfo_group ldap_group
>> --enable-auth
>> =basic ntlm --enable-basic-auth-helpers=LDAP --enable-err-languages=English
>>
>>
>> Snippet of squid.conf
>>
>> emulate_httpd_log on
>> buffered_logs on
>> debug_options ALL,1
>>
>> logfile_daemon /usr/local/squid/libexec/logfile-daemon
>> access_log daemon:/var/log/squid/access.log
>>
>> shutdown_lifetime 10 seconds
>>
>> cache_dir aufs /var/spool/squid/cache01 62500 32 256
>> cache_dir aufs /var/spool/squid/cache02 62500 32 256
>> cache_dir aufs /var/spool/squid/cache03 62500 32 256
>> cache_dir aufs /var/spool/squid/cache04 62500 32 256
>>
>> cache_replacement_policy heap GDSF
>> memory_replacement_policy heap GDSF
>>
>> cache_log /var/log/squid/cache.log
>> cache_store_log none
>> pid_filename /var/run/squid.pid
>> coredump_dir /var/spool/squid/crash
>> log_icp_queries off
>> client_db on
>> half_closed_clients off
>>
>> cache_mem 512 MB
>> maximum_object_size 768000 KB
>> maximum_object_size_in_memory 96 KB
>> memory_pools off
>>
>> forwarded_for off
>>
>> snmp_port 1601
>> snmp_incoming_address 0.0.0.0
>> snmp_outgoing_address 255.255.255.255
>>
>> auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
>> auth_param ntlm children 30
>> auth_param ntlm keep_alive on
>>
>> auth_param basic program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-basic
>> auth_param basic children 5
>> auth_param basic realm User Authentication
>>
>> external_acl_type ads-group children=20 %LOGIN
>> /usr/local/squid/libexec/wbinfo_group.pl
>>
>> url_rewrite_children 50
>> redirector_bypass off
>> url_rewrite_program /opt/Websense/bin/WsRedtor
>>
>>
>> Does anyone have any idea how to fix this problem?
>>
>> Many Thanks
>>
>> myOcella
>>

--
Mark Nottingham       mnot_at_yahoo-inc.com