Anyone?
On Wed, Jan 20, 2010 at 4:17 PM, myocella <myocella_at_gmail.com> wrote:
> Greetings,
>
> A Squid server (RHEL-5.2 64-bit) is serving around 1000 users with
> NTLM (winbindd),
> Basic, wbinfo, and URL rewriter (Websense). Squid has randomly crashed with
> the following messages:
>
> FATAL: Received Segment Violation...dying.
> 2010/01/20 15:53:29| storeDirWriteCleanLogs: Starting...
> 2010/01/20 15:53:29| WARNING: Closing open FD 183
> 2010/01/20 15:53:29| commSetEvents: epoll_ctl(EPOLL_CTL_DEL): failed
> on fd=183: (1) Operation not permitted
> 2010/01/20 15:53:29| 65536 entries written so far.
> 2010/01/20 15:53:29| 131072 entries written so far.
> ...
> ...
> 2010/01/20 15:53:35| Finished. Wrote 10511684 entries.
> 2010/01/20 15:53:35| Took 6.2 seconds (1702503.8 entries/sec).
> CPU Usage: 3406.086 seconds = 2431.073 user + 975.013 sys
> Maximum Resident Size: 0 KB
> Page faults with physical i/o: 3
> Memory usage for squid via mallinfo():
> total space in arena: -1092812 KB
> Ordinary blocks: -1096748 KB 3986 blks
> Small blocks: 0 KB 24 blks
> Holding blocks: 24108 KB 8 blks
> Free Small blocks: 1 KB
> Free Ordinary blocks: 3935 KB
> Total in use: -1072640 KB 100%
> Total free: 3936 KB 0%
> 2010/01/20 15:54:48| Starting Squid Cache version 2.7.STABLE7 for
> x86_64-unknown-linux-gnu...
>
>
>
> Ran gdb over the core dump file and got this result:
>
> # gdb /usr/local/squid/sbin/squid core.30068
> GNU gdb Red Hat Linux (6.5-37.el5rh)
> Copyright (C) 2006 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db
> library "/lib64/libthread_db.so.1".
>
> Reading symbols from /lib64/libcrypt.so.1...done.
> Loaded symbols for /lib64/libcrypt.so.1
> Reading symbols from /lib64/libpthread.so.0...done.
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /lib64/libm.so.6...done.
> Loaded symbols for /lib64/libm.so.6
> Reading symbols from /lib64/libnsl.so.1...done.
> Loaded symbols for /lib64/libnsl.so.1
> Reading symbols from /lib64/libc.so.6...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib64/libnss_files.so.2...done.
> Loaded symbols for /lib64/libnss_files.so.2
> Reading symbols from /lib64/libnss_dns.so.2...done.
> Loaded symbols for /lib64/libnss_dns.so.2
> Reading symbols from /lib64/libresolv.so.2...done.
> Loaded symbols for /lib64/libresolv.so.2
> Core was generated by `(squid)'.
> Program terminated with signal 6, Aborted.
> #0 0x000000376b630215 in raise () from /lib64/libc.so.6
> (gdb) bt
> #0 0x000000376b630215 in raise () from /lib64/libc.so.6
> #1 0x000000376b631cc0 in abort () from /lib64/libc.so.6
> #2 0x0000000000474515 in death (sig=<value optimized out>) at tools.c:327
> #3 <signal handler called>
> #4 0x000000000044271a in httpHeaderGetEntry (hdr=0x141868f8,
> pos=0x7fff2c609774) at HttpHeader.c:595
> #5 0x00000000004427fe in httpHeaderFindEntry (hdr=0x141868f8,
> id=HDR_CONTENT_TYPE) at HttpHeader.c:619
> #6 0x0000000000443037 in httpHeaderGetStr (hdr=0x141868f8,
> id=HDR_CONTENT_TYPE) at HttpHeader.c:1086
> #7 0x00000000004075b9 in aclMatchAclList (list=0x2aaac246fa40,
> checklist=0x2aaad3524738) at acl.c:2010
> #8 0x0000000000407d82 in aclCheck (checklist=0x2aaad3524738) at acl.c:2178
> #9 0x000000000042daae in externalAclHandleReply (data=0x2aaab26015d0,
> reply=0x2aaac6951e38 "") at external_acl.c:985
> #10 0x000000000043cd65 in helperHandleRead (fd=<value optimized out>,
> data=<value optimized out>) at helper.c:769
> #11 0x0000000000427511 in comm_select (msec=<value optimized out>)
> at comm_generic.c:264
> #12 0x000000000044f7a6 in main (argc=<value optimized out>,
> argv=0x7fff2c609a58) at main.c:863
>
> The configure options read
>
> ./configure --prefix=/usr/local/squid --enable-async-io --enable-snmp --enab
> le-epoll --enable-storeio=aufs null --enable-removal-policies=heap lru
> --enable-
> large-cache-files --with-large-files --disable-poll --disable-select
> --disable-c
> arp --disable-wccp --disable-ident-lookups --with-maxfd=32768
> --enable-ntlm-auth
> -helpers=SMB --enable-external-acl-helpers=wbinfo_group ldap_group
> --enable-auth
> =basic ntlm --enable-basic-auth-helpers=LDAP --enable-err-languages=English
>
>
> Snippet of squid.conf
>
> emulate_httpd_log on
> buffered_logs on
> debug_options ALL,1
>
> logfile_daemon /usr/local/squid/libexec/logfile-daemon
> access_log daemon:/var/log/squid/access.log
>
> shutdown_lifetime 10 seconds
>
> cache_dir aufs /var/spool/squid/cache01 62500 32 256
> cache_dir aufs /var/spool/squid/cache02 62500 32 256
> cache_dir aufs /var/spool/squid/cache03 62500 32 256
> cache_dir aufs /var/spool/squid/cache04 62500 32 256
>
> cache_replacement_policy heap GDSF
> memory_replacement_policy heap GDSF
>
> cache_log /var/log/squid/cache.log
> cache_store_log none
> pid_filename /var/run/squid.pid
> coredump_dir /var/spool/squid/crash
> log_icp_queries off
> client_db on
> half_closed_clients off
>
> cache_mem 512 MB
> maximum_object_size 768000 KB
> maximum_object_size_in_memory 96 KB
> memory_pools off
>
> forwarded_for off
>
> snmp_port 1601
> snmp_incoming_address 0.0.0.0
> snmp_outgoing_address 255.255.255.255
>
> auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
> auth_param ntlm children 30
> auth_param ntlm keep_alive on
>
> auth_param basic program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-basic
> auth_param basic children 5
> auth_param basic realm User Authentication
>
> external_acl_type ads-group children=20 %LOGIN
> /usr/local/squid/libexec/wbinfo_group.pl
>
> url_rewrite_children 50
> redirector_bypass off
> url_rewrite_program /opt/Websense/bin/WsRedtor
>
>
> Does anyone have any idea how to fix this problem?
>
> Many Thanks
>
> myOcella
>
Received on Tue Jan 26 2010 - 23:50:40 MST
This archive was generated by hypermail 2.2.0 : Wed Jan 27 2010 - 12:00:05 MST