RE: [squid-users] squid on redhat6 crash

More information on cache.log

2012/08/20 14:46:35| WARNING: swapfile header inconsistent with available data
FATAL: Received Segment Violation...dying.
2012/08/20 14:46:35| storeDirWriteCleanLogs: Starting...
2012/08/20 14:46:35| WARNING: Closing open FD 36
2012/08/20 14:46:35| 65536 entries written so far.
2012/08/20 14:46:35| 131072 entries written so far.
2012/08/20 14:46:35| 196608 entries written so far.

-----Original Message-----
From: Julie Xu [mailto:J.Xu_at_uws.edu.au]
Sent: Tuesday, 21 August 2012 7:49 AM
To: Eliezer Croitoru; squid-users_at_squid-cache.org
Subject: RE: [squid-users] squid on redhat6 crash

Eliezer

Great thanks for the reply. From briefly dump information I only get is:
Crash comman: is squid -f /etc/squid/squid.conf
Reason: is Process /usr/sbin/squid was killed by signal 6 (SIGABRT)
Kernel: is 2.6.32-279.2.1.el6.x86_64 (Red Hat Enterprise Linux Server release 6.3 (Santiago))
Pwd: /var/cache/squid

Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 10485760 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 256718 256718 processes
Max open files 1024 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 256718 256718 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

Below to answer your question:

>you mean load by req\s?memory?cpu?disk i\o?

The load I mean is when squid client increase; I found it only crash between 9am-9pm, did not crash at night time.

>how many requests per sec?

Now, I can not check, it is out of service now.

>squid 3.1 self compiled or from rpm?

It is from system rpm.

>using any special helpers?how many child process(for helpers)?

Squid_ldap_auth, 10 child process, do not anything else.

>"squid -v" output?
]# squid -v
Squid Cache: Version 3.1.10
configure options: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr' '--libexecdir=/usr/lib64/squid' '--localstatedir=/var' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--with-logdir=$(localstatedir)/log/squid' '--with-pidfile=$(localstatedir)/run/squid.pid' '--disable-dependency-tracking' '--enable-arp-acl' '--enable-follow-x-forwarded-for' '--enable-auth=basic,digest,ntlm,negotiate' '--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SMB,YP,getpwnam,multi-domain-NTLM,SASL,DB,POP3,squid_radius_auth' '--enable-ntlm-auth-helpers=smb_lm,no_check,fakeauth' '--enable-digest-auth-helpers=password,ldap,eDirectory' '--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-external-acl-helpers=ip_user,ldap_group,session,unix_group,wbinfo_group' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-epoll' '--enable-icap-client' '--enable-ident-lookups' '--enable-linux-netfilter' '--enable-referer-log' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl' '--enable-storeio=aufs,diskd,ufs' '--enable-useragent-log' '--enable-wccpv2' '--enable-esi' '--with-aio' '--with-default-user=squid' '--with-filedescriptors=16384' '--with-dl' '--with-openssl' '--with-pthreads' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fpie' 'LDFLAGS=-pie' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fpie' --with-squid=/builddir/build/BUILD/squid-3.1.10

>squid.conf ?
Related part:
auth_param basic program /usr/lib64/squid/squid_ldap_auth ........
auth_param basic children 10
auth_param basic realm xx
auth_param basic credentialsttl 15 minutes authenticate_ip_ttl 5 minutes acl all src all icp_access deny all acl_uses_indirect_client off log_uses_indirect_client off http_port xxx hierarchy_stoplist cgi-bin ? squid-internal- cache_mem 2048 MB maximum_object_size_in_memory 1 MB memory_replacement_policy heap GDSF cache_replacement_policy heap LFUDA cache_dir aufs xxxx maximum_object_size 64 MB cache_store_log none dns_retransmit_interval 1 second dns_timeout 1 minute ipcache_size 8192 ipcache_low 95 ipcache_high 98 memory_pools_limit none forwarded_for off uri_whitespace deny coredump_dir /var/cache/squid pipeline_prefetch on sleep_after_fork 500000

>transparent proxy?
no

>"ulimit -Sa" output?
ore file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 256718
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

>"ulimit -Ha" output?
# ulimit -Ha
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 256718
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

many regards

Julie

-----Original Message-----
From: Eliezer Croitoru [mailto:eliezer_at_ngtech.co.il]
Sent: Tuesday, 21 August 2012 12:33 AM
To: squid-users_at_squid-cache.org
Subject: Re: [squid-users] squid on redhat6 crash

On 8/20/2012 4:01 PM, Julie Xu wrote:
> Hi
>
> Used redhat6' squid 3.1, and the system is crashing when there is load increased.
>
you mean load by req\s?memory?cpu?disk i\o?
how many requests per sec?

> Aug 20 14:30:01 proxy-server squid[2450]: Squid Parent: child process
> 6591 exited due to signal 6 with status 0 Aug 20 14:30:04 proxy-server
> squid[2450]: Squid Parent: child process 7091 started Aug 20 14:46:40
> proxy-server abrt[22338]: Saved core dump of pid 7091
> (/usr/sbin/squid) to /var/spool/abrt/ccpp-2012-08-20-14:46:36-7091
> (668131328 bytes)
>
> Briefly look the dump directory, only thing I found is the direct is the cache direct, there is a coredump file, try to use crash to read it, not successful.
>
> Could anyone advise me what is possible reason for the crash, and the solution?
squid 3.1 self compiled or from rpm?
using any special helpers?how many child process(for helpers)?
"squid -v" output?
squid.conf ?
transparent proxy?
"ulimit -Sa" output?
"ulimit -Ha" output?

the more data you will provide us we can pinpoint the reason for the problem.

Regards,
Eliezer
>
> Thanks in advance
>
> Julie
>

--
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations eliezer <at> ngtech.co.il