[squid-users] assertion failed: errorpage.cc

Hello Squid Users --

I'm very new to running squid. I've quickly learned that there are many parameters to configure and therefore many places problems can occur.

I am prototyping a squid environment on a CentOS 6.4 32-bit system with 256MB RAM. Presently, I'm observing strange behavior with the proxy, and the most notable symptom is that either the proxy server becomes unavailable after a period of time or DNS names stop resolving. I should mention that I'm also sending the traffic over a Poptop (pptpd) VPN tunnel, but these issues persist even without the tunnel up as far as I can tell.

Info from the environment:

# squid -v
Squid Cache: Version 3.1.10
configure options: '--build=i386-redhat-linux-gnu' '--host=i386-redhat-linux-gnu' '--target=i686-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-internal-dns' '--disable-strict-error-checking' '--exec_prefix=/usr' '--libexecdir=/usr/lib/squid' '--localstatedir=/var' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--with-logdir=$(localstatedir)/log/squid' '--with-pidfile=$(localstatedir)/run/squid.pid' '--disable-dependency-tracking' '--enable-arp-acl' '--enable-follow-x-forwarded-for' '--enable-auth=basic,digest,ntlm,negotiate' '--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SMB,YP,getpwnam,multi-domain-NTLM,SASL,DB,POP3,squid_radius_auth' '--enable-ntlm-auth-helpers=smb_lm,no_check,fakeauth' '--enable-digest-auth-helpers=password,ldap,eDirectory' '--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-external-acl-helpers=ip_user,ldap_group,session,unix_group,wbinfo_group' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-epoll' '--enable-icap-client' '--enable-ident-lookups' '--with-large-files' '--enable-linux-netfilter' '--enable-referer-log' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl' '--enable-storeio=aufs,diskd,ufs' '--enable-useragent-log' '--enable-wccpv2' '--enable-esi' '--with-aio' '--with-default-user=squid' '--with-filedescriptors=16384' '--with-dl' '--with-openssl' '--with-pthreads' 'build_alias=i386-redhat-linux-gnu' 'host_alias=i386-redhat-linux-gnu' 'target_alias=i686-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i686 -mtune=atom -fasynchronous-unwind-tables -fpie' 'LDFLAGS=-pie' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i686 -mtune=atom -fasynchronous-unwind-tables -fpie' --with-squid=/builddir/build/BUILD/squid-3.1.10

squid.conf:

http_port 3128 intercept
http_port 4334
http_port 127.0.0.1:33699
http_port 127.0.0.1:32623
http_port 127.0.0.1:26226
http_port 127.0.0.1:26499
http_port 127.0.0.1:18108
http_port 127.0.0.1:49236
http_port MY.PUB.IP.ADR:3128
icp_port 0

refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
acl apache rep_header Server ^Apache

cache_mem 128 MB
maximum_object_size 1024 KB
maximum_object_size_in_memory 16 KB
cache_dir aufs /opt/squid/cache 16384 64 256

error_directory /opt/squid/logs
allow_underscore off
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 4320

acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl Safe_ports port 21 # ftp
acl Safe_ports port 554 # rtsp

http_access deny !Safe_ports

## authentication

auth_param basic program /usr/lib/squid/squid_db_auth --user ******** --password ******** --plaintext --persist --dsn DBI:mysql:database=the_database
auth_param basic children 5
auth_param basic realm Web-Proxy
auth_param basic credentialsttl 1 minute
auth_param basic casesensitive off

acl db-auth proxy_auth REQUIRED
http_access allow db-auth
http_access deny all

cache_mgr thisisme_at_asite.com

httpd_suppress_version_string on
visible_hostname myserver
dns_nameservers 8.8.8.8 8.8.4.4

cache.log:

After a service start, the daemon will run for a while, and then I see the following events (note the first line):

2013/07/24 09:29:02| assertion failed: errorpage.cc:1064: "(size_t)content->contentSize() == strlen(content->content())"
2013/07/24 09:29:05| Starting Squid Cache version 3.1.10 for i386-redhat-linux-gnu...
2013/07/24 09:29:05| Process ID 2273
2013/07/24 09:29:05| With 1024 file descriptors available
2013/07/24 09:29:05| Initializing IP Cache...
2013/07/24 09:29:05| DNS Socket created at [::], FD 7
2013/07/24 09:29:05| DNS Socket created at 0.0.0.0, FD 8
2013/07/24 09:29:05| Adding nameserver 8.8.8.8 from squid.conf
2013/07/24 09:29:05| Adding nameserver 8.8.4.4 from squid.conf
2013/07/24 09:29:05| helperOpenServers: Starting 5/5 'squid_db_auth' processes
2013/07/24 09:29:05| User-Agent logging is disabled.
2013/07/24 09:29:05| Referer logging is disabled.
2013/07/24 09:29:05| Unlinkd pipe opened on FD 23
2013/07/24 09:29:05| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
2013/07/24 09:29:05| Store logging disabled
2013/07/24 09:29:05| Swap maxSize 16777216 + 131072 KB, estimated 1300637 objects
2013/07/24 09:29:05| Target number of buckets: 65031
2013/07/24 09:29:05| Using 65536 Store buckets
2013/07/24 09:29:05| Max Mem size: 131072 KB
2013/07/24 09:29:05| Max Swap size: 16777216 KB
2013/07/24 09:29:05| Version 1 of swap file with LFS support detected...
2013/07/24 09:29:05| Rebuilding storage in /opt/squid/cache (DIRTY)
2013/07/24 09:29:05| Using Least Load store dir selection
2013/07/24 09:29:05| Current Directory is /
2013/07/24 09:29:05| Loaded Icons.
2013/07/24 09:29:05| Accepting intercepted HTTP connections at 0.0.0.0:3128, FD 26.
2013/07/24 09:29:05| Accepting HTTP connections at [::]:4334, FD 27.
2013/07/24 09:29:05| Accepting HTTP connections at 127.0.0.1:33699, FD 28.
2013/07/24 09:29:05| Accepting HTTP connections at 127.0.0.1:32623, FD 29.
2013/07/24 09:29:05| Accepting HTTP connections at 127.0.0.1:26226, FD 30.
2013/07/24 09:29:05| Accepting HTTP connections at 127.0.0.1:26499, FD 31.
2013/07/24 09:29:05| Accepting HTTP connections at 127.0.0.1:18108, FD 32.
2013/07/24 09:29:05| commBind: Cannot bind socket FD 33 to MY.PUB.IP.ADR:3128: (98) Address already in use
2013/07/24 09:29:05| Accepting HTTP connections at 127.0.0.1:49236, FD 34.
2013/07/24 09:29:05| HTCP Disabled.
2013/07/24 09:29:05| Squid plugin modules loaded: 0
2013/07/24 09:29:05| Adaptation support is off.
2013/07/24 09:29:05| Ready to serve requests.
2013/07/24 09:29:05| Done reading /opt/squid/cache swaplog (717 entries)
2013/07/24 09:29:05| Finished rebuilding storage from disk.
2013/07/24 09:29:05| 699 Entries scanned
2013/07/24 09:29:05| 0 Invalid entries.
2013/07/24 09:29:05| 0 With invalid flags.
2013/07/24 09:29:05| 681 Objects loaded.
2013/07/24 09:29:05| 0 Objects expired.
2013/07/24 09:29:05| 18 Objects cancelled.
2013/07/24 09:29:05| 0 Duplicate URLs purged.
2013/07/24 09:29:05| 0 Swapfile clashes avoided.
2013/07/24 09:29:05| Took 0.01 seconds (111896.16 objects/sec).
2013/07/24 09:29:05| Beginning Validation Procedure
2013/07/24 09:29:05| Completed Validation Procedure
2013/07/24 09:29:05| Validated 1387 Entries
2013/07/24 09:29:05| store_swap_size = 9696
2013/07/24 09:29:06| storeLateRelease: released 0 objects

I confirmed that the daemon is in fact restarting by looking at the process tree and confirming both the PID revealed in the above cache.log as well as the timestamp of the event:

# ps -ef | grep squid
root 2194 1 0 09:06 ? 00:00:00 squid -f /etc/squid/squid.conf
squid 2273 2194 0 09:29 ? 00:00:00 (squid) -f /etc/squid/squid.conf

Aside from this issue, my main goal is to eventually build a config that handles dynamic content (YouTube and its CDNs, for example). Right now, content coming from YouTube or its CDNs is not loading through my cache, however other sites display content rapidly. Example: content from cnn.com displays perfectly, including video content from the Turner CDN.

My take on all this is that it appears the server may be starved for memory. I have seen other strange behaviors such mysqld, squid, and pptpd daemons dying randomly. I've seen squid daemon restart for reasons unknown (though the above cache.log suggests a cache digest rebuild every 3600 seconds).

Are there any obvious mistakes I've made here? I do plan to run my production proxy with more memory and am not in objection to adding more memory to this environment, but would like to do so with an understanding of the issues before moving forward.

Thanks in advance!
Mark
Received on Wed Jul 24 2013 - 06:29:33 MDT