Re: [squid-users] Why squid instance' cpu load so high, can rebuild so frequency ?

Hey there,

You might not familiar with how to debug things so there you have the
chance.

Rebuilding of the cache can be because of some problem in the cache
index or something similar.
If the rebuilding is a part of a crash recovery which cause the cache
dir to have a problem you will need to find the bug itself that causes
the crash while the rebuilding is a side effect of it.
Is there a reason for you to use DISKD cache dir?(wondering)

- there was a bug with null char in the request which caused many
problems and I am not sure if it was fixed or not yet but it's in the
bugzilla and squid-dev.
Using the bugzilla you get more attention from the developers about the
bug and the fix progress.

There is a newer version of 3.2 which you should always try since it's
the latest stable version and contains couple important fixes.

couple questions about this server:
- is it for cache only or any other use?
- what clients are using the proxy? firefox? chrome? others? etc..
- the http_port settings is more for a reverse proxy rather then a
forward proxy.
I dont know the exact needs of your project but you do have some
specific settings which seems to not make sense to me but it might be
because you didn't mentioned the purpose of the proxy and missing some
squid.conf parts if i'm not wrong.

All the information you gave until now points to some bug but it can
also be from a bad client and reveals squid weakness but it's still
unknown what is causing the exact problem.
The proxy runs for more then 5 minutes after some incident without any
information more then a client tries to abuse\harm in a way squid.

I suggest you to take a look at the access.log of the same time it
happens to maybe find the culprit computer or software to isolate the
problem and then narrow down the issues.

Get familiar with debug sections:
http://wiki.squid-cache.org/KnowledgeBase/DebugSections

Which can give you more information of the clients and on the requests.
In a case of loaded server you will need to narrow down a lot.

So try to upgrade to the latest 3.2 and in a case you still have the
problem you can try to debug squid based on this article:
http://wiki.squid-cache.org/SquidFaq/BugReporting

Feel free to ask any question you have in mind.

Best Regards,
Eliezer

On 1/29/2013 6:02 AM, 金 戈 wrote:
> Sorry for my lackless of the information.
>
> The squid options and distro below.
> FreeBSD cache4.cnwisp.com 9.1-RELEASE FreeBSD 9.1-RELEASE
>
> Squid Cache: Version 3.2.4
> configure options: '--with-default-user=squid' '--bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid' '--libexecdir=/usr/local/libexec/squid' '--localstatedir=/var/squid' '--sysconfdir=/usr/local/etc/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid/squid.pid' '--enable-auth' '--enable-build-info' '--enable-loadable-modules' '--enable-removal-policies=lru heap' '--disable-epoll' '--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-translation' '--enable-auth-basic=DB MSNT MSNT-multi-domain NCSA PAM POP3 RADIUS fake getpwnam' '--enable-auth-digest=file' '--enable-external-acl-helpers=file_userip unix_group' '--enable-auth-negotiate=none' '--enable-auth-ntlm=fake smb_lm' '--enable-storeio=diskd rock ufs aufs' '--enable-disk-io=AIO Blocking DiskDaemon IpcIo Mmapped DiskThreads' '--enable-log-daemon-helpers=file' '--enable-url-rewrite-helpers=fake' '--disable-ipv6' '--enable-ssl' '--with-openssl=/usr/local' '--ena
 ble-ssl
-crtd' '--enable-htcp' '--enable-forw-via-db' '--disable-cache-digests' '--disable-wccp' '--enable-wccpv2' '--disable-ident-lookups' '--enable-ipfw-transparent' '--enable-pf-transparent' '--disable-ipf-transparent' '--enable-follow-x-forwarded-for' '--disable-ecap' '--enable-icap-client' '--disable-esi' '--enable-kqueue' '--with-large-files' '--with-filedescriptors=64000' '--with-aufs-threads=511' '--enable-err-languages=English' '--enable-default-err-language=English' '--disable-eui' '--disable-auto-locale' '--enable-http-violations' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/' '--build=amd64-portbld-freebsd9.1' 'build_alias=amd64-portbld-freebsd9.1' 'CC=cc' 'CFLAGS=-O2 -pipe -I/usr/local/include -fno-strict-aliasing' 'LDFLAGS= -pthread -Wl,-rpath=/usr/local/lib -L/usr/local/lib' 'CPPFLAGS=' 'CXX=c++' 'CXXFLAGS=-O2 -pipe -I/usr/local/include -fno-strict-aliasing' 'CPP=cpp' --enable-ltdl-convenience
>
> and my squid.conf
>
> cache_mem 512 MB
> memory_replacement_policy heap GDSF
> #memory_replacement_policy heap LRU
> memory_cache_shared off
> minimum_object_size 0 KB
> #cache_replacement_policy heap GDSF
> cache_replacement_policy heap LFUDA
> maximum_object_size 512 KB
> cache_swap_low 85
> cache_swap_high 95
> logfile_daemon /usr/local/libexec/squid/log_file_daemon
> buffered_logs on
> negative_ttl 15 seconds
> positive_dns_ttl 6 hours
> negative_dns_ttl 30 seconds
> store_avg_object_size 26 KB
> store_objects_per_bucket 30
> read_ahead_gap 64 KB
> request_header_max_size 64 KB
> reply_header_max_size 64 KB
> via off
> request_entities on
> forward_timeout 1 minutes
> connect_timeout 15 seconds
> peer_connect_timeout 10 seconds
> read_timeout 3 minutes
> write_timeout 3 minutes
> request_timeout 30 seconds
> client_idle_pconn_timeout 1 minutes
> client_lifetime 1 hours
> server_idle_pconn_timeout 1 minute
> cache_effective_user squid
> cache_effective_group squid
> httpd_suppress_version_string on
> client_persistent_connections off
> query_icmp off
> accept_filter httpready
> dns_v4_first on
> check_hostnames off
> ipcache_size 65535
> fqdncache_size 65535
> max_filedescriptors 50000
> memory_pools on
> memory_pools_limit 50 MB
> forwarded_for transparent
> client_db off
> http_port 10.10.1.1:3128 accel allow-direct ignore-cc
> snmp_incoming_address 10.10.1.1
> udp_incoming_address 10.10.1.1
> icp_port 0
> htcp_port 0
> snmp_port 3401
> cache_dir diskd /cache1/aufs-32k 8000 32 256 max-size=32768 Q1=100 Q2=128
> cache_dir diskd /cache2/aufs-32k 8000 32 256 max-size=32768 Q1=100 Q2=128
> cache_dir diskd /cache3/aufs-32k 8000 32 256 max-size=32768 Q1=100 Q2=128
> cache_dir diskd /cache4/aufs-512k 32000 16 256 min-size=32769 max-size=524288 Q1=100 Q2=128
>
>
>
>
>
> 在 2013-1-29，上午11:34，Eliezer Croitoru <eliezer_at_ngtech.co.il> 写道：
>
>> Hey,
>>
>> What exact version of 3.2+ are you using?
>> What distro?
>> squid.conf..
>> Self compiled or from a Repo?
>> squid build options from "squid -v"
>>
>> There is a problem but it's not enough basic data to even look at it.
>> Squid 3.2+ was patched couple times and it might have been fixed in one of these patches.
>>
>> Regards,
>> Eliezer
>>
>> On 1/29/2013 5:19 AM, 金 戈 wrote:
>>> Greetings!
>>> We use squid as a forward proxy for our project.Recently we found that one of the instance rebuilding a lot of times.
>>> And we check the configure found all the instance use the same. But just this instance always rebuilding( about 3 ~ 4 times per day)
>>> And we found the cache log has some thing below.
>>>
>>> 2013/01/29 09:56:40 kid1| ctx: enter level 0: 'http://bo.ok168.com/music/<CD><F4><CB><D5><E3><F1>/<BA><C3><B0><B2><BE><B2>.wma'
>>> 2013/01/29 09:56:40 kid1| WARNING: unparseable HTTP header field {<h1><body bgcolor="red">Bad Request (Invalid Hostname)</body></h1>}
>>>
>>> #this is when i shutdown the instance.
>>> 2013/01/29 10:03:38 kid1| Preparing for shutdown after 9796891 requests
>>> 2013/01/29 10:03:38 kid1| Waiting 30 seconds for active connections to finish
>>> 2013/01/29 10:03:38 kid1| Closing HTTP port 192.168.134.16:3128
>>> 2013/01/29 10:03:38 kid1| Closing SNMP receiving port 192.168.134.16:3401
>>> 2013/01/29 10:03:38 kid1| Shutdown: NTLM authentication.
>>> 2013/01/29 10:03:38 kid1| Shutdown: Negotiate authentication.
>>> 2013/01/29 10:03:38 kid1| Shutdown: Digest authentication.
>>> 2013/01/29 10:03:38 kid1| Shutdown: Basic authentication.
>>> 2013/01/29 10:03:38 kid1| assertion failed: errorpage.cc:608: "entry->isEmpty()"
>

-- 
Eliezer Croitoru