Re: [squid-users] squid cpu problem

Dear Amos
we are using 2 GB of memory as cache_mem and 128 KB as max object size
in memory

any way i decreased max object size in memory to 32 KB and still have
the spikes
this is GDB to squid in 100% spike bit the error related to epoll is
this the same error like the previous one

0x00007ffa7466e103 in epoll_wait () from /lib64/libc.so.6
(gdb) backtrace
#0 0x00007ffa7466e103 in epoll_wait () from /lib64/libc.so.6
#1 0x00000000004bdee2 in comm_select (msec=<optimized out>) at
comm_epoll.cc:266
#2 0x000000000057606e in CommSelectEngine::checkEvents
(this=<optimized out>, timeout=<optimized out>) at comm.cc:2688
#3 0x00000000004d35d4 in EventLoop::checkEngine (this=0x7ffff604d7a0,
engine=0x7ffff604d810, primary=<optimized out>) at EventLoop.cc:50
#4 0x00000000004d3845 in EventLoop::runOnce (this=0x7ffff604d7a0) at
EventLoop.cc:124
#5 0x00000000004d3938 in EventLoop::run (this=0x7ffff604d7a0) at
EventLoop.cc:94
#6 0x000000000051d35b in FalconMain (argc=<optimized out>,
argv=<optimized out>) at main.cc:1418
#7 0x000000000051dd83 in FalconMainSafe (argv=<optimized out>,
argc=<optimized out>) at main.cc:1176
#8 main (argc=<optimized out>, argv=<optimized out>) at main.cc:1168

Regards
Ayham

On 2014-04-05 03:37, Amos Jeffries wrote:
> This looks like the CPU cycles are being consumed by walking one or
> more
> very long lists of memory pieces and writing them to disk one by one.
> Note the UFSStoreState::write parameter size=4096 in the backtrace for
> how bit those memory pages are.
>
> Which could happen if you cached a very big object in cache_mem and
> then
> a random time later it needed swapping out to disk to free up memory.
>
> It could also happen if Squid needed to suddenly swap out a large
> number
> of smaller items to make memory space available for a large one which
> is
> about to arrive.
>
> So, have you configured Squid to allow very large objects (many MB or
> GB) in memory storage?
>
>
> Note these causes would not show up in the testing you mentioned
> unless
> you had a very wide range of test object sizes being pumped randomly
> through the proxy. A tool like web polygraph is best to test that
> traffic behaviour accurately.
>
> Amos
>
>
> On 5/04/2014 1:59 a.m., a.afach wrote:
>> Dear all
>> i still have the CPU spikes even when i used
>> disable-strict-error-checking without using Cflags
>>
>> this is the gdb backtrace while the CPU spikes
>>
>> 0x000000000051b348 in linklistPush (L=0x11853e188, p=0xce6d4300) at
>> list.cc:47
>> 47 while (*L)
>> (gdb) backtrace
>> #0 0x000000000051b348 in linklistPush (L=0x11853e188, p=0xce6d4300)
>> at
>> list.cc:47
>> #1 0x00000000005a70a1 in UFSStoreState::write (this=0xb3970e28,
>> buf=0x11fe69ca0
>> "!v\253r[/\307\232G\b\375`\237:\213\256^\335\373{\241%\232\363\021\071>`\342\033\177a\202G\320{\323%\236K\342\243*\332\316\351\231=\360\370\313Ro=\317\262\243\315\027\351,\221\230\353Z\023\024q\"QSC\036\214:M\242{@\351m\020\337Cw_\214\216\304\226\265\a\375\031\211\243V\222T\320\016\227\312-\211Sz\326^\346\230\251\327\222\n\373I\032\341\303==U\214\277\264\244\205\b1\346S=\230\215\204\245\254>\312\223\066\336\230PpP\227\271\370\266;\362\226\242\036\225\235w\330\325\061\316{o_\364\021\062\351\376\062|\313\006`\357m\206FQ0\021\030C\224\004]\336\315\371\033h1\361\363\350d\366\066"...,
>> size=4096, aOffset=-1, free_func=0x5203b0
>> <memNodeWriteComplete(void*)>)
>> at ufs/store_io_ufs.cc:247
>> #2 0x0000000000554ca0 in doPages (anEntry=<optimized out>) at
>> store_swapout.cc:160
>> #3 StoreEntry::swapOut (this=0x372ca10) at store_swapout.cc:279
>> #4 0x000000000054c986 in StoreEntry::invokeHandlers (this=0x372ca10)
>> at
>> store_client.cc:714
>> #5 0x00000000004dc1a7 in FwdState::complete (this=0xbb502b48) at
>> forward.cc:341
>> #6 0x00000000005579a5 in ServerStateData::completeForwarding
>> (this=0xf8030588) at Server.cc:239
>> #7 0x00000000005571bd in ServerStateData::serverComplete2
>> (this=0xf8030588) at Server.cc:207
>> #8 0x00000000004ff3dc in HttpStateData::processReplyBody
>> (this=0xf8030588) at http.cc:1382
>> #9 0x00000000004fd367 in HttpStateData::readReply (this=0xf8030588,
>> io=...) at http.cc:1161
>> #10 0x0000000000503156 in JobDialer<HttpStateData>::dial
>> (this=0xde75ca50, call=...) at base/AsyncJobCalls.h:175
>> #11 0x0000000000569ee4 in AsyncCall::make (this=0xde75ca20) at
>> AsyncCall.cc:34
>> #12 0x000000000056cb76 in AsyncCallQueue::fireNext (this=<optimized
>> out>) at AsyncCallQueue.cc:53
>> #13 0x000000000056ccf0 in AsyncCallQueue::fire (this=0x2586400) at
>> AsyncCallQueue.cc:39
>> #14 0x00000000004d385c in EventLoop::runOnce (this=0x7fffcb3518d0) at
>> EventLoop.cc:130
>> #15 0x00000000004d3938 in EventLoop::run (this=0x7fffcb3518d0) at
>> EventLoop.cc:94
>> #16 0x000000000051d35b in SquidMain (argc=<optimized out>,
>> argv=<optimized out>) at main.cc:1418
>> #17 0x000000000051dd83 in SquidMainSafe (argv=<optimized out>,
>> argc=<optimized out>) at main.cc:1176
>> #18 main (argc=<optimized out>, argv=<optimized out>) at main.cc:1168
>>
>>
>> any idea about what's causing the cpu spike
>>
>>
>> On 2014-03-31 16:34, Amos Jeffries wrote:
>>> On 2014-04-01 02:10, a.afach wrote:
>>>> Dear Eliezer
>>>> these are the configure options ...
>>>> configure options: '--prefix=/usr/local/squid-3.1.19'
>>>> '--sysconfdir=/etc' '--sysconfdir=/etc/squid'
>>>> '--localstatedir=/var'
>>>> '--enable-auth=basic,digest,ntlm'
>>>> '--enable-removal-policies=lru,heap'
>>>> '--enable-digest-auth-helpers=password'
>>>> '--enable-basic-auth-helpers=PAM,getpwnam,NCSA,MSNT'
>>>> '--enable-external-acl-helpers=ip_user,session,unix_group'
>>>> '--enable-ntlm-auth-helpers=fakeauth'
>>>> '--enable-ident-lookups--enable-useragent-log'
>>>> '--enable-cache-digests' '--enable-delay-pools'
>>>> '--enable-referer-log'
>>>> '--enable-arp-acl' '--with-pthreads' '--with-large-files'
>>>> '--enable-htcp' '--enable-carp' '--enable-follow-x-forwarded-for'
>>>> '--enable-snmp' '--enable-ssl' '--enable-storeio=ufs,diskd,aufs'
>>>> '--enable-async-io' '--enable-linux-netfilter' '--enable-epoll'
>>>> '--with-squid=/usr/squid-3.1.19' '--disable-ipv6' '--with-aio'
>>>> '--with-aio-threads=128' 'build_alias=x86_64-pc-linux-gnu'
>>>> 'host_alias=x86_64-pc-linux-gnu' 'CC=x86_64-pc-linux-gnu-gcc'
>>>> 'CFLAGS=-O2 -pipe -m64 -mtune=generic' 'LDFLAGS=-Wl,-O1
>>>> -Wl,--as-needed' 'CXXFLAGS=' '--cache-file=/dev/null' '--srcdir=.'
>>>>
>>>
>>> Some more reasons to upgrade:
>>> * --disable-strict-error-checking avoids issues on Gentoo with
>>> -Werror
>>> * CFLAGS affects the C compiler, not the C++ compiler. C compiler
>>> is
>>> only used by Squid-3 to build some libraries.
>>> * current verified stable Gentoo Squid version is 3.3.8.
>>> * updating aything on Gentoo involves rebuilding a surprising
>>> number
>>> of components from scratch. So when you get a difference like this
>>> it
>>> really could be anywhere. Including buried in the compiler itself -
>>> your flags are possibly changing optimization levels and
>>> CPU-specific
>>> assembly instructions used by it.
>>>
>>> Amos
Received on Sun Apr 06 2014 - 15:00:52 MDT