Re: [squid-users] Squid does not respond to TCP SYN when there are thousands of connection

2013/2/15 Amos Jeffries <squid3_at_treenet.co.nz>:
> On 15/02/2013 11:53 p.m., dahanhsi wrote:
>>
>> Hi Amos,
>>
>>
>> 2013/2/15 Amos Jeffries <squid3_at_treenet.co.nz>:
>>>
>>> On 15/02/2013 10:43 p.m., dahanhsi wrote:
>>>>
>>>> Thanks for your reply,
>>>> provide more information below:
>>>>
>>>> 2013/2/15 Amos Jeffries <squid3_at_treenet.co.nz>:
>>>>>
>>>>> On 15/02/2013 10:12 p.m., dahanhsi wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I use squid as a reverse proxy, and make thousands of connection to
>>>>>> the
>>>>>> it.
>>>>>
>>>>> Which version of Squid?
>>>>
>>>> I use Squid 2.7
>>>
>>>
>>>
>>> Output of "squid -v" please.
>>
>> # squid -v
>> Squid Cache: Version 2.7.STABLE9
>> configure options: '--prefix=/usr' '--exec_prefix=/usr'
>> '--bindir=/usr/sbin' '--sbindir=/usr/sbin'
>> '--libexecdir=/usr/lib/squid' '--sysconfdir=/etc/squid'
>> '--localstatedir=/var/spool/squid' '--datadir=/usr/share/squid'
>> '--enable-async-io' '--with-pthreads'
>> '--enable-storeio=ufs,aufs,coss,diskd,null' '--enable-linux-netfilter'
>> '--enable-arp-acl' '--enable-epoll'
>> '--enable-removal-policies=lru,heap' '--enable-snmp'
>> '--enable-delay-pools' '--enable-htcp' '--enable-cache-digests'
>> '--enable-underscores' '--enable-referer-log' '--enable-useragent-log'
>> '--enable-auth=basic,digest,ntlm,negotiate'
>> '--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-carp'
>> '--enable-follow-x-forwarded-for' '--with-large-files'
>> '--with-maxfd=65536' 'amd64-debian-linux'
>> 'build_alias=amd64-debian-linux' 'host_alias=amd64-debian-linux'
>> 'target_alias=amd64-debian-linux' 'CFLAGS=-Wall -g -O2'
>> 'LDFLAGS=-Wl,-Bsymbolic-functions' 'CPPFLAGS='
>>
>>>
>>>
>>>>> What do you mean by "thousands of connections". 1's of tousands? 10's
>>>>> of
>>>>> thousands? 100's of thousands?
>>>>
>>>> # netstat -nat|grep -i "80"|wc -l
>>>> the result vary from 4651 to 9404
>>>>
>>>>>> There are one ten of all connections can not establish in TCP layer,
>>>>>> because squid does not respond SYN-ACK to client's SYN packet. How can
>>>>>> I solve it?
>>>>>> Thanks
>>>>>
>>>>>
>>>>> Check ulimit settings for Squid?
>>>>>
>>>> # ulimit -a
>>>> core file size (blocks, -c) 0
>>>> data seg size (kbytes, -d) unlimited
>>>> scheduling priority (-e) 20
>>>> file size (blocks, -f) unlimited
>>>> pending signals (-i) 16382
>>>> max locked memory (kbytes, -l) 64
>>>> max memory size (kbytes, -m) unlimited
>>>> open files (-n) 655360
>>>> pipe size (512 bytes, -p) 8
>>>> POSIX message queues (bytes, -q) 819200
>>>> real-time priority (-r) 0
>>>> stack size (kbytes, -s) 8192
>>>> cpu time (seconds, -t) unlimited
>>>> max user processes (-u) unlimited
>>>> virtual memory (kbytes, -v) unlimited
>>>> file locks (-x) unlimited
>>>>
>>>>> Check your cache.log for messages about running out of filedescriptors?
>>>>
>>>> I set my limit.conf to:
>>>> root soft nofile 655360
>>>> root hard nofile 655360
>>>
>>>
>>> That does not answer the question. Squid may have been built or
>>> configured
>>> with a limit of less than 655360 filedescriptors.
>>> cache.log should tell you if Squid is reaching some limit like this.
>>
>> my cache.log:
>> 2013/02/15 8:30:10| Starting Squid Cache version 2.7.STABLE9 for
>> x86_64-debian-linux-gnu...
>> 2013/02/15 8:30:10| Process ID 8136
>> 2013/02/15 8:30:10| With 2048 file descriptors available
>> 2013/02/15 8:30:10| Using epoll for the IO loop
>> 2013/02/15 8:30:10| DNS Socket created at 0.0.0.0, port 6450, FD 6
>> 2013/02/15 8:30:10| Adding nameserver 8.8.8.8 from /etc/resolv.conf
>> 2013/02/15 8:30:10| User-Agent logging is disabled.
>> 2013/02/15 8:30:10| Referer logging is disabled.
>> 2013/02/15 8:30:10| logfileOpen: opening log /var/log/squid/access.log
>> 2013/02/15 8:30:10| Unlinkd pipe opened on FD 12
>> 2013/02/15 8:30:10| Swap maxSize 8192 + 8388608 KB, estimated 645907
>> objects
>> 2013/02/15 8:30:10| Target number of buckets: 32295
>> 2013/02/15 8:30:10| Using 32768 Store buckets
>> 2013/02/15 8:30:10| Max Mem size: 8388608 KB
>> 2013/02/15 8:30:10| Max Swap size: 8192 KB
>> 2013/02/15 8:30:10| Local cache digest enabled; rebuild/rewrite every
>> 3600/3600 sec
>> 2013/02/15 8:30:10| logfileOpen: opening log /var/log/squid/store.log
>> 2013/02/15 8:30:10| Rebuilding storage in /var/spool/squid (CLEAN)
>> 2013/02/15 8:30:10| Using Least Load store dir selection
>> 2013/02/15 8:30:10| Set Current Directory to /var/spool/squid
>> 2013/02/15 8:30:10| Loaded Icons.
>> 2013/02/15 8:30:10| Accepting accelerated HTTP connections at 0.0.0.0,
>> port 80, FD 14.
>> 2013/02/15 8:30:10| Accepting ICP messages at 0.0.0.0, port 3130, FD 15.
>> 2013/02/15 8:30:10| HTCP Disabled.
>> 2013/02/15 8:30:10| WCCP Disabled.
>> 2013/02/15 8:30:10| Configuring localhost Parent localhost/12080/0
>> 2013/02/15 8:30:10| Ready to serve requests.
>> 2013/02/15 8:30:10| Done reading /var/spool/squid swaplog (11 entries)
>> 2013/02/15 8:30:10| Finished rebuilding storage from disk.
>> 2013/02/15 8:30:10| 11 Entries scanned
>> 2013/02/15 8:30:10| 0 Invalid entries.
>> 2013/02/15 8:30:10| 0 With invalid flags.
>> 2013/02/15 8:30:10| 11 Objects loaded.
>> 2013/02/15 8:30:10| 0 Objects expired.
>> 2013/02/15 8:30:10| 0 Objects cancelled.
>> 2013/02/15 8:30:10| 0 Duplicate URLs purged.
>> 2013/02/15 8:30:10| 0 Swapfile clashes avoided.
>> 2013/02/15 8:30:10| Took 0.3 seconds ( 41.8 objects/sec).
>> 2013/02/15 8:30:10| Beginning Validation Procedure
>> 2013/02/15 8:30:10| Completed Validation Procedure
>> 2013/02/15 8:30:10| Validated 11 Entries
>> 2013/02/15 8:30:10| store_swap_size = 44k
>> 2013/02/15 8:30:11| storeLateRelease: released 0 objects
>> 2013/02/15 8:30:35| CACHEMGR: <unknown>@127.0.0.1 requesting 'info'
>> 2013/02/15 8:30:39| CACHEMGR: <unknown>@127.0.0.1 requesting 'info'
>> 2013/02/15 8:30:40| CACHEMGR: <unknown>@127.0.0.1 requesting 'info'
>> 2013/02/15 8:30:42| CACHEMGR: <unknown>@127.0.0.1 requesting 'info'
>> 2013/02/15 8:30:44| CACHEMGR: <unknown>@127.0.0.1 requesting 'info'
>> 2013/02/15 8:33:10| CACHEMGR: <unknown>@127.0.0.1 requesting 'info'
>>
>> when connection timeout error occurred, I do not see error about file
>> descriptor.
>>
>> my /etc/sysclt.conf
>> net.ipv4.tcp_syncookies = 1
>> net.ipv4.tcp_tw_reuse = 1
>> net.ipv4.tcp_tw_recycle = 1
>> net.ipv4.tcp_fin_timeout = 30
>> fs.file-max = 65536
>>
>> my squid.conf has:
>> max_filedescriptors 2048
>>
>> and my squidclient says:
>> squidclient -p 80 mgr:info | grep "file desc"
>> Maximum number of file descriptors: 2048
>> Available number of file descriptors: 1651
>> Reserved number of file descriptors: 100
>
>
> There you go then. Squid is not permitted to _use_ more than 1651 FD. Every
> client TCP connection uses at least 1, sometimes 2 FD.
> When all the FD are used up Squid waits until some are free'd before
> accepting more client connections.
>
> With "from 4651 to 9404" I would set your max_filedescriptors to at least
> 18000. It can be anything up to the ulimit max.

I set max_filedescriptors to 655360, and confirm that ulimit -n is also 655360.
After restart Squid, I observe that rate of connection timeout in
client is still about 10%, and no additional error such as
filedescriptors error found in cache.log or dmesg.

any ideas?
thanks

>
> Amos
Received on Fri Feb 15 2013 - 14:12:10 MST