Re: [squid-users] Ongoing Running out of filedescriptors

From: Luis Daniel Lucio Quiroz <luis.daniel.lucio_at_gmail.com>
Date: Tue, 9 Feb 2010 17:39:37 -0600

Le Mardi 9 Février 2010 17:29:23, Landy Landy a écrit :
> I don't know what to do with my current squid, I even upgraded to
> 3.0.STABLE21 but, the problem persist every three days:
>
> /usr/local/squid/sbin/squid -v
> Squid Cache: Version 3.0.STABLE21
> configure options: '--prefix=/usr/local/squid' '--sysconfdir=/etc/squid'
> '--enable-delay-pools' '--enable-kill-parent-hack' '--disable-htcp'
> '--enable-default-err-language=Spanish' '--enable-linux-netfilter'
> '--disable-ident-lookups' '--localstatedir=/var/log/squid3.1'
> '--enable-stacktraces' '--with-default-user=proxy' '--with-large-files'
> '--enable-icap-client' '--enable-async-io' '--enable-storeio=aufs'
> '--enable-removal-policies=heap,lru' '--with-maxfd=32768'
>
> I built with --with-maxfd=32768 option but, when squid is started it says
> is working with only 1024 filedescriptor.
>
> I even added the following to the squid.conf:
>
> max_open_disk_fds 0
>
> But it hasn't resolve anything. I'm using squid on Debian Lenny. I don't
> know what to do. Here's part of cache.log:
>
> 2010/02/09 17:14:29| ctx: exit level 0
> 2010/02/09 17:14:29| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 17:16:50| client_side.cc(2843) WARNING!
> Your cache is running out of filedescriptors 2010/02/09 17:18:45|
> client_side.cc(2843) WARNING! Your cache is running out of filedescriptors
> 2010/02/09 17:20:01| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 17:20:17| client_side.cc(2843) WARNING!
> Your cache is running out of filedescriptors 2010/02/09 17:20:38|
> client_side.cc(2843) WARNING! Your cache is running out of filedescriptors
> 2010/02/09 17:21:33| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 17:22:26| client_side.cc(2843) WARNING!
> Your cache is running out of filedescriptors 2010/02/09 17:22:41|
> clientParseRequestMethod: Unsupported method attempted by 172.16.100.83:
> This is not a bug. see squid.conf extension_methods 2010/02/09 17:22:41|
> clientParseRequestMethod: Unsupported method in request
> '_u_@________.#C5U_________e__:___{_Q_"___L_____________r' 2010/02/09
> 17:22:41| clientProcessRequest: Invalid Request
> 2010/02/09 17:22:43| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 17:22:59| client_side.cc(2843) WARNING!
> Your cache is running out of filedescriptors 2010/02/09 17:23:16|
> client_side.cc(2843) WARNING! Your cache is running out of filedescriptors
> 2010/02/09 17:23:36| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 17:23:52| client_side.cc(2843) WARNING!
> Your cache is running out of filedescriptors 2010/02/09 17:24:19|
> client_side.cc(2843) WARNING! Your cache is running out of filedescriptors
> 2010/02/09 17:24:23| clientNatLookup: NF getsockopt(SO_ORIGINAL_DST)
> failed: (2) No such file or directory 2010/02/09 17:24:38|
> client_side.cc(2843) WARNING! Your cache is running out of filedescriptors
> 2010/02/09 17:24:41| clientParseRequestMethod: Unsupported method
> attempted by 172.16.100.83: This is not a bug. see squid.conf
> extension_methods 2010/02/09 17:24:41| clientParseRequestMethod:
> Unsupported method in request
> '_E__&_b_____%_w__Pw__M_}z________%__I____@_T__Q___d__?_G' 2010/02/09
> 17:24:41| clientProcessRequest: Invalid Request
> 2010/02/09 17:24:54| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 17:25:12| client_side.cc(2843) WARNING!
> Your cache is running out of filedescriptors 2010/02/09 17:25:12|
> clientParseRequestMethod: Unsupported method attempted by 172.16.100.83:
> This is not a bug. see squid.conf extension_methods 2010/02/09 17:25:12|
> clientParseRequestMethod: Unsupported method in request
> '_Z___|G3_7^_%____U_r_1____.h__gd__8C' 2010/02/09 17:25:12|
> clientProcessRequest: Invalid Request
> 2010/02/09 17:25:29| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 17:25:41| clientNatLookup: NF
> getsockopt(SO_ORIGINAL_DST) failed: (2) No such file or directory
> 2010/02/09 17:25:45| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 17:26:01| client_side.cc(2843) WARNING!
> Your cache is running out of filedescriptors 2010/02/09 17:26:18|
> client_side.cc(2843) WARNING! Your cache is running out of filedescriptors
> 2010/02/09 17:26:34| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 17:26:59| client_side.cc(2843) WARNING!
> Your cache is running out of filedescriptors 2010/02/09 17:27:26|
> client_side.cc(2843) WARNING! Your cache is running out of filedescriptors
> 2010/02/09 17:27:29| clientNatLookup: NF getsockopt(SO_ORIGINAL_DST)
> failed: (2) No such file or directory 2010/02/09 17:27:56|
> client_side.cc(2843) WARNING! Your cache is running out of filedescriptors
> 2010/02/09 17:28:12| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 17:28:35| client_side.cc(2843) WARNING!
> Your cache is running out of filedescriptors 2010/02/09 17:28:35|
> clientNatLookup: NF getsockopt(SO_ORIGINAL_DST) failed: (2) No such file
> or directory 2010/02/09 17:29:25| client_side.cc(2843) WARNING! Your cache
> is running out of filedescriptors 2010/02/09 17:29:37| clientNatLookup: NF
> getsockopt(SO_ORIGINAL_DST) failed: (2) No such file or directory
> 2010/02/09 17:29:43| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 17:30:25| client_side.cc(2843) WARNING!
> Your cache is running out of filedescriptors 2010/02/09 17:30:41|
> client_side.cc(2843) WARNING! Your cache is running out of filedescriptors
> 2010/02/09 17:30:41| clientNatLookup: NF getsockopt(SO_ORIGINAL_DST)
> failed: (2) No such file or directory 2010/02/09 17:30:57|
> client_side.cc(2843) WARNING! Your cache is running out of filedescriptors
> 2010/02/09 17:31:15| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 17:31:29| clientParseRequestMethod:
> Unsupported method attempted by 172.16.100.83: This is not a bug. see
> squid.conf extension_methods 2010/02/09 17:31:29|
> clientParseRequestMethod: Unsupported method in request
> '_5I_A_+___=_`_______p_#_______M_g_Z__3_(_____r____ay[`____W[' 2010/02/09
> 17:31:29| clientProcessRequest: Invalid Request
> 2010/02/09 17:31:36| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 17:31:42| clientNatLookup: NF
> getsockopt(SO_ORIGINAL_DST) failed: (2) No such file or directory
> 2010/02/09 17:32:01| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 17:32:24| client_side.cc(2843) WARNING!
> Your cache is running out of filedescriptors 2010/02/09 17:32:59|
> client_side.cc(2843) WARNING! Your cache is running out of filedescriptors
> 2010/02/09 17:33:03| clientNatLookup: NF getsockopt(SO_ORIGINAL_DST)
> failed: (2) No such file or directory 2010/02/09 17:33:19|
> client_side.cc(2843) WARNING! Your cache is running out of filedescriptors
> 2010/02/09 17:33:19| clientParseRequestMethod: Unsupported method
> attempted by 172.16.100.83: This is not a bug. see squid.conf
> extension_methods 2010/02/09 17:33:19| clientParseRequestMethod:
> Unsupported method in request
> '________:__A_uW_*_____oM_`;______________j__G________z' 2010/02/09
> 17:33:19| clientProcessRequest: Invalid Request
> 2010/02/09 17:33:20| clientParseRequestMethod: Unsupported method attempted
> by 172.16.100.83: This is not a bug. see squid.conf extension_methods
> 2010/02/09 17:33:20| clientParseRequestMethod: Unsupported method in
> request '_____l_______(B__gbE_6(M_FN___R____a7)________$____Y__!____;q'
> 2010/02/09 17:33:20| clientProcessRequest: Invalid Request
> 2010/02/09 17:35:11| clientNatLookup: NF getsockopt(SO_ORIGINAL_DST)
> failed: (2) No such file or directory 2010/02/09 17:35:11|
> client_side.cc(2843) WARNING! Your cache is running out of filedescriptors
> 2010/02/09 17:35:31| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 17:36:53| client_side.cc(2843) WARNING!
> Your cache is running out of filedescriptors 2010/02/09 17:36:53|
> clientNatLookup: NF getsockopt(SO_ORIGINAL_DST) failed: (2) No such file
> or directory 2010/02/09 17:39:42| client_side.cc(2843) WARNING! Your cache
> is running out of filedescriptors 2010/02/09 17:39:42| clientNatLookup: NF
> getsockopt(SO_ORIGINAL_DST) failed: (2) No such file or directory
> 2010/02/09 17:51:53| clientNatLookup: NF getsockopt(SO_ORIGINAL_DST)
> failed: (2) No such file or directory 2010/02/09 17:51:53|
> client_side.cc(2843) WARNING! Your cache is running out of filedescriptors
> 2010/02/09 17:55:33| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 17:55:33| clientNatLookup: NF
> getsockopt(SO_ORIGINAL_DST) failed: (2) No such file or directory
> 2010/02/09 18:07:11| WARNING: Closing client 172.16.100.59 connection due
> to lifetime timeout 2010/02/09 18:07:11|
> http://207.46.124.167/gateway/gateway.dll?SessionID=1116141591.166024643
> 2010/02/09 18:07:11| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 18:07:11| clientNatLookup: NF
> getsockopt(SO_ORIGINAL_DST) failed: (2) No such file or directory
> 2010/02/09 18:07:48| WARNING: Closing client 172.16.100.59 connection due
> to lifetime timeout 2010/02/09 18:07:48|
> http://65.54.48.73/gateway/gateway.dll?SessionID=2000755641.1814461333
> 2010/02/09 18:07:48| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 18:09:20| WARNING: Closing client
> 172.16.100.59 connection due to lifetime timeout 2010/02/09 18:09:20|
> http://65.54.50.155/gateway/gateway.dll?SessionID=847035839.1893599263
> 2010/02/09 18:09:20| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 18:09:20| clientNatLookup: NF
> getsockopt(SO_ORIGINAL_DST) failed: (2) No such file or directory
> 2010/02/09 18:11:07| WARNING: Closing client 172.16.100.59 connection due
> to lifetime timeout 2010/02/09 18:11:07|
> http://65.54.172.52/gateway/gateway.dll?SessionID=1169589683.1509835191
> 2010/02/09 18:11:07| client_side.cc(2843) WARNING! Your cache is running
> out of filedescriptors 2010/02/09 18:11:07| clientNatLookup: NF
> getsockopt(SO_ORIGINAL_DST) failed: (2) No such file or directory

You got a bug! that behaivor happens when a coredump occurs in squid, please
file a ticket with gdb output, rice debug at maximum if you can.

solutions
1) try latest stable
2) recompile kernel with these patches:
[root_at_greycoat SOURCES]# cat fs.patch
--- include/linux/fs.h.dlucio 2009-09-04 13:38:25.000000000 -0500
+++ include/linux/fs.h 2009-09-04 13:46:40.000000000 -0500
@@ -22,7 +22,7 @@
 /* Fixed constants first: */
 #undef NR_OPEN
 #define NR_OPEN (1024*1024) /* Absolute upper limit on fd num */
-#define INR_OPEN 1024 /* Initial setting for nfile rlimits */
+#define INR_OPEN (1024*8*8) /* Initial setting for nfile rlimits */

 #define BLOCK_SIZE_BITS 10
 #define BLOCK_SIZE (1<<BLOCK_SIZE_BITS)
@@ -54,7 +54,7 @@
 extern int dir_notify_enable;
 #endif

-#define NR_FILE 8192 /* this can well be larger on a larger system */
+#define NR_FILE (8192*1024*1024*8192) /* this can well be larger on a larger
system */

 #define MAY_EXEC 1
 #define MAY_WRITE 2

[root_at_greycoat SOURCES]# cat limits.patch
--- include/linux/limits.h.dlucio 2009-09-04 13:36:48.000000000 -0500
+++ include/linux/limits.h 2009-09-04 13:47:15.000000000 -0500
@@ -1,7 +1,7 @@
 #ifndef _LINUX_LIMITS_H
 #define _LINUX_LIMITS_H

-#define NR_OPEN 1024
+#define NR_OPEN (1024*1024*8)

 #define NGROUPS_MAX 65536 /* supplemental group IDs are available */
 #define ARG_MAX 131072 /* # bytes of args + environ for exec() */

these will permit squid when coredump to have at least 8192 descriptors
instead of 1024.

LD
Received on Tue Feb 09 2010 - 23:39:47 MST

This archive was generated by hypermail 2.2.0 : Wed Feb 10 2010 - 12:00:05 MST