Re: [squid-users] Log Files running out disk space

Nyamul Hassan wrote:
> On Tue, Aug 17, 2010 at 17:03, Amos Jeffries <squid3_at_treenet.co.nz> wrote:
>> Nyamul Hassan wrote:
>>> Hi,
>>>
>>> One of proxies died down today, because the log files were overwhelming:
>>>
>>> -rw-r----- 1 squid squid 61440 Aug 17 16:01 access.log
>>> -rw-r----- 1 squid squid 523366451 Aug 17 02:59 access.log.0
>>> -rw-r----- 1 squid squid 771658231 Aug 17 00:00 access.log.1
>>> -rw-r----- 1 squid squid 562853886 Aug 16 21:00 access.log.2
>>> -rw-r----- 1 squid squid 618221433 Aug 16 18:00 access.log.3
>>> -rw-r----- 1 squid squid 572403480 Aug 16 15:00 access.log.4
>>> -rw-r----- 1 squid squid 379977665 Aug 16 12:00 access.log.5
>>> -rw-r----- 1 squid squid 348474013 Aug 16 09:00 access.log.6
>>> -rw-r----- 1 squid squid 367307983 Aug 16 06:00 access.log.7
>>> -rw-r----- 1 squid squid 663904388 Aug 16 03:00 access.log.8
>>> -rw-r----- 1 squid squid 735110835 Aug 16 00:00 access.log.9
>>> -rw-r----- 1 squid squid 36715761664 Aug 17 16:01 cache.log
>>> -rw-r----- 1 squid squid 14262776941 Aug 17 03:00 cache.log.0
>>> -rw-r----- 1 squid squid 955445 Aug 17 00:00 cache.log.1
>>> -rw-r----- 1 squid squid 748262 Aug 16 21:00 cache.log.2
>>> -rw-r----- 1 squid squid 1069482 Aug 16 18:00 cache.log.3
>>> -rw-r----- 1 squid squid 698758 Aug 16 15:00 cache.log.4
>>> -rw-r----- 1 squid squid 497547 Aug 16 11:59 cache.log.5
>>> -rw-r----- 1 squid squid 271153 Aug 16 08:59 cache.log.6
>>> -rw-r----- 1 squid squid 355351 Aug 16 05:59 cache.log.7
>>> -rw-r----- 1 squid squid 759748 Aug 16 02:59 cache.log.8
>>> -rw-r----- 1 squid squid 1037802 Aug 15 23:59 cache.log.9
>>>
>>> As you can see, those "HUGE" cache log files were filled up in less
>>> than 12 hours. Opening them up, I find they were filled with the
>>> following lines, repeated over and over again:
>>>
>>> 2010/08/17 02:33:11| comm_accept: FD 28: (22) Invalid argument
>>> 2010/08/17 02:33:11| httpAccept: FD 28: accept failure: (22) Invalid argument
>>> 2010/08/17 02:33:11| comm_accept: FD 28: (22) Invalid argument
>>> 2010/08/17 02:33:11| httpAccept: FD 28: accept failure: (22) Invalid argument
>>> 2010/08/17 02:33:11| comm_accept: FD 28: (22) Invalid argument
>>> 2010/08/17 02:33:11| httpAccept: FD 28: accept failure: (22) Invalid argument
>>>
>>> And, that is the time from when it started. Is there any way to
>>> determine what is causing this?
>> Start with the Squid version and what settings your http_port are configured with.
>>
>> Then we check for what it means. Google locates several requests, strangely around August each year for the last few.
>>
>> Someone describes it thus: "The problem is however elsewhere, since it somewhere fails to obtain a socket (or has its socket destroyed by the kernel somehow) so that when it calls accept(2) on the socket it's not a socket any more."
>>
>> Might be a SYN-flood DoS by that description. But your OS security should be catching such a thing before it gets near any internal software like Squid.
>>
>
> Squid 2.7STABLE9
> http_port 3128 transparent
>
> iptables is running, but no rules are there.

One interesting thing I note is that you have your logs rotated every 3
hours. Except during the event. The Squid problem seems to be that
something (possibly the accepting) blocked the rotation from happening
several times.

FWIW; Squid has a connection limiter to prevent more connections being
opened than there are available FD resource on the system. There is an
outside chance this limiter paused a great number of sudden connections
which died off. Which at a later point got 'kicked' for acceptance but
were already gone. Generating that error.

Might be something else. I've cc'd Henrik who still maintains 2.7.

The 40GB size of logs seems to point at a DoS behind it all anyway.

Meanwhile if its still going I suggest finding some SYN-flood protection
rules and adding them to iptables. See what changes with that in place.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.6
   Beta testers wanted for 3.2.0.1