Re: async-io for 2.4

Andres Kroonmaa wrote:
>
> On 3 Nov 2000, at 5:10, Joe Cooper <joe@swelltech.com> wrote:
>
> > > Will commit the code to 2.4 in a few days unless someone has any reason
> > > not to.
> >
> > I'm not going to argue that it shouldn't go in because this is much more
> > stable than the current 2.3 async code (which is probably unusable), but
>
> hmm, I'm using async on 2.3 and am pretty happy. no problems and its faster
> than without. Although I couldn't use it on Linux when I tested an alpha box.
> Linux threads are no good.

Haven't noticed severe degradation in hit ratio?

> > (the box is freezing due to a ReiserFS deadlock condition...but I think
> > it is triggered by a Squid issue, since I don't see this problem with
> > 2.2STABLE5+hno--I don't think...I've never run this set of benchmarks on
>
> rule of thumb: no application is to be able to take whole system down.
> if that happens, then the OS has a big problem. If reiserFS is running
> in kernel space, then it may be part of the problem, squid could only
> be unmasking it.

ReiserFS is running in kernel space. And I do suspect the deadlock is a
ReiserFS problem.

> > Anyway...Here's what happens:
> >
> > 2000/11/03 01:19:46| storeAufsOpenDone: (2) No such file or directory
> > 2000/11/03 01:19:46| /cache1/01/5F/00015F6A
>
> sounds like some race after unlink, or simply lost file. Happens here
> too from time to time. Are you sure this is related?

Nope.

> > 2000/11/03 01:20:15| comm_poll: poll failure: (12) Cannot allocate
> > 2000/11/03 01:20:15| Select loop Error. Retry 1
> > 2000/11/03 01:20:15| comm_poll: poll failure: (12) Cannot allocate
>
> > 2000/11/03 01:20:15| Select loop Error. Retry 2
> > 2000/11/03 04:14:49| comm_poll: poll failure: (12) Cannot allocate
>
> interesting, 3 hours later?

Yeah...Confused me too. But here's my theory...the deadlock froze Squid
in mid-action. After I consoled in and got the box out of deadlock
Squid went right on with what it was doing. Which was writing poll
failure errors. Then again...maybe that's not at all what happened. I
wasn't watching the box closely when it died. I'm going to work on it
some more tomorrow, after I've had some sleep (it's 6AM here..time for
bed).
 
> > 2000/11/03 04:14:49| comm_poll: poll failure: (12) Cannot allocate
> > 2000/11/03 04:14:49| Select loop Error. Retry 10
> > FATAL: Select Loop failed!
> >
> > ReiserFS also reports memory allocation failures, and triggers a
> > deadlock condition (which I'll report to the ReiserFS guys...Chris Mason
> > over there is a hero on these kinds of problems). Now...memory is not
> > filled when this condition occurs. There is some stuff in swap (136MB
> > worth, actually) but it was put there much earlier and didn't cause
> > problems. CPU saturation seems to be the culprit here.
> >
> > Here is a snippet of 'vmstat 1' as the deadlock is happening (when
> > system usage goes to 100% is when you know the box is locked).
> >
> > procs memory swap io
> > system cpu
> > r b w swpd free buff cache si so bi bo in cs us sy id
> > 7 0 0 136 2864 115564 57664 0 0 0 0 738 22634 16 84 0
> > 10 0 0 136 2824 115564 57664 0 0 4 0 822 504 0 100 0
> > 17 0 0 136 2796 115564 57664 0 0 8 0 508 283 1 99 0
> > 18 0 1 136 2792 115564 57664 0 0 2 690 347 164 0 100 0
>
> I suppose running (r) processes are actually linux threads? As they are
> not blocking nor waiting, they most probably are spinning on a mutex.
> "mpstat 1" could add some to the picture.

Don't have mpstat. Not running SMP, what does mpstat tell us that
vmstat doesn't for a single CPU?

> sounds like a bug or design error in kernel or threads library.
> Interrupts and signals can cause mutex spins, so if there is insanely
> high interrupt rate, this might hint on hardware issue. SCSI devices
> are notorious in failing to properly implement tagged queueing, for eg.
> try disabling this.

Not running SCSI. It is running on two IDE disks, with the latest
stable Linux kernel and IDE patchset. This combination of hardware and
software has proven very nearly rock solid under Squid 2.2.STABLE5+hno
using Async i/o.

I don't think it's hardware issue, but I won't rule it out without
further testing.

> error message about memory shortage seems either mswindows like
> "general error", ie. misleading, or could be related to some kernel
> space shortage, like being too fragmented to accommodate needed
> sequential chunk, or something like that. Threads may be blocked
> onto mutex, and kernel is desperately trying to schedule threads to
> let them make progress, in hope that some resource is released.

I agree that it's probably not actually a memory issue. This type of
error and deadlock has popped up in the past with ReiserFS, but was
fixed several versions ago...If it is another ReiserFS issue, then it
will get fixed pretty quickly, I'm confident of that.

> in any case, to me it looks more like kernel/OS problem, rather than
> Squid problem.

Possibly.

> CPU burn seems to be the result of some problem, as is also reiserfs
> malloc failures.
>
> > failing, I'm not sure. Also note that the box doesn't gradually get
> > overloaded as in butterfly or in the old 2.2 async...this box falls over
> > within seconds of the response time climbing over 2.5 seconds (while the
>
> brickwall effect. Seen such before on Linux with weird loads.

I haven't. But as I mentioned, I haven't tried this set of benchmarks
on the 2.2.STABLE5+hno Squid that our boxes ship with yet...but I think
(in fact I strongly believe) that in the weeks of heavy stress testing
I've done on the old Squid I would have run into any problems that still
existed.

Will find out more tomorrow, after another round of benchmarks.
                                  --
                     Joe Cooper <joe@swelltech.com>
                 Affordable Web Caching Proxy Appliances
                        http://www.swelltech.com
Received on Fri Nov 03 2000 - 05:15:08 MST