threading (Re: ideas)

Squid has reached a certain point of high stability and SMPs a certain
low level of price where it might be worth taking the plunge and
adding threading, at least on an experimental basis, but it is
important that it be done cautiously.

The highest cost of a threaded implementation lies in the
development penalties and increased bugginess.

I have had experience debugging fully threaded middleware, Isis
Distributed Systems' Isis Distributed Toolkit. The threading design
was pretty good, and yet it was a vast source of continuous trouble.
Most of the hardest-to-debug conditions arose from threading -
deadlocks, race conditions, etc.. Threading problems tend to be the
hardest problems to debug because of the nondeterminism involved and
because reasoning about multithread conditions is far harder than
reasoning about serial conditions.

> Do you remember the discussion we had earlier about breaking Squid up in
> various separate components/processes:
>
> * Connection manager(s)
> . . .

There is an idea going around, a bit of a virus, that one can achieve
higher-quality code by making threading/execution boundaries
synonymous with abstraction module boundaries.

For many years I tried to see this in action in code I wrote and dealt
with, going all the way back to my old Amiga 2000, with its
multithreaded OS based on Tripos, with drivers and fs' encapsulated as
servers.

In fact, such code always turned out to be harder to write and
maintain. It is actually the least bit easier, in a dismayingly
superficial way, when sketching out designs in boxes and thinking
about said designs. The boxes you draw at this stage are accurate
both for your abstraction layering and processing breakdowns.

But coding and debugging are far more time-consuming than design.
Once you get much into that stage, you understand your design. And
you begin to pay. For openers, your code suddenly became
substantially more complicated, because you need to add rpc stubs and
interfaces. Even if you have a tool to help you, you often find that
you have added 30-50%, even 100% complexity to your original module.

Once you have coded it, though, the nightmare is about to begin. Many
software engineering texts will tell you that debugging protocols is
more difficult than debugging most conventional algorithms. They're
right. You'll see it yourself. And debugging synchronization
problems is far harder still, by an order of magnitude. You just
unleashed both these kinds of bugs upon your poor, innocent code.

If you read the BSD daemon book (mine is in a box somewhere, or I'd
quote it at this juncture), you find that classic BSD minimizes
locking and multithreading precisely because its authors had negative
experience with synchronization bugs.

OK, so what is the 'right' way to do threading? Miminize its impact.
Minimize the number of locks (rpc calls count as locks) and the
numbers of times that a context switch will happen in the common case.
Use locking rather than rpc calls. Try to keep processing of a given
system 'request' in one thread of control.

In the case of Squid, I would suggest processing requests very nearly
as much as is already done in a single thread of control, and adding
thread-conscious code at three points: request dispatch from
comm_select(), storeGet()/storeLockEntry(), and where diskio threads
off now. Request dispatch should probably be the only place where
threads are created, and only if 'necessary'. This is the minimum
needed to take advantage of SMP and processing requests while waiting
for disk.

Those incremental measures alone will provide no shortage of debugging
challenges, but it's probably about as good as you're going to get.

No doubt you would have to change some data structures to be
associated either with a request or with a storeEntry.

                                                Jon
Received on Wed Jun 14 2000 - 13:31:58 MDT