[RFC] shutdown stages

At present shutdown happens over two stages. With server socket and auth
shutdown immediately and everything else 30 seconds or so later.

We have a number of bugs open about this, ranging from complaints that
idle squid still take a full 30sec shutdown when they should fast-track
the process and close quickly. Through to crashes when helper shutdown
or swap.state dump are started too late and not given enough time to
complete.

I would like to split that process down into several steps with as much
stuff being shut down as fast as reasonable.

stage 1:
  * TIMEOUT: shutdown_lifetime/2 converted to a timeout, to trigger
stage-2 early if needed.
  * close ICP, HTCP, HTTP, HTTPS ports, but leave DNS open
  * prevent keep-alive, and close all connections on completion
  * disable the background cyclic events (ie garbage collection). we can
trigger these "manually" as part of the shutdown if they are needed at
certain points.
  * begin swap.state dump of index
  * call stage 2 immediately IF no active client connections AND stage<2.

stage 2:
  * TIMEOUT: shutdown_lifetime/2 converted to a timeout, to trigger
stage-3 early if needed.
  * disable new reads.
  * disconnect delay pools. push any buffered information out to clients
at full speed and terminate. (bad idea? good idea?)
  * call stage 3 immediately IF no active client connections AND stage<3.

stage 3:
  * terminate remaining incomplete active clients using their close
handlers.
  * loop until async call queue is empty.
  * skip to stage N immediately.

stage N:
  * terminate the event loop and exit.

Attached is a proposed patch which alters shutting_down into an
incremental stage counter and alters the documentation. But does not
alter the fundamentals of the current state.

I believe we can easily add several shutdown functions to the
SignalEngine. Or move these to the cache manager as series of async
calls with the SignalEngine simply triggering the cache manager shutdown
event.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.14
   Beta testers wanted for 3.2.0.10