RE: updating squidng ?

> .. whats wrong with running a resolver on your localhost which handles
> caching? :-)

Because if you have in-process caching (as it is now and how it would
be with shmem), you don't have to pause the request handling
while looking up for an entry, plus you save context switches
(unless you want to give the resolver a processor all by itself).
This is, for instance, the strongest reason why I and Robert agreed
to move NTLM credentials caching inside squid, rather than in the
helper processes as it was up to two days ago.

> > > Here's the cute bit - either (a) we can have a socket
> > > redirector process
> > > sitting at the front of the X squids, or (b) we can modify
> > > the ipf/ipfw/
> > > ipchains/whatever code to do load-balancing redirection.
> >
> > (a) has some drawbacks. Either the front-end process does HTTP
> > header parsing, or we won't be able to correctly log accesses.
> > (b) is heavily OS dependent, I fear.
>
> Nothing wrong with heavily OS dependant if we give people the
> option of being able to do it.

It makes code management a tad more difficoult, because then you
start to have low-level options and weird bugs might crop up.
Besides that, not much.

> > Why can't just the OS do the load-balancing for us? We can't
> > sit on accept(), but there should be some other way to just
> > let the OS do the job for us.
>
> as far as I know under UNIX, we have to sit on accept(). -)

How so? In the main loop, we do an accept with a zero-timeout
(I believe that is how it's done now). The first process to
get to that point will accept the connection.. Traffic might be
a bit bursty, but it seems simple enough. Should even avoid
the thundering-herd problem apache used to have, since we're
not actually sitting in accept(). Under this approach, we can't
sit on select()/poll() forever, but I doubt we're doing that
even now (too lazy to read the code).

> > > Depends how it is implemented. See, if we use aio, we can
> > > schedule reads
> > > of the file from multiple processes and have the reads block
> > > whilst the
> > > object is being written. Squid can't do that now due to how
> > > the storage
> > > manager is written but I don't imagine it'd be that hard.
> >
> > Can't we just expand on the diskd concept?
> > Idea: the parent process forks one diskd for each storage, and
> > passes shmem IDs to all children. Then accesses are done
> > through this helper processes farm.
> >
> > The idea would be to delegate much of the storage management to
> > this "storage manager" process. If possible.
>
> Yes, this is right. But say you have a kernel-mode cache FS. You
> want the flexibility to use this directly ..

It depends on the features such a FS has. I have seen no specs yet.
Besides that, it's just a matter to see if we can modularize
this cleanly enough.

> This is what I'm trying to do. Make squid flexible enough to pick
> and choose how you want to do things, rather than being locked down
> like we are right now.

You can't get infinite flexibility, but having more options is
of course good (sorry for the obviety of this all).

-- 
	/kinkie