Re: userlogging module?

----- Original Message -----
From: "Adrian Chadd" <adrian@creative.net.au>
To: "Robert Collins" <robert.collins@itdomain.com.au>
Cc: <squid-dev@squid-cache.org>
Sent: Friday, February 16, 2001 8:52 AM
Subject: Re: userlogging module?

> On Fri, Feb 16, 2001, Robert Collins wrote:
> > The MySQL acl patch provided here perviously might provide some inspiration, or even details :]
>
> I should look at it.

So should I :]. I'm hoping to this weekend..

> > If you mean _just_ the actual write to disk bit, won't we have a terribly small helper?
>
> Well, depends if the helper is going to log to a database.
>

You end up with a single INSERT statement (for SQL based databases) - that's going to be pretty small :]. Point taken however.

> > I don't see any benefit in breaking any more of the access log code out to a helper - why not just have a queue of pending log
> > entries that are only cbLocked by the logging code. Then the logging code runs a pair of async i/o functions (similar to
> > clientsendmoredata & clientwritecomplete) that call the low level async functions
>
> It depends if the db code thats out there lets you break the db accesses
> out into async chunks focused around poll/select.
>
> If not, then you could possibly risk a stall if the sql functions don't
> return immediately.
>
> > To add in a database we write a new set of low level async functions for that db, and configure them via squid.conf.

So we have a set of low level write functions that have the same API but send the formatted log entries to helpers. Perhaps a
trivial wrapper around helper.c that always opens with 1 helper & restarts the log helper as needed.
Then we can swap that set of functions for ones that wrap syslog | standard files as today | ...

Not much harder, and we won't introduce the headaches of an external helper that can be killed (leaving squid unlogged!) for the
standard case. (We won't even have to #ifdef that ).

> > The reason I'm against the external helper is a) that the operation being effected is so trivial, and b) we already have most of
an
> > async framework in squid - lets use it and c) the volume of access entries is going to very high for a single helper (and to
keep it
> > serialised you can only have a single helper ?)
>
> We have most of an async framework in squid, its just not used for
> the disk ops (in disk.c) anymore.

Hmm. How hard would it be to introduce async disk ops framework (falling back to blocking behind the scenes as needed per platform)
?. Not suggesting this for use from the object store, but for re-reading config files | logging | reading internal objects |...
And hopefully generic enough that the same API will lend itself to reading|writing to things like db's RPC's IPC's ? (ie
asyncopen(type,path,parameters)) can at runtime backend into type=stdio type=mysql etc

> Yes, I realise that its going to tax whatever method is used for
> IPC as its going to be high throughput. I suggested it because
> an external process(es) allows log agents to be written that
> don't lend themselves towards being IO or pthreaded (eg perhaps
> mysql, but I'm sure you can think of others..)
>
> Its just an idea. :-)
>

Yup.. and just comments from me :]

Does the mix suggested above (call it take #2) make more sense? Abstract out the log write to an async logic with internal queue
etc.
Write an initial low level async io wrapper for the existing log file writes. (Call this goal 1 - async logging).
Write either an internal low level async wrapper for the rff database || an internal low level async wrapper for running an external
helper and a helper that writes to the database. (call this goal 2 - async database logging).

That way we gain a little choice, no #ifdef's to keep the current compact layout, and the potential difficulties in keeping the core
non-blocking are resolved.

Rob
Received on Thu Feb 15 2001 - 15:10:31 MST