Re: Release Schedule

On 21 Aug 2001 18:49:25 -0500, Joe Cooper wrote:
> Robert Collins wrote:
>
> So far so good. I'm running it at 50 reqs/sec (on very modest hardware
> that maxes at 70 on a recent aufs squid), and I'm pretty sure it's
> catching everything. My only test for this at the moment is that I see
> all of /my/ browser requests being logged accurately--I can't be sure
> the polygraph requests are all being logged until I do a line count and
> compare that to the number of requests sent.

Excellent.

> It doesn't appear to be slowing Squid in any major way, though I'll have
> to give it a harder push to know that for sure (I didn't compile in
> aufs, wanting to keep the first run as simple as possible).
>
> It certainly compiled without complaint and started up the logd daemon
> without a hitch (two of them, actually...is that normal? for cache.log
> and access.log, I presume? store.log is disabled). It also doesn't
> seem to be eating an inordinate amount of CPU (i.e. almost none, as I
> would expect from logging). Rotate works fine as well.

Yes, one helper per log file. This is both for simplicity (no need to
identifty the log in the passed information) and flexability (can have a
_different_ helper for each log file.

> After this has run for about 2 hours, I'll do a line count comparison
> and then run it with aufs just to be sure it can handle a real load.
>
> (I like this testing cool new stuff and having it Just Work.

Doh! :]

> So, this
> means rotate requests will now effect the log daemon rather than the

Yes. In fact rotate no longer restarts the helper process's (other than
the logd's). This _could be_ optional for non-ntlm-helper process's, but
I actually quite like the idea of making rotate as fast as possible.
 
> running Squid? And does this also mean that we could do something
> creepy like create a log analyser daemon that does analyses in realtime?)

Absolutely. You'd want it multi-threaded so as to prevent the pipe ever
filling up - or squid performance would take a hit. (Log writes are
blocking I/O by design - although I'm willing to examine non-blocking).

All you'd need to do is have a replacement logd for (say) cache.log that
writes the log file and looks for warning messages, sending off an
email, or an alert to netsaint or whatever.

Likewise for access.log, the helper could
1) write to MYSQL,
2) check for problematic activity - say default.ida and
3) (say) update a mrtg database for graphing.

Of course, tail can do much of that, but not as cleanly, and not directly to the database.

Rob
Received on Tue Aug 21 2001 - 17:51:47 MDT