importing logfiles into MySQL

There's tens of ways of importing Squid logfiles into MySQL (mostly as part
of a web utility to analyse/browse/report on Squid use.)

I'm writing one for a client and I figure the best thing we can do is
include a "Squid format" MySQL database schema and logfile parser which
others can then write their applications to use. It may hopefully mean
that clients can have one database backend for their logs and choose
a reporting program to use. Or, in my particular case, they can suck
the files over to a central server nightly, import them into MySQL
and give out ODBC access to Crystal Reports.

I've put the preliminary code up at
http://www.squid-cache.org/~adrian/logfile/ .

Its ugly and it mostly works, but splitting the Squid log line up is sometimes
done "wrong". It is however pretty nippy at importing logfiles into MySQL.
There's not much in the way of indexing because, well, I haven't spent much
time on the web frontend side of things to know what needs indexing.
I've also given some thought to breaking out the URL field into a seperate table.
Maybe. (You can tell I'm not a full-time DBA.)

I'd like to get this included in the Squid-2 and Squid-3 scripts directory.

Comments?

Adrian
Received on Wed Mar 21 2007 - 03:38:15 MDT