Re: [squid-users] script conv httpd log fmt -> squid native

On Sat, 17 May 2003 13:01:31 -0700, you wrote:
>I saw an awk script for squid native -> httpd native log format.
>Does anyone have a similar script off hand to do the opposite?
>
>I usually like httpd format logs since _I_ can read the
>logs directly, whereas, the squid native log format isn't so
>easily human readable, but I want to run a report generator
>that only groks squid format (yeah, I know there's a bunch that
>only grok httpd format as well, but they didn't seem as directly
>appropriate to what I wanted).

A google groups search for awk+convert came up with two useful hits
from the mailing list, the post by Peter Smith looks good:
http://groups.google.com/groups?as_q=awk%20convert&as_ugroup=mailing.unix.squid-users

Furthermore, http://www.squid-cache.org/Scripts/ has a link to a perl
script that does that. The links to Martin's pwebstats and related
squid2common.pl script is out of date but they redirect you to his own
website. Here is a link directly to his perl program that does what
you want (albeit not in awk):
http://martin.gleeson.com/pwebstats/squid2common.pl

As to your original assertion about making the logs "easily readable,"
the only thing cryptic is the UTCseconds timestamp which is easy to
filter. The other values are just in a different order. To read the
logs I just pipe whatever section of the logs I am extracting (usually
a tail -500 or similar) to a perl one liner that converts the date.
Actually I created a simple alias (tsquid for tail squid) that does
this for me. The FAQ (
http://www.squid-cache.org/Doc/FAQ/FAQ-6.html#ss6.6 ) has an example:
        #! /usr/bin/perl -p
        s/^\d+\.\d+/localtime $&/e;
I'm not sure if that is what I use or something similar, as I am not
at work today. Personally since many tools work with squid's native
format, the additional data might be needed for troubleshooting, and I
can "on-the-fly" "fix" the date, I don't think it is worth it to not
log in the native format. The above linke to section 6.6 gives more
detail about the access log.

Lastly, not sure what kind of report generator you are wanting or
since first you want httpd native format logs but then want a script
that "only groks squid format" but I find Calamaris, also linked from
the above squid Scripts url, very useful. The line I like to use is
"cat access.log | calamaris -a -O -c -U M > squidrep.out" as it gives
all reports (-a), sorts by size (-O), is case insesitive (-c), and
switches from K to MB, making for cleaner tables. What I liked about
Calamaris is it has few dependencies. I just downloaded it, ran the
script, put it in cron, and voila I had handy daily reports.

anyhow do whatever you like - I hope some of the above proves useful,

Adam
Received on Sat May 17 2003 - 15:19:41 MDT