RE: [squid-users] Looking for web usage reporting solution

> -----Original Message-----
> Subject: Re: [squid-users] Looking for web usage reporting solution

> On Fri, Nov 13, 2009 at 11:31 AM, Aaron Spurlock
> > I am looking for a web usage reporting solution that can run via
> sniffing or from a mirror port on a switch. I envision this solution
> would simply log each URL request it sees and allow reports to be
> generated on web sites that internal users have gone to.  I've searched
> high and low, but cannot find a "ready-made" solution, so I'm looking
> to put it together myself.
> >
> > Most people/posts suggest using squid/squidgard/dan's guardian, but
> it appears to me that is only an inline solution, and I would prefer a
> sniffing solution for safety (if machine crashes, it doesn't take down
> Internet). In that sense, it would work a lot like websense, but
> without the blocking, only reporting.
> >
> > From a high-level pseudo-code standpoint, it would simply sniff all
> traffic, and when it sees a packet requesting a webpage, it parses it
> and dumps these results into a database:
> >
> > -Date
> > -Time
> > -Source IP
> > -Dest IP
> > -URL requested
> > -FQDN portion of web request - IE: if request was for
> >  http://www.microsoft.com/windows/server/2003, it records only
> >  www.microsoft.com here
> > -domain portion of web request - only microsoft.com in above example
> >
> > Using this data, I can then produce reports for the client on who
> went where when.... Personally, I thought this would be a great program
> for open source, but I can't find anything like this already out
> there!!! It seems like kind of a mix between Squid, NTOP and Snort...
> >
> > Thanks for any thoughts on this project!
> >
>
> Sounds like you've got it pretty much worked out...not sure what the
> question is? Sniffing for HTTP traffic should work fine, then you just
> need to parse the traffic. I would probably just use iptables to sniff
> for traffic on port 80 and 8080. Of course, you can't sniff SSL
> communications, but there's really no solution to that unless you can
> break the encryption (which you can't). I guess this is technically
> still an inline solution, but short of actually buying a hardware
> packet sniffer, I'm not sure that there is any "out of line" solution.

LOL, just saw your post about the reply-all issue...I almost did that with this message! Anyway, back to the regularly scheduled program.

Can I use IPtables to log those packets to a file that I could then parse? I've been thinking about doing it with tcpdump, but I'm not sure what kind of output I'd get!
Received on Fri Nov 13 2009 - 17:00:33 MST