RE: Record all traffic option?

On Fri, 2003-02-07 at 05:49, LA Walsh wrote:

> > Well, first thoughts:
> >
> > 1) if you want to redirect everything coming to from squid,
> > use network
> > layer interception. It exists, and works. It has issues - but no
> more
> > than 100% alteration via squid itself.
> ---
> One of the issues, is it makes it harder to session analysis or am I
> missing something? It's also an easy of use issue.

Well, squid doesn't do sessions, so thats really orthogonal. For ease of
use, ngrep is pretty darn good.

> icap? -- maybe something like that...I never heard of it before
> your note, but looking at it, It seems like it would do most of #4.
> Are there plans to mainline the patch? Conceptually, it appears to
> provide much of what I was looking for...not sure how obvious it
> is to setup yet.

Yes, there are plans. Hopefully the authors will be in a position for
merges to begin post-3.0.

> > Perhaps if you were do describe what you want to *achieve* we
> > could more accurately critique the design.
>
> Good point.
>
> 1) allow recording. I have one or more devices, or programs that
> want to speak to a webserver. I don't trust them. I want
> to audit everything
> they say to each other. Policy or design is that the programs aren't
> using encoded information (or https). I don't want to see
> "ACK"s and "SYN"s,
> Just "he said/she said" (well it and it, but nevermind).

ngrep again is the right tool for this.

> 2) I want to be able to monitor the recording in real time and
> potentially block any request to a site I don't recognize.
> I may want to edit out selective HTML elements: embedded objects,
> scripts, images, site references.
> Under '2', I'd like to see the ability to use something like
> junkbuster but the ultimate goal would be to gain efficiency
> over using multiple TCP sockets. I don't know if local pipes are more
> efficient than TCP sockets
> or not, but the concept of a pipe could possibly be enhanced to
> provide
> a high-speed message transfer using shared memory between processes to
> lower overhead.

This has many concepts in it. Let me separate them out into a couple of
areas:
a) implementation details - sockets / pipes / shared memory.
b) architecture - integrate with squid / be part of squid / be
completely separate from squid.

Now, a) really doesn't matter until b) is sorted out. Monitoring the
traffic in real time and editing the content is a form of in-line
content transformation, a long running interest of mine. icap is an
outofprocess implementation of the same. squid-3.0 is nearly ready for
an implementation of generic content transformations - but not quite.
squid-3.1 should have all the kinks ironed out to allow the TE
implementation I did with squid-2 to be integrated without the semantic
problems it had. And when thats in, other applications can be done.

Now there are two issues:
*) RT blocking of URL's.
*) inline content alteration.

The first requires an interface to add dynamic ACL entries. That applies
whether the architecture is part of squid or integrates with squid. For
something completely separate from squid, it doesn't need this.
The second I've addressed above for in-squid implementation, and for
integrates with squid implementation. For completely separate from squid
implementations (i.e. dansguardian / junkbuster) there is no issue.

IMO a squid-integrated / in-squid capability would be nice. For ease of
use, a upstream proxy implementation + IDS with active capabilities is
probably the easiest to do today.

> Eventually, latency and efficiency could be concerns.
>
> Somehow I just like the idea of extending squid's logging (seems
> conceptually simple) and adding more "plugin" points. Conceptually
> all
> those seem simple (perhaps the icap patch does exactly the latter). I
> know there are complex ways to do what I want, but my
> preference would just
> be to use a different squid log file or plugin reference. Seems to
> be, conceptually, simple.
>

Conceptually simply, practically a *lot* harder. Snort is built to do
what it does well, and upstream proxies are able to be very small and
compact non-caching devices. This is also conceptually simple, and has
the advantage of clean functionality separation.

Squid logs are really not what you are talking about. They are a
historical view of request *metadata*. To log everything via that
interface will create race conditions where Gb's of memory will be
required.

> Think of squid as a form of a high level TCP language.

No thanks. Squid is a HTTP implementor, not TCP. Squid's core can
operate on anything that can be transformed to HTTP semantics - which
means an IPX socket client could work (for instance).

Rob

-- 
GPG key available at: <http://users.bigpond.net.au/robertc/keys.txt>.