Re: cachemgr output normalization

On 29/10/11 08:39, Alex Rousskov wrote:
> On 10/28/2011 12:36 PM, Kinkie wrote:
>> Hi all,
>> Now that cachemgr happily responds to proper http requests, I've
>> started toying with a browser-based, all-javascript/DHTML cachemgr.cgi
>> replacement.
>> I have a first beta available in launchpad at
>> lp:~kinkie/squid/cachemgr-js.
>
> Can you start it somewhere so that we can actually see it in action?
>
>
>> It currently requires help from squid
>> via a rproxy setup to serve the two HTML and one javascript file, but
>> if we can have squid serve them internally, it'll be a no-brainer for
>> single-instance monitoring.
>
> Will this be a part of Squid source tarball distribution? It may make
> sense to keep it separate because it may require separate translations,
> have a separate release cycle, etc.
>

The bundle tarball seems to have turned into a 'suite' of things long
ago. Its up to teh distros hpw they split and install the separate
tools. Many for example drop the cache utilities and announce tools
(maybe not even noticing they are packaged), bundle the squidclient and
CGI separately for install, and group the helpers and squid core
binaries together as a set.

This seems to work well.

If we like, we can easily package a separate tarball in parallel, like
the langpack is done.

>
>> There's a few open points which need a bit of thinking though:
>> 1- what is the best way to serve a few static html and javascript files?
>
> Same way we serve icons, I guess. IIRC, Henrik has summarized how
> Squid-as-web-server should [not] work some time ago, but those changes
> are probably outside this project scope so I would just reuse icons code
> for now.

Yes. I still have the internal-server branch almost completed. Just
needs to be updated to the latest sources and re-tested, re-audited.

If the JS files are installed and loaded from the /var/www/squid area
same as the icons it will be easier to both get out immeditaely and
integrate with internal-server when we gets to finishing it.

>
> If this is difficult, for any reason, just put them somewhere on
> squid-cache.org and hard-code the URLs in your Javascript, for now.
>

At first glance I think serving the JS page loader and a stub HTML page
to run it as the http://*/squid-internal-manager/ 'index' URL is
probably the user-friendliest way to roll it out.

I think I spoke to you a while ago about implementing the /menu action
with TXT + HTML format. But got stuck at the problem of generating right
TXT in the workers or the HTTP headers output. And report aggregating
duplication of the page.

>
>> 2- does cachemgr get engaged only via GET method or can we have it
>> also answer to POST requests? The reason is that GET requests in
>> javascript are subject to a same-origin policy, while POST are not. It
>> would allow for multi-server monitoring and it would make point 1 a
>> nice-to-have and not a requirement
>
> The difference is minor from Squid code point of view so we can support
> both, eventually. IIRC, we do that in Co-Advisor, with little code (that
> we would be happy to copy to Squid).

I looked at this when advising Arthur on the shell backend handling. He
needs POST as well.

Cachemgr takes any method, but filtered GET for all the actions so far
implemente. I think we can easily add any method we want for new
actions. Or adjust the existing actions to handle non-GET.

>
>
>> 3- we need to make the output from cachemgr handlers follow some
>> common guidelines.
>
> Sure.
>
> How do you want to post-process that output in Javascript? Some
> find-and-replace commands using regular expressions? Is it very
> difficult to have action-specific post-processing?
>

The JS should make use of the format parameter to do format-specific
processing. Like CGI does action-specific processing right now.

The existing TXT format can be dumped wholesale straight into PRE tags
and the actions updated to more useful formats one by one. This resolves
the back-compat and third-party script problems as text can be phased
out/upgraded over the long term.

  * For tabular data CSV, TSV, HTML, XML formats are all commonly
supported and useful under different scenarios.
  * For list data XML or HTML is probably best.
  * For key=value data XML, HTML, or TXT is probably best.

>
>> This poses a problem of
>> compatibility with third-party software. We can either have a
>> transition phase where we duplicate actions, or we can just decide
>> that we don't have the resources to care, and we just warn the authors
>> that we know of about our intentions so that they have time to adapt.
>
> Indeed. Perhaps this should be discussed on squid-users as well.
>

This is where I greatly favour the format upgrade. We can both warn them
the TXT is going to become human-only view, and that there is more
efficient machine-readable formats available for their immediate use.

>
>> This is also in my opinion a prerequisite step to support multiple
>> output formats in cachemgr.
>
> Is there consensus that Squid itself should support multiple output
> formats? I kind of doubt it is the right thing to do in general. If
> Squid outputs easy-to-parse, consistent data, other applications can
> post-process and beautify it in many different ways.
>

 From me yes.

I also see and agree that opening the door to anything at all is a bad
idea. We can and should provide a small restricted set of suitable
formats for the data based on its type table/list etc. Such as the ones
I mention up above.

>
>> I'm willing to spend the time to do this if we agree that it should be done.
>
> Yes, the output should be standardized.
>

Agreed.

The first step though, is standardizing and checking that all actions
are upgraded to use a single internal-only format for data transfer from
workers to somewhere for assembly/aggregation and transformation the
other formats as needed. ASN was proposed earlier. Yes?

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.16
   Beta testers wanted for 3.2.0.13