Re: cachemgr output normalization

On Sat, Oct 29, 2011 at 3:41 AM, Amos Jeffries <squid3_at_treenet.co.nz> wrote:
> On 29/10/11 08:39, Alex Rousskov wrote:
>>
>> On 10/28/2011 12:36 PM, Kinkie wrote:
>>>
>>> Hi all,
>>>   Now that cachemgr happily responds to proper http requests, I've
>>> started toying with a browser-based, all-javascript/DHTML cachemgr.cgi
>>> replacement.
>>> I have a first beta available in launchpad at
>>> lp:~kinkie/squid/cachemgr-js.
>>
>> Can you start it somewhere so that we can actually see it in action?

Ok. Will do so in the next few days.

>>> It currently requires help from squid
>>> via a rproxy setup to serve the two HTML and one javascript file, but
>>> if we can have squid serve them internally, it'll be a no-brainer for
>>> single-instance monitoring.
>>
>> Will this be a part of Squid source tarball distribution? It may make
>> sense to keep it separate because it may require separate translations,
>> have a separate release cycle, etc.
>>
>
> The bundle tarball seems to have turned into a 'suite' of things long ago.
> Its up to teh distros hpw they split and install the separate tools. Many
> for example drop the cache utilities and announce tools (maybe not even
> noticing they are packaged), bundle the squidclient and CGI separately for
> install, and group the helpers and squid core binaries together as a set.
>
> This seems to work well.
>
> If we like, we can easily package a separate tarball in parallel, like the
> langpack is done.
>
>>
>>> There's a few open points which need a bit of thinking though:
>>> 1- what is the best way to serve a few static html and javascript files?
>>
>> Same way we serve icons, I guess. IIRC, Henrik has summarized how
>> Squid-as-web-server should [not] work some time ago, but those changes
>> are probably outside this project scope so I would just reuse icons code
>> for now.
>
> Yes. I still have the internal-server branch almost completed. Just needs to
> be updated to the latest sources and re-tested, re-audited.
>
> If the JS files are installed and loaded from the /var/www/squid area same
> as the icons it will be easier to both get out immeditaely and integrate
> with internal-server when we gets to finishing it.

If the installation is connected to the Internet, we don't even need
to bundle the .js files, all that would be needed is a 1Kb-ish html
launcher.

>> If this is difficult, for any reason, just put them somewhere on
>> squid-cache.org and hard-code the URLs in your Javascript, for now.
>>
>
> At first glance I think serving the JS page loader and a stub HTML page to
> run it as the http://*/squid-internal-manager/ 'index' URL is probably the
> user-friendliest way to roll it out.
>
>
> I think I spoke to you a while ago about implementing the /menu action with
> TXT + HTML format. But got stuck at the problem of generating right TXT in
> the workers or the HTTP headers output. And report aggregating duplication
> of the page.

I'm aiming for something simpler now: rework the text output so that
it is still human-readable but a bit more machine-parser-friendly so
that transcoding (mostly transcoding tables) is simpler.

The way I'd think to do the separation between data collection and
rendering is not really to use ASN.1 but structs. If multiple types
need to be supported it'll just have to be unions with a type
identifier attached. But in order to get there we need to reorganize
the actions' contents: most actions return two or more tables, this
needs to be reduced to one table per action, or it won't be renderable
by snmp.

>>> 2- does cachemgr get engaged only via GET method or can we have it
>>> also answer to POST requests? The reason is that GET requests in
>>> javascript are subject to a same-origin policy, while POST are not. It
>>> would allow for multi-server monitoring and it would make point 1 a
>>> nice-to-have and not a requirement
>>
>> The difference is minor from Squid code point of view so we can support
>> both, eventually. IIRC, we do that in Co-Advisor, with little code (that
>> we would be happy to copy to Squid).
>
> I looked at this when advising Arthur on the shell backend handling. He
> needs POST as well.
>
> Cachemgr takes any method, but filtered GET for all the actions so far
> implemente. I think we can easily add any method we want for new actions. Or
> adjust the existing actions to handle non-GET.

The actions don't really need to know for now: we can safely ignore
body-supplied parameters. If we don't, the adaptation should not be
done in the actions, that's too much duplication.

>>> 3- we need to make the output from cachemgr handlers follow some
>>> common guidelines.
>>
>> Sure.
>>
>> How do you want to post-process that output in Javascript? Some
>> find-and-replace commands using regular expressions? Is it very
>> difficult to have action-specific post-processing?

Find-and-replace, inspired to cachemgr.cgi.

> The JS should make use of the format parameter to do format-specific
> processing. Like CGI does action-specific processing right now.

Current Js is limited to browser-based rendering. Think of it as an
alternative to cachemgr.cgi, not as a self-contained tool.

> The existing TXT format can be dumped wholesale straight into PRE tags and
> the actions updated to more useful formats one by one. This resolves the
> back-compat and third-party script problems as text can be phased
> out/upgraded over the long term.
>
>  * For tabular data CSV, TSV, HTML, XML formats are all commonly supported
> and useful under different scenarios.
>  * For list data XML or HTML is probably best.
>  * For key=value data XML, HTML, or TXT is probably best.

That's the point: key = value data is really a two-columns table.
Which is rendered as key = value probably due to historic reasons.
That's exactly what I'd like to change.

>>> This poses a problem of
>>> compatibility with third-party software. We can either have a
>>> transition phase where we duplicate actions, or we can just decide
>>> that we don't have the resources to care, and we just warn the authors
>>> that we know of about our intentions so that they have time to adapt.
>>
>> Indeed. Perhaps this should be discussed on squid-users as well.
>>
>
> This is where I greatly favour the format upgrade. We can both warn them the
> TXT is going to become human-only view, and that there is more efficient
> machine-readable formats available for their immediate use.

That's for phase-two. Phase-one would be to make txt more machine-friendly.

>>> This is also in my opinion a prerequisite step to support multiple
>>> output formats in cachemgr.
>>
>> Is there consensus that Squid itself should support multiple output
>> formats? I kind of doubt it is the right thing to do in general. If
>> Squid outputs easy-to-parse, consistent data, other applications can
>> post-process and beautify it in many different ways.
>>
>
> From me yes.
>
> I also see and agree that opening the door to anything at all is a bad idea.
> We can and should provide a small restricted set of suitable formats for the
> data based on its type table/list etc. Such as the ones I mention up above.

I'd add JSON, but apart from that yes.

>>> I'm willing to spend the time to do this if we agree that it should be
>>> done.
>>
>> Yes, the output should be standardized.
>>
>
> Agreed.
>
> The first step though, is standardizing and checking that all actions are
> upgraded to use a single internal-only format for data transfer from workers
> to somewhere for assembly/aggregation and transformation the other formats
> as needed. ASN was proposed earlier. Yes?

That's second step :)

First step, standardize textual representation in output with current
infrastructure
Second step, restructure actions (one table per action)
Third step, separate internal and extenral representation
Fourth step, add more formats.

-- 
    /kinkie