Re: [RFC] helper API

On 09/16/2010 11:03 PM, Amos Jeffries wrote:
> On 17/09/10 10:18, Alex Rousskov wrote:
>> On 09/16/2010 02:46 AM, Amos Jeffries wrote:
>>> After the helper C++ build migration we have a partial API for the
>>> helper tools. Some of them even make use of the #defined macros.
>>>
>>> I managed to bungle and put the old-style debug() definition for helpers
>>> into the libcompat. It's now clear that this would be better suited in
>>> the API for helpers and non-squid tools.
>>>
>>>
>>> What I'm looking at right now for the helpers is:
>>> * some wrapper for main() that calls out to user functions for handling
>>> a line received and processing command line options.
>> > * some definition of the user functions required to do the above.
>>
>> If I understand what a helper is, and helper API should not wrap main or
>> standardize command line options processing. Main should be left for the
>> helper to control and there are probably enough libraries out there for
>> options parsing.
>>
>> We can standardize a subset of common command line options, of course.
>>
>> If you want to offer a helper development framework with such things as
>> built-in I/O and reconfiguration handling, then things change. You may
>> indeed take main() ownership and just leave some places for users to
>> plug their code into. I would not classify that as API though because it
>> restricts the implementation way beyond a common API needs.
>>
>> And if we are doing a framework, we can remove current limitations
>> related to handling helper's standard input/output. We can just use
>> sockets...
>
> Yes. Framework sounds more like what I'm asking about.
>
> You mean provide a call to fetch a socket needed to send/receive from
> Squid. Hiding the current std* ones behind the call until some future
> re-work finds a need to change it?

If we are doing a Framework, then the user code should not know anything
about sockets or other low-level ways to communicate with Squid. The
code will integrate using something like noteSquidRequest() and
provideResponseToSquid() API. If we do not want to switch to sockets
now, we can do it later, transparently to the helper code.

>>> * some macros (as now) for performing OK/ERR etc feedback to squid.
>>> These take a char* parameter for additional key-pairs or messages.
>>
>> By "performing feedback", do you mean actually writing bytes to the
>> stdout stream? Or just formatting the output for the helper to write. Do
>> you want to support helpers that want to manage I/O on their own? Do you
>> want to support embedded low-overhead helpers, eCAP-style?
>
> At the moment its just this:
> #define SEND_OK(x) fprintf(stdout,"OK %s\n", (x))

 From API point of view, this implies that the Framework is managing
I/O. There is not much management here, of course, but that is not
important.

> helpers may use the wrapper for convenience and easier to read code.
> I've converted several of the simple helpers over already. But they are
> still free to use the direct socket/pipe feedback if they need/want.

If needed, the Framework can expose the descriptor or class for writing
responses but it will become more complex and rigid if we allow such raw
access. I do not know whether it is needed. I am not a helper expert.

>>> * the debug() call doing printf-style output as now but with automatic
>>> prefixing of helper name and PID (matching the kidN for cache.log)
>>
>> Printf() is far from ideal for C++ helpers. I do not know whether we
>> have or expect any though.
>
> We are on the border. The helpers are C code built with g++ now. Future
> ones could be pure C++.

Again, I do not know whether supporting C++ helpers is important. This
is something you and others should decide. If it is, and if current
helpers already compile with g++, I would not use printf.

>>> * standardizing the -d (debug on) and -h (help) parameters for all
>>> helpers compiled.
>>
>>
>>> Does anyone have any advice about good ways to make a formal public API
>>> that the existing bundled helpers, and potentially third-parties could
>>> use when building C/C++ helpers for Squid?
>>>
>>> ie things that must be one for versioning alterations over time.
>>
>> I would start with defining what a helper is, and what you want to
>> provide helper authors/code with: A simple API/library their programs
>> would use, a framework where they can just plugin their custom stuff and
>> that would do I/O and process management for them, eCAP-like integration
>> API for embedded helpers, something else? You probably know all these
>> answers, but they are not obvious to some of us :-).
>>
>
> Definition (as per status-quo):
> A third-party program for use by auth, external ACL, url/store/location
> rewrite, and daemon logging. One which is started by Squid automatically
> and processes requests received from Squid.

Three questions based on your definition:

1) If you want to provide a Framework, why limit helpers to programs? We
can define a helper as "code implementing required Framework
functionality for use by ..." and leave open the possibility of embedded
helpers (without necessarily immediately implementing it, of course).

In general, once you take control from the helper and limit its code to
well-defined Framework API, you can usually embed that code without
major problems. And SMP support makes such embedding even more
attractive because it scales with the number of CPU cores...

2) Are auth, external ACL, and url/store/location rewrite needs similar?
Do we want to provide the same API for all three categories? Or one
common Framework with three customization options, one for each helper
category?

3) I know that url/store/location rewriters constantly push for access
to more info like message headers. Do you want to provide them with
access to message headers (now or later)? If yes, we should make
Framework API compatible with eCAP.

I do not know much about auth and external ACL needs, but if they are
also hungry for more info, the same argument applies.

Again, compatibility with eCAP does _not_ imply that helpers become
embedded. It just leaves that possibility open in the future, without
rewriting all helpers again.

>>> Can we do it without a built library? ie only inlines, templates and
>>> macros?
>>
>> Yes, most likely, especially if you just want a simple API. Boost
>> manages to do a lot with just headers. Doing a good API for both C and
>> C++ helpers may be impossible though. You will probably need to focus on
>> one and then optionally provide a wrapper for another.

Framework as a headers-only library might still work, but writing one
would be a challenge. We can try and bail later if it becomes too
complex, I guess.

HTH,

Alex.
Received on Fri Sep 17 2010 - 18:20:48 MDT