Re: [RFC] post-cache REQMOD

The post-cache REQMOD and post-cache RESPMOD is a must for squid.

The example of PageSpeed also is very good. I must note that there are
already similar features integrated to other commercial products, for
example:
 
http://www.citrix.com/products/bytemobile-adaptive-traffic-management/tech-info.html
   ("Web and video optimization" -> "Quality Aware Transcoding",
"Smartphone Application Acceleration" and "Web Optimization")

The PageSpeed example fits better to a post-cache RESPMOD feature. Is
the post-cacge REQMOD just a first step to support all post-cache
vectoring points?

On 07/11/2014 01:15 AM, Alex Rousskov wrote:
> Hello,
>
> I propose adding support for a third adaptation vectoring point:
> post-cache REQMOD. Services at this new point receive cache miss
> requests and may adapt them as usual. If a service satisfies the
> request, the service response may get cached by Squid. As you know,
> Squid currently support pre-cache REQMOD and pre-cache RESPMOD.
>
>
> We have received many requests for post-cache adaptation support
> throughput the years, and I personally resisted the temptation of adding
> another layer of complexity (albeit an optional one) because it is a lot
> of work and because many use cases could be addressed without post-cache
> adaptation support.
>
> The last straw (and the motivation for this RFC) was PageSpeed[1]
> integration. With PageSpeed, one can generate various variants of
> "optimized" content. For example, mobile users may receive smaller
> images. Apache and Nginx support PageSpeed modules. It is possible to
> integrate Squid with PageSpeed (and similar services) today, but it is
> not possible for Squid to _cache_ those generated variants unless one is
> willing to pay for another round trip to the origin server to get
> exactly the same unoptimized content.
>
> The only way to support Squid caching of PageSpeed variants without
> repeated round trips to the origin server is using two Squids. The
> parent Squid would cache origin server responses while the child Squid
> would adapt parent's responses and cache adapted content. Needless to
> say, running two Squids (each with its own cache) instead of one adds
> significant performance/administrative overheads and complexity.
>
>
> As far as internals are concerned, I am currently thinking of launching
> adaptation job for this vectoring point from FwdState::Start(). This
> way, its impact on the rest of Squid would be minimal and some adapters
> might even affect FwdState routing decisions. The initial code name for
> the new class is MissReqFilter, but that may change.
>
>
>
> The other candidate location for plugging in the new vectoring point is
> the Server class. However, that class is already complex. It handles
> communication with the next hop (with child classes doing
> protocol-specific work and confusing things further) as well as
> pre-cache RESPMOD vectoring point with caching initiation on top of
> that. The Server code already has trouble distinguishing various content
> streams it has to juggle. I am worried that adding another vectoring
> point there would make that complexity significantly worse.
>
> It is possible that we would be able to refactor/encapsulate some of the
> code so that it can be reused in both the existing Server and the new
> MissReqFilter classes. I will look out for such opportunities while
> trying to keep the overall complexity in check.
>
>
> Any objections to adding post-cache REQMOD or better implementation ideas?
>
>
> Thank you,
>
> Alex.
> [1] https://developers.google.com/speed/pagespeed/
>
Received on Fri Jul 11 2014 - 11:27:48 MDT