Re: Re: Caching responses with "Vary" responses

Since nobody replied, I will answer myself ;)

Vary header is created in StoreEntry::setPublicKey() function.

1) First problem is that we are writing data to the wrong store key. The
reason is that we are calling startWriting() for Vary header StoreEntry
(through replaceHttpReply() ) before we have the correct key. Easy to
fix (see attachment).

2) Second one: even when we call startWriting() for the correct store
key (which one is the same store hash as an response for the same URL
without Vary - thanks to Henrik for the explanation!), our StoreEntry
won't get swapped: after startWriting() the writeComplete() routine is
called, which calls invokeHandlers() and, eventually, swapOut().
In swapOut() after checking that mayStartSwapOut() is false (complete()
was not called yet, so no content length is set), we are calling
trimMemory(), which sees our StoreEntry as a subject for trimming - so
it releases the request and makes our entry private - this indeed
prevents the entry to be swapped.

I can suggest two possible solutions for the last problem:
either we tell trimMemory() to ignore our Vary header StoreEntry using
entry flags (see squidv3-vary-cache.patch) - but it's certainly just a
dirty hack -
or we shall create a fake store_client for our StoreEntry, so the
condition mem_obj->policyLowestOffsetToKeep(0) == 0 in trimMemory() will
evaluate to 'true' as it should and StoreEntry will be successfully
swapped on complete() (see squidv3-vary-cache-1.patch).

Though the store_client way seems to me much more convenient than some
dirty hack, I'm not quite sure about side effects that could arise, can
somebody clarify this one for me?

After applying any of attached patches, Squid correctly saves varying
objects to the swap store and all next requests to the same site (tested
with http://www.ietf.org/rfc/rfc1517.txt) are HIT.

However, this uncovers another issue (SURPRISE!): first HIT is ok, but
others are producing the following log output:

varyEvaluateMatch: Oops. Not a Vary object on second attempt,
'http://www.ietf.org/rfc/rfc1517.txt' 'accept-encoding'

It seems that on the first HIT we have successfully loaded data from the
swap store, but then we save the data to shared memory. So the next
request takes data from shm. And it looks like Squid doesn't save
mem_obj->vary_headers to shm!!

On Fri, 2012-07-13 at 12:21 +0400, Alexander Komyagin wrote:
> OK. So, regarding the log I've previously attached, those 235 bytes
> (vary header) shall be saved on C447BB767EC4BC4DAC1820CE7999F4C4, while
> the response itself (7808 bytes) can be saved on any other, right?
>
> If it's right, we got two problems here:
> 1) Vary header hash is not correct (shall be
> C447BB767EC4BC4DAC1820CE7999F4C4, but got
> 4DB6DCFE80050CF0CA306DB8EDFF718E)
> 2) Vary header doesn't get saved to the store
> ( StoreEntry::checkCachable: NO: private key)
>
> On Thu, 2012-07-12 at 22:29 +0200, Henrik Nordström wrote:
> > tor 2012-07-12 klockan 14:32 +0400 skrev Alexander Komyagin:
> >
> > > 2) What are those 235 bytes that are written to the 'vary' storeEntry
> > > (created in setPublicKey)? And what is the purpose of that additional
> > > storeEntry?
> >
> > It's the Vary header of the response so future lookups know how to build
> > the store key.
> >
> > This object should get saved on the same store hash as an response for
> > the same URL without Vary.
> >
> > Regards
> > Henrik
> >
>

-- 
Best wishes,
Alexander Komyagin