Re: Caching responses with "Vary" responses

On 24.07.2012 05:50, Alex Rousskov wrote:
> On 07/23/2012 02:31 AM, Alexander Komyagin wrote:
>>
>>> On 07/19/2012 08:46 PM, Amos Jeffries wrote:
>>>> On 19/07/2012 9:50 p.m., Alexander Komyagin wrote:
>>>>> On Wed, 2012-07-18 at 11:10 -0600, Alex Rousskov wrote:
>>>>>> On 07/17/2012 09:28 AM, Alexander Komyagin wrote:
>>>>
>>>>>>> It seems that on the first HIT we have successfully loaded data
>>> from
>>>>>>> the
>>>>>>> swap store, but then we save the data to shared memory. So the
>>> next
>>>>>>> request takes data from shm. And it looks like Squid doesn't
>>>>>>> save
>>>>>>> mem_obj->vary_headers to shm!!
>>>>>> Shared memory cache is meant to be just like a disk cache in
>>>>>> this
>>>>>> aspect. It does not know anything about mem_obj->vary_headers
>>>>>> and
>>> just
>>>>>> packs/unpacks the STORE_META_STD TLV (see StoreMapSlot
>>>>>> src/ipc/StoreMap.h) part of StoreEntry.
>>>>>>
>>>>>> I do not see vary_headers used by any src/fs/* code. Do you know
>>> how the
>>>>>> disk cache gets them? Perhaps there is a higher level call to
>>>>>> pack
>>> and
>>>>>> unpack Vary request headers that the MemStore is missing. We can
>>> add
>>>>>> special field/code to StoreMapSlot but I would prefer to see how
>>> it is
>>>>>> done for the disk storage first, in hope that we can reuse the
>>> same
>>>>>> code.
>>>>>>
>>>>> For the disc cache, STORE_META_VARY_HEADERS is used (see
>>>>> store_swapmeta.cc). Appropriate tlv is added to the list.
>>>>>
>>>>> But for shm I don't see any SwapMeta work! There are some calls
>>> that are
>>>>> writing reply headers and body to the internal mem_hdr object and
>>> then I
>>>>> can see this object data copied to shm!
>>>
>>>
>>> The current SwapMeta work is in populating StoreMapSlot with
>>> STORE_META_STD TLV. Clearly, that is not enough to support Vary.
>>>
>>
>> So why we can't use for shm the same saving procedure as for Swap
>> Cache?
>> It's much easier to have only one store object packing procedure for
>> all
>> different types of memory store. How can we do that?
>
> Current shared memory cache requires a synchronous "saving procedure"
> (one function call must save everything). Disk store API uses an
> asynchronous one. Correctly supporting asynchronous "saving
> procedure"
> using shared memory is desirable but difficult, as discussed on the
> "object size restrictions" squid-dev thread.
>
> However, we may be able to use the same meta-data part of the "saving
> procedure" for both disk and shared memory cache if that part of the
> procedure is synchronous. I do not remember why I did not, but this
> needs to be re-investigated in case I missed that opportunity.

For the Vary meta its only a few hundred bytes normally as Alexander
noted (~230 in his test).

However some brilliant sites decide to use Cookie or User-Agent as one
of the vary components. This can bloat the entry up to a worst-case of
more than 64KB on some occasions. Other proxies and Apache allow ~10KB
Cookie to be set so it is more the browser or scripted tools which
generate these nasties.

You will need special-case logics to handle this mess for shared-memory
pages (drop out of shared memory?), but most traffic should be fine.

Amos
Received on Mon Jul 23 2012 - 23:05:14 MDT