Re: [squid-users] Store object length

On 02/25/2013 04:45 AM, Amos Jeffries wrote:

> This is a code question. Please followup on squid-dev mailing list. The
> developers can help you more with this than the Squid users can.

Responding on squid-dev only.

> On 25/02/2013 8:00 p.m., anita wrote:
>> Hi All,
>>
>> I am trying to write an API that will fetch objects from the squid cache
>> given the url and http method.
>> When I analyzed the code, I found that this can be achieved with a
>> storeGetPublic(url, http_method) to see if the url has an entry in the
>> cache.
>> This followed by storeClientCopy() call.
>> However, the StoreIOBuffer used here has been initialized with a
>> lenght of HTTP_REQBUF_SZ (4096 or 4 k) by default.
>
> Err. Yes storeGetPublic() is the API to do this with.
>
> Why are you writing a new one? what are you trying to do with Squid?

I agree with Amos that these questions are important, for many reasons,
ranging from compatibility with official code to being able to get
support on squid-dev.

To answer your buffer size question: The API does not restrict what
buffer sizes the caller should use. Some client side code uses
HTTP_REQBUF_SZ, but your code may use a different size. The store_client
subscriber will receive callbacks when more data is available. It is a
complex, iterative process; not a single "give me the whole response
data" call.

In theory, even a 1-byte buffer should work, but, besides being too
slow, I suspect some Store code assumes that HTTP response headers will
fit into the first buffer given to Store by store_client. Thus, it would
be unwise to use smallish buffers without fixing Store first.

Also, please note that storeGetPublic() returns an entry that is not
necessarily fully cached. In the extreme case of collapsed forwarding,
it returns an entry that is only expected to be cached when received
from the origin server.

>> I would like to know how larger objects are stored and retrieved from
>> cache? How can I determine the length of the object read in that case?

In general, you should not rely on knowing the response size a priori
because some store modules may allow storage of objects where the size
is not known when swap out begins (that is a good thing) and do not
update the recorded size when swap out ends (that would be a performance
optimization). I doubt Squid fully supports that today, but eventually
it should, so it is better not to make a priori size knowledge a
critical precondition in your code (if possible).

Please describe what problem you are trying to solve and why you think
you need a new API for what is already supported. With some luck,
squid-dev folks will be able to help you along the way.

Thank you,

Alex.

>> Version used: Squid ver 3.1.16
>> squid.conf : default configuration file with cache_dir directive
>> uncommented.
>>
>> Any pointers are appreciated.
>
> When playing with the Squid code please always upgrade to the most
> recent you can before starting. Preferrably that would be 3.HEAD which
> is our rolling development version.
> 3.1 Squid series has been officially deprecated for about five months
> now, and even 3.2 is nearing end of life.
>
> The Squid-3 series is one very long process of converting the code from
> C code constructs to C++ objects APIs and pluggable mechanisms. So you
> may find it much easier to code later releases, or that your fix has
> already been done.
>
> Amos
Received on Mon Feb 25 2013 - 16:15:39 MST