Hi Amos,
On Mon, Feb 21, 2011 at 5:37 PM, Amos Jeffries <squid3_at_treenet.co.nz> wrote:
> On Mon, 21 Feb 2011 11:52:11 -0500, John Craws wrote:
>>
>> Hi,
>>
>> Thank you for the clarification. Maybe I'm just not correctly
>> interpreting whether the object is in the cache or not.
>> Here's the info you asked for, based on the config I posted
>> previously. I'm downloading a +- 16M file.
>>
>> 1. Before downloading the object:
>>
>> john.craws@jjj:~/wget$ curl -I http://172.16.199.150/popeye.mp4
>> HTTP/1.0 200 OK
>> Date: Mon, 21 Feb 2011 16:46:56 GMT
>> Server: Apache/2.2.3 (Red Hat)
>> Last-Modified: Thu, 24 Sep 2009 19:22:32 GMT
>> ETag: "e2800c-1013726-47457c0c5ae00"
>> Accept-Ranges: bytes
>> Content-Length: 16856870
>> Content-Type: video/mp4
>> X-Cache: MISS from jnk
>> Via: 1.0 jnk (squid/3.1.11)
>> Connection: keep-alive
>>
>>
>> john.craws_at_jjj:~/wget$ /opt/squid/bin/squidclient mgr:objects
>> HTTP/1.0 200 OK
>> Server: squid/3.1.11
>> Mime-Version: 1.0
>> Date: Mon, 21 Feb 2011 15:51:02 GMT
>> Content-Type: text/plain
>> Expires: Mon, 21 Feb 2011 15:51:02 GMT
>> Last-Modified: Mon, 21 Feb 2011 15:51:02 GMT
>> X-Cache: MISS from jnk
>> Via: 1.0 jnk (squid/3.1.11)
>> Connection: close
>>
>> (lists cached objects, no trace of the object -- normal).
>>
>> 2. Downloading the object
>>
>> john.craws@jjj:~/wget$ curl http://172.16.199.150/popeye.mp4 -o popeye.mp4
>> % Total % Received % Xferd Average Speed Time Time
>> Time Current
>> Dload Upload Total Spent Left
>> Speed
>> 100 16.0M 100 16.0M 0 0 2446k 0 0:00:06 0:00:06
>> --:--:-- 1634k
>>
>> 3. This time the object appears in the list
>>
>> john.craws_at_jjj:~/wget$ /opt/squid/bin/squidclient mgr:objects
>> HTTP/1.0 200 OK
>> Server: squid/3.1.11
>> Mime-Version: 1.0
>> Date: Mon, 21 Feb 2011 15:51:18 GMT
>> Content-Type: text/plain
>> Expires: Mon, 21 Feb 2011 15:51:18 GMT
>> Last-Modified: Mon, 21 Feb 2011 15:51:18 GMT
>> X-Cache: MISS from jnk
>> Via: 1.0 jnk (squid/3.1.11)
>> Connection: close
>>
>> (...)
>> KEY 669AB801B7640FA80E4BA73193FDAC2A
>> STORE_OK IN_MEMORY SWAPOUT_NONE PING_DONE
>> CACHABLE,DISPATCHED,VALIDATED
>> LV:1298303469 LU:1298303469 LM:1253820152 EX:-1
>> 0 locks, 0 clients, 1 refs
>> Swap Dir -1, File 0XFFFFFFFF
>> GET http://172.16.199.150/popeye.mp4
>> inmem_lo: 0
>> inmem_hi: 16857134
>> swapout: 0 bytes queued
>> (...)
>>
>>
>> 4. This time it's a HIT
>>
>> john.craws@jjj:~/wget$ curl -I http://172.16.199.150/popeye.mp4
>> HTTP/1.0 200 OK
>> Date: Mon, 21 Feb 2011 15:51:09 GMT
>> Server: Apache/2.2.3 (Red Hat)
>> Last-Modified: Thu, 24 Sep 2009 19:22:32 GMT
>> ETag: "e2800c-1013726-47457c0c5ae00"
>> Accept-Ranges: bytes
>> Content-Length: 16856870
>> Content-Type: video/mp4
>> Age: 16
>> X-Cache: HIT from jnk
>> Via: 1.0 jnk (squid/3.1.11)
>> Connection: keep-alive
>>
>> 5. access.log
>>
>> 1298304218.303 19 127.0.0.1 TCP_MISS/200 322 HEAD
>> http://172.16.199.150/popeye.mp4 - DIRECT/172.16.199.150 video/mp4
>> 1298304222.694 0 127.0.0.1 TCP_MEM_HIT/200 329 HEAD
>> http://172.16.199.150/popeye.mp4 - NONE/- video/mp4
>>
>> I also notice the difference the major time difference between the two
>> curl operations.
>>
>
> Thanks for the detail. You are right about it being in the memory cache.
>
> What I expect to see with your config is that file pushed to disk, since it
> is within the 17MB but over the 32KB. But you have no on-disk cache right?
You are correct. I have (intentionally) no disk cache (no cache_dir
directive). I expected the object to be discarded.
>
> Something funky is going on with the swapout.
>
> I think there are 2 bugs visible here, the easy one is that the config
> parser is not detecting and warning about the global limit being larger than
> the biggest specific limit. Second being the object not discarded when over
> memory size and push to disk not possible.
That's what I expected also. Let me know if I can do anything to help.
Is it reasonable to open a bug?
Thanks,
John
>
> Amos
>
>> Thanks!
>>
>> John
>>
>>
>> On Fri, Feb 18, 2011 at 5:56 PM, Amos Jeffries <squid3_at_treenet.co.nz>
>> wrote:
>>>
>>> On 19/02/11 07:28, John Craws wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have a squid 3.1.11 instance configured with no disk cache.
>>>> Stripped down configuration below.
>>>>
>>>>
>>>>
>>>>
>>>> #--------------------------------------------------------------------------------
>>>> # squid.conf
>>>>
>>>>
>>>>
>>>> #--------------------------------------------------------------------------------
>>>>
>>>> shutdown_lifetime 0 seconds
>>>> http_port 3128
>>>> http_access allow all
>>>> forwarded_for transparent
>>>>
>>>> acl VIDEO-CONTENT rep_header Content-Type video/.+
>>>>
>>>> maximum_object_size_in_memory 32 KB
>>>> maximum_object_size 17 MB
>>>> cache_mem 4 GB
>>>> cache allow all
>>>> debug_options ALL,1
>>>>
>>>>
>>>>
>>>>
>>>> #--------------------------------------------------------------------------------
>>>> # end squid.conf
>>>>
>>>>
>>>>
>>>> #--------------------------------------------------------------------------------
>>>>
>>>> I use curl to request a file through the proxy.
>>>>
>>>> What I understood from the docs is that maximum_object_size limits the
>>>> size of objects to disk cache and maximum_object_size_in_memory does
>>>> so for RAM cache.
>>>>
>>>> I have no disk cache, so I tried to apply a limit using
>>>> maximum_object_size_in_memory. It didn't work. However, it does work
>>>> when I specify a limit using maximum_object_size.
>>>>
>>>> What am I doing wrong?
>>>
>>> You are almost correct.
>>> maximum_object_size is a global limit which nothing stored anywhere can
>>> evade.
>>> maximum_object_size_in_memory is a RAM-only limit. The smaller of memory
>>> and global limits is used for cache_mem.
>>>
>>> There are matching *-size limits on individual cache_dir in the latest
>>> Squid that do the same for each disk dir.
>>>
>>>
>>> It should be preventing long-term storage of anything over 32KB, provided
>>> the size info is known in the headers.
>>> Note that this does not limit objects which are currently in transit.
>>> They
>>> are stored for as long as that transit use needs, then discarded.
>>>
>>> Do you have a copy of the headers and a display of how you are
>>> identifying
>>> the failure please?
>>>
>>> Amos
>>> --
>>> Please be using
>>> Current Stable Squid 2.7.STABLE9 or 3.1.11
>>> Beta testers wanted for 3.2.0.5
>>>
>
>
Received on Tue Feb 22 2011 - 15:15:22 MST
This archive was generated by hypermail 2.2.0 : Wed Feb 23 2011 - 12:00:03 MST