Re: [squid-users] Re: Splitting objects by size into different cache_dir not working for me

On 2/1/2013 10:06 PM, Luciano Ruete wrote:
>>
>> You can set these settings in squid.conf
>> store_dir_select_algorithm round-robin
>> #^^default is: "least-load" which can cause your problem.
>> cache_dir rock /var/spool/squid/rock 1000 min-size=1024 max-size=31000
>> max-swap-rate=250 swap-timeout=350
>> cache_dir aufs /var/spool/squid/aufs 30000 16 256 min-size=209715
>> max-size=734003200
>> maximum_object_size_in_memory 512 KB #default
>> minimum_object_size 0 KB #default
>> maximum_object_size 300 MB #non default
>> ##end conf
>>
>> First try only the round-robin to make sure this is not the basic
>> cause for problems.
>> reload\restart check the cache_dir sizes before and after getting into
>> a website like yahoo movies or any site with many objects in it.
>
> With this conf, objects larger than 512KB does not get cache into AUFS
> cache_dir, example:
>
> requesting
> 1359747785.360 97216 127.0.0.1 TCP_MISS/200 9225893 GET
> http://repo.ngtech.co.il/rpm/centos/6/x86_64/squid-debuginfo-3.2.6-1.el6.x86_64.rpm
> - HIER_DIRECT/58.28.153.233 application/x-rpm
>
> results in this in store log:
> 1359747785.360 RELEASE -1 FFFFFFFF E8A98EBEE591A2878F4795D16C3A434D 200
> 1357962518 1357827095 1389498518 application/x-rpm 9225364/9225364 GET
> http://repo.ngtech.co.il/rpm/centos/6/x86_64/squid-debuginfo-3.2.6-1.el6.x86_64.rpm
>
> And no change in AUFS size at all.
>
> File with less than 512KB (and above 205K min-size) does goes to AUFS,
> example

This is since they are cached in RAM cache until they dont need to be
swapped into disk.
The above is normal squid behavior.

>
> requesting
> 1359747696.788 8644 127.0.0.1 TCP_MISS/200 231025 GET
> http://repo.ngtech.co.il/rpm/centos/6/others/openssh-6.1p1-81.el6.x86_64.rpm
> - HIER_DIRECT/58.28.153.233 application/x-rpm
>
> results in this in store log:
> 1359747696.784 SWAPOUT 01 00000000 9641F2C5678BA0B610DC647C16AE8EAF 200
> 1359744929 1357195218 1391280929 application/x-rpm 230420/230420 GET
> http://repo.ngtech.co.il/rpm/centos/6/others/openssh-6.1p1-81.el6.x86_64.rpm
>
>
> And AUFS dir size gets incremented acording.
>
> ROCK storage also grows if small objects are requested.
>
Good this is what it suppose to do.

>> Then try to change only the maximum_object_size_in_memory from 512
>> default to 8 KB which will cause almost no ram cache and mostly dir
>> cache to make sure that the reason for the low counters is not because
>> of mem caching.
>> Check the cache_dir before and after...
>>
>
> If i lower maximum_object_size_in_memory to 8 KB then
>
> requesting:
> 1359749023.713 1683 127.0.0.1 TCP_MISS/200 1466 GET
> http://repo.ngtech.co.il/rpm/centos/6/x86_64/squid-debuginfo-3.2.5-1.el6.x86_64.rpm.asc
> - HIER_DIRECT/58.28.153.233 text/plain
> 1359749024.590 2561 127.0.0.1 TCP_MISS/200 33255 GET
> http://repo.ngtech.co.il/squid/cachecluster.png -
> HIER_DIRECT/58.28.153.233 image/png
> 1359749028.613 6583 127.0.0.1 TCP_MISS/200 231025 GET
> http://repo.ngtech.co.il/rpm/centos/6/others/openssh-6.1p1-81.el6.x86_64.rpm
> - HIER_DIRECT/58.28.153.233 application/x-rpm
> 1359749121.203 99173 127.0.0.1 TCP_MISS/200 9225894 GET
> http://repo.ngtech.co.il/rpm/centos/6/x86_64/squid-debuginfo-3.2.6-1.el6.x86_64.rpm
> - HIER_DIRECT/58.28.153.233 application/x-rpm
>
> results in:
> 1359749023.712 SWAPOUT 00 000031E8 F104D7F44EDABE94F758EC16BD02ADA0 200
> 1359744786 -1 1360608786 text/plain 876/876 GET
> http://repo.ngtech.co.il/rpm/centos/6/x86_64/squid-debuginfo-3.2.5-1.el6.x86_64.rpm.asc
>
> 1359749024.590 RELEASE -1 FFFFFFFF 5BFF306024844E48B8ED6A3BD89312FF 200
> 1359744790 -1 1360608790 image/png 32719/32719 GET
> http://repo.ngtech.co.il/squid/cachecluster.png
> 1359749028.613 RELEASE -1 FFFFFFFF 45F6D81746DB2C8703359DEF1494F04D 200
> 1359744929 1357195218 1391280929 application/x-rpm 230420/230420 GET
> http://repo.ngtech.co.il/rpm/centos/6/others/openssh-6.1p1-81.el6.x86_64.rpm
>
> 1359749121.203 RELEASE -1 FFFFFFFF 64FDA3C6D9598638709C7D5188D99793 200
> 1357962518 1357827095 1389498518 application/x-rpm 9225364/9225364 GET
> http://repo.ngtech.co.il/rpm/centos/6/x86_64/squid-debuginfo-3.2.6-1.el6.x86_64.rpm

>
>
> Only the smallest object goes to cache and if I flood the proxy with
> user's traffic only the rock storage grows. And only object that are
> less than 8k get a SWAPOUT in store_log.

OK and in this exact scenario with weither 64KB and 512KB everything is
fine and I think it might be because of something with the buffer but
i'm not sure about it yet.
If there is a bug this is the only one exists and only in specific cases
which a wrong tuning of squid.conf is being done.

>
>> If there is no straight answer if it's ok or not and if no clean
>> answer try to use all the directives I gave you together and try to
>> download each of the set of files I gave you.
>>
>> The tiny should be cached in mem.
>> all the others should be cached in a cache_dir by the size.
>> the large ISO file should be cached in any case in the UFS cache_dir.
>>
>> Feel free to ask me anything.
>> I am also in squid IRC channel here and there.
>>
>
> Let me know if you need any further test.

Just use the defaults and change to RoundRobin selection.
Dont change any object size default else then maximum_object_size to 700 MB

If you have any visible problem after that file a bug at the bugzilla
and also refer to this mail as part of the process.

http://bugs.squid-cache.org/

-- 
Eliezer Croitoru
http://www1.ngtech.co.il