Hi,
I identify that it is not cached ( or maybe cached but not served
from the cache !) when looking at the logs : it is always a TCP_MISS
when getting this object + it is reported in the X-Cache header
whenever I get the object with wget :
wget -e http_proxy=X.X.X.X:3128 -S
http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/22.0/update/win32/fr/firefox-22.0.complete.mar
-O /dev/null
--2013-07-08 14:54:09--
http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/22.0/update/win32/fr/firefox-22.0.complete.mar
Connecting to X.X.X.X:3128... connected.
Proxy request sent, awaiting response...
HTTP/1.1 200 OK
Last-Modified: Tue, 18 Jun 2013 15:38:37 GMT
ETag: "dcfc8c-19e4648-4df6f81aff940"
Server: Apache
X-Backend-Server: ftp3.dmz.scl3.mozilla.com
Content-Type: application/octet-stream
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
X-Cache-Info: caching
Content-Length: 27149896
Cache-Control: max-age=164515
Expires: Wed, 10 Jul 2013 10:37:05 GMT
Date: Mon, 08 Jul 2013 12:55:10 GMT
X-Cache: MISS from prxpusx104
X-Cache: MISS from prxpusx001
Via: 1.1 prxpusx104 (squid/3.2.11), 1.1 prxpusx001 (squid/3.2.11)
Connection: keep-alive
Length: 27149896 (26M) [application/octet-stream]
Saving to: ‘/dev/null’
100%[=========================================================================================>]
27,149,896 8.57MB/s in 3.0s
2013-07-08 14:54:12 (8.57 MB/s) - ‘/dev/null’ saved [27149896/27149896]
More info. Here is the squid.conf file ( squid v 3.2.11 ). I striped
down some ACLs so that it is more readable ( I hope ).
So yes SMP with 4 workers.
# pour le debogage (ne pas mettre plus de 2)
#debug_options ALL,2
# Pas de limitation sur les FD
max_filedescriptors 65535
# Utilisateurs
cache_effective_user nobody
cache_effective_group nobody
# Format access.log
strip_query_terms off
#logformat Squid %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un %Sh/%<A %mt
logformat PAS-Bdx %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un %Sh/%<A
%mt %rv %tl "%{Referer}>h" "%{User-Agent}>h"
# chemins
coredump_dir /var/cache/squid
pid_filename /var/run/squid/squid.pid
access_log stdio:/var/log/squid/access.log PAS-Bdx
cache_log /var/log/squid/cache.log
cache_store_log none
mime_table /etc/squid/mime.conf
error_directory /etc/squid/errors
error_default_language fr
err_page_stylesheet /etc/squid/errorpage.css
# Fichier hosts
hosts_file /etc/hosts
# SNMP
acl snmpcommunity snmp_community read_only_user
snmp_access allow snmpcommunity
snmp_port 3401
###########################
# FONCTIONNEMENT DU PROXY #
###########################
#SMP
workers 4
#Ports d'ecoute
http_port 3128
http_access allow localhost manager
http_access deny manager
#définition de la VIP des squid Père
cache_peer X.X.X.X parent 3128 0 default no-query no-digest
# Filtrage avec squidGuard
url_rewrite_program /usr/local/squidGuard/bin/squidGuard
url_rewrite_children 1000 startup=10 idle=5 concurrency=0
# Time Out / Time To Live
negative_ttl 1 seconds
read_timeout 15 minutes
request_timeout 5 minutes
client_lifetime 4 hours
positive_dns_ttl 2 hours
negative_dns_ttl 5 minutes
shutdown_lifetime 5 seconds
dns_nameservers 127.0.0.1
# Divers
ftp_passive on
ftp_epsv off
logfile_rotate 2
request_header_access Via deny all
request_header_access X-Forwarded-For allow all
refresh_all_ims on
###########################
# FONCTIONNEMENT DU CACHE #
###########################
#definition du cache
#8Gb of shared rock cache, for 32Ko objects max
cache_dir rock /var/cache/squid/mem/ 8192 max-size=32768
if ${process_number} =1
cache_dir aufs /var/cache/squid/mem/W${process_number} 3000 16 256
min-size=32768 max-size=131072
cache_dir aufs /var/cache/squid/W${process_number} 9000 16 256 min-size=131072
endif
if ${process_number} =2
cache_dir aufs /var/cache/squid/mem/W${process_number} 3000 16 256
min-size=32768 max-size=131072
cache_dir aufs /var/cache/squid/W${process_number} 9000 16 256 min-size=131072
endif
if ${process_number} =3
cache_dir aufs /var/cache/squid/mem/W${process_number} 3000 16 256
min-size=32768 max-size=131072
cache_dir aufs /var/cache/squid/W${process_number} 9000 16 256 min-size=131072
endif
if ${process_number} =4
cache_dir aufs /var/cache/squid/mem/W${process_number} 3000 16 256
min-size=32768 max-size=131072
cache_dir aufs /var/cache/squid/W${process_number} 9000 16 256 min-size=131072
endif
# pages dynamiques non mises en cache
acl QUERY urlpath_regex cgi-bin \? \.fcgi \.cgi \.pl \.php3 \.asp \.php \.do
no_cache deny QUERY
#Rafraichissement du cache
memory_cache_shared on
cache_mem 4 GB
max_filedesc 65535
maximum_object_size 512 MB
maximum_object_size_in_memory 2048 KB
ipcache_size 8192
fqdncache_size 8192
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
# Least frequently used with dynamic aging
cache_replacement_policy heap LFUDA
memory_replacement_policy heap LFUDA
# pas de redirecteur
server_persistent_connections on
client_persistent_connections on
half_closed_clients off
#tout le trafic passe par le squid père sauf les pages d'erreur
always_direct allow to_localhost
never_direct allow all
# Full internet
acl fullinternet src "/etc/squid/acl/src/full.src"
http_access allow fullinternet
http_access deny all
__________________________________________
here is the output of the mgr:storedir :
[root] # /usr/local/squid/bin/squidclient mgr:storedir
HTTP/1.1 200 OK
Server: squid/3.2.11
Mime-Version: 1.0
Date: Mon, 08 Jul 2013 12:57:27 GMT
Content-Type: text/plain
Expires: Mon, 08 Jul 2013 12:57:27 GMT
Last-Modified: Mon, 08 Jul 2013 12:57:27 GMT
Connection: close
by kid1 {
Store Directory Statistics:
Store Entries : 69552
Maximum Swap Size : 12288000 KB
Current Store Swap Size: 11058996.00 KB
Current Capacity : 90.00% used, 10.00% free
Shared Memory Cache
Maximum Size: 4194304 KB
Maximum entries: 131072
Current entries: 125817 95.99%
Store Directory #1 (aufs): /var/cache/squid/mem/W1
FS Block Size 4096 Bytes
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 3072000 KB
Current Size: 2764880.00 KB
Percent Used: 90.00%
Filemap bits in use: 45166 of 65536 (69%)
Filesystem Space in use: 18200988/20971520 KB (87%)
Filesystem Inodes in use: 197750/4627610 (4%)
Flags:
Removal policy: lru
LRU reference age: 5.14 days
Store Directory #2 (aufs): /var/cache/squid/W1
FS Block Size 4096 Bytes
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 9216000 KB
Current Size: 8294116.00 KB
Percent Used: 90.00%
Filemap bits in use: 23235 of 32768 (71%)
Filesystem Space in use: 39747760/59850496 KB (66%)
Filesystem Inodes in use: 117081/15451392 (1%)
Flags:
Removal policy: lru
LRU reference age: 7.19 days
} by kid1
by kid2 {
Store Directory Statistics:
Store Entries : 71099
Maximum Swap Size : 12288000 KB
Current Store Swap Size: 11059168.00 KB
Current Capacity : 90.00% used, 10.00% free
Shared Memory Cache
Maximum Size: 4194304 KB
Maximum entries: 131072
Current entries: 125817 95.99%
Store Directory #1 (aufs): /var/cache/squid/mem/W2
FS Block Size 4096 Bytes
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 3072000 KB
Current Size: 2764784.00 KB
Percent Used: 90.00%
Filemap bits in use: 45277 of 65536 (69%)
Filesystem Space in use: 18200988/20971520 KB (87%)
Filesystem Inodes in use: 197750/4627610 (4%)
Flags:
Removal policy: lru
LRU reference age: 4.95 days
Store Directory #2 (aufs): /var/cache/squid/W2
FS Block Size 4096 Bytes
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 9216000 KB
Current Size: 8294384.00 KB
Percent Used: 90.00%
Filemap bits in use: 24949 of 32768 (76%)
Filesystem Space in use: 39747760/59850496 KB (66%)
Filesystem Inodes in use: 117081/15451392 (1%)
Flags:
Removal policy: lru
LRU reference age: 7.03 days
} by kid2
by kid3 {
Store Directory Statistics:
Store Entries : 71223
Maximum Swap Size : 12288000 KB
Current Store Swap Size: 11057428.00 KB
Current Capacity : 89.99% used, 10.01% free
Shared Memory Cache
Maximum Size: 4194304 KB
Maximum entries: 131072
Current entries: 125817 95.99%
Store Directory #1 (aufs): /var/cache/squid/mem/W3
FS Block Size 4096 Bytes
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 3072000 KB
Current Size: 2764776.00 KB
Percent Used: 90.00%
Filemap bits in use: 45360 of 65536 (69%)
Filesystem Space in use: 18200988/20971520 KB (87%)
Filesystem Inodes in use: 197750/4627610 (4%)
Flags:
Removal policy: lru
LRU reference age: 4.18 days
Store Directory #2 (aufs): /var/cache/squid/W3
FS Block Size 4096 Bytes
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 9216000 KB
Current Size: 8292652.00 KB
Percent Used: 89.98%
Filemap bits in use: 24888 of 32768 (76%)
Filesystem Space in use: 39747760/59850496 KB (66%)
Filesystem Inodes in use: 117081/15451392 (1%)
Flags:
Removal policy: lru
LRU reference age: 7.14 days
} by kid3
by kid4 {
Store Directory Statistics:
Store Entries : 71236
Maximum Swap Size : 12288000 KB
Current Store Swap Size: 11058892.00 KB
Current Capacity : 90.00% used, 10.00% free
Shared Memory Cache
Maximum Size: 4194304 KB
Maximum entries: 131072
Current entries: 125817 95.99%
Store Directory #1 (aufs): /var/cache/squid/mem/W4
FS Block Size 4096 Bytes
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 3072000 KB
Current Size: 2764760.00 KB
Percent Used: 90.00%
Filemap bits in use: 45489 of 65536 (69%)
Filesystem Space in use: 18200988/20971520 KB (87%)
Filesystem Inodes in use: 197750/4627610 (4%)
Flags:
Removal policy: lru
LRU reference age: 5.20 days
Store Directory #2 (aufs): /var/cache/squid/W4
FS Block Size 4096 Bytes
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 9216000 KB
Current Size: 8294132.00 KB
Percent Used: 90.00%
Filemap bits in use: 24771 of 32768 (76%)
Filesystem Space in use: 39747760/59850496 KB (66%)
Filesystem Inodes in use: 117081/15451392 (1%)
Flags:
Removal policy: lru
LRU reference age: 10.99 days
} by kid4
by kid5 {
Store Directory Statistics:
Store Entries : 52
Maximum Swap Size : 8388608 KB
Current Store Swap Size: 8387184.00 KB
Current Capacity : 99.98% used, 0.02% free
Store Directory #0 (rock): /var/cache/squid/mem/
FS Block Size 1024 Bytes
Maximum Size: 8388608 KB
Current Size: 8387184.00 KB 99.98%
Maximum entries: 262143
Current entries: 262099 99.98%
Pending operations: 1 out of 0
Flags:
} by kid5
Thanks!
Alex
2013/7/5 Amos Jeffries <squid3_at_treenet.co.nz>:
> On 5/07/2013 3:33 a.m., Alexandre Chappaz wrote:
>>
>> Hi,
>>
>> I have this object not being cached, and I can't understand why.
>
>
> Lets start with... how are you identifying that it is not being cached?
>
>>
>> http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/22.0/update/win32/fr/firefox-22.0.complete.mar
>
> redbot.org confirms this is a cacheable resource with no protocol problems
> visible.
>
>
>> I have this cachedir set for objects of size > 128Ko :
>>
>> cache_dir aufs /var/cache/squid/W${process_number} 9000 16 256
>> min-size=131072
>
>
> SMP macros in use I see. How many workers do you have?
>
> Also, what *other* size limits on object sizes are in your configuration
> file?
> please list those *and* all cache_dir lines in your configuration file in
> exactly the order they appear in the config.
>
>
>
>>
>>
>> the headers of the object :
>> HTTP/1.1 200 OK
>> Last-Modified: Tue, 18 Jun 2013 15:38:37 GMT
>> ETag: "dcfc8c-19e4648-4df6f81aff940"
>> Server: Apache
>> X-Backend-Server: ftp3.dmz.scl3.mozilla.com
>> Content-Type: application/octet-stream
>> Accept-Ranges: bytes
>> Access-Control-Allow-Origin: *
>> X-Cache-Info: caching
>> Content-Length: 27149896
>> Cache-Control: max-age=266561
>> Expires: Sat, 06 Jul 2013 18:26:03 GMT
>> Date: Wed, 03 Jul 2013 16:23:22 GMT
>>
>>
>> the storedir is full, but I understand that since the object is
>> requested very often, it should replace anotherone in the cachedir.
>> (cache_replacement_policy heap LFUDA )
>
>
> You say *the storedir* ... but there are multiple store directories yes? one
> for each Squid worker process.
>
>
>
>> Do you have any hint on how to make sure th object gets cached?
>> Thanks
>
>
> More info needed.
>
> Please also list your refresh_pattern rules.
>
>
> Amos
Received on Mon Jul 08 2013 - 12:58:03 MDT
This archive was generated by hypermail 2.2.0 : Mon Jul 08 2013 - 12:00:08 MDT