RE: [squid-users] Dynamic Content Caching/Windowsupdate/Facebook/youtube

On Mon, 17 May 2010 16:22:19 +0000, "GIGO ." <gigoz_at_msn.com> wrote:
> You recommended the change in order of refresh_patterns same is written
> in the reference materials. I tried to understand what could be the
reason

The refresh_pattern options configure the staleness heuristic (estimate of
how long objects can be served from storage).

They get checked from top-down and the first matching one wins. It can
best be visualized as being made from two sections:
 1) first the section of your own custom rules for storage. Specific
patterns to match certain websites and file types.

 2) a section for patterns which enforce RFC compliance. These are
provided in the default config and have been worked out by us devs and
other experts to provide the best general-use estimate calculation for
storage of certain data types:
 * refresh_pattern ^ftp: 1440 20% 10080
  - Squid uses FTP operations to figure out last-modified times of fetched
objects and this allows squid to store them for another 20% of that age
into the future without doing the lookup again. Maximum of a week if those
ops fail to provide a modified date.

 * refresh_pattern ^gopher: 1440 0% 1440
  - Same deal for Gopher traffic, but with a shorter lifetime of one day
maximum.

 * refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
  - RFC 2616 states that dynamic objects MUST NOT be cached unless
explicit expiry and cache control information is provided. This rule rides
the fine line of exception to prevent Squid from holding on to dynamic
objects without expiry information. It does the same as "cache deny QUERY"
used to but allows caching of dynamic websites which do send proper
Expires: Cache-Control: headers.

 * refresh_pattern . 0 20% 4320
  - Objects which are not matched by any other refresh pattern. Seems to
work storing them for 3 days.

> for that but have no clue yet please guide. Further for windows clients
> (xpwithservice pack 3 an latest windows mostly ) do i need to manually
do
> the configuration for winhttp proxysettings through proxycfg.exe on each
> computer?

For XP you will need to set the clients IE settings to whatever you think
best for your network.

Two catched to be aware of with Windows XP:
 Make sure you run "proxycfg -u" at the command line to update the non-IE
background services which XP still has lying around it.

 Background services will not honour the auto-detect setting of IE, but
will use the proxy or PAC file settings even if disabled.

Vista and later no longer need the proxycfg manually run, but some
machines still seem to have the auto-detect problem. I'm not certain of how
to detect those ones reliably yet. The workaround to use auto-detect seems
to be setting the proxy setting manually in IE, saving, then leaving the
PAC/proxy text there disabling and setting IE to auto and saving again
(leaving the manual configuration text in place for backgrounds services
but disabled for IE regular use).

>
> regards,
>
> Bilal
>
---------------------------------------------------------------------------------------------------
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> refresh_pattern . 0 20% 4320>>>>
>
> refresh_pattern (get_video\?|videoplayback\?|videodownload\?) 5259487
> 99999999% 5259487
>
> Amos>> The youtube pattern and all other custom refresh_patterns' need
to
> be configured above the default set (ftp:, gopher:, cgi-bin, and . ).
>
> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
>
> Amos>> This dynamic content needs to be between the refresh_pattern
> ^gopher: and the refresh_pattern . patterns.
>
>
>
>
>
> ----------------------------------------
>> Date: Sat, 15 May 2010 18:57:18 +1200
>> From: squid3_at_treenet.co.nz
>> To: squid-users_at_squid-cache.org
>> Subject: Re: [squid-users] Dynamic Content
>> Caching/Windowsupdate/Facebook/youtube
>>
>> GIGO . wrote:
>>> All,
>>>
>>> I am really sorry i was looking at the access.log file of squid
>>> instance that is user facing and not the instance that is doing the
>>> fetching/caching and there i can see mp4 files being cached. However i
>>> am not very much confident about my settings so please read my queries
>>> and the configuration file and advice.
>>>
>>> I would be really thankful.
>>>
>>>
>>> ----------------------------------------
>>>> From: gigoz_at_msn.com
>>>> To: squid-users_at_squid-cache.org
>>>> Date: Fri, 14 May 2010 12:00:46 +0000
>>>> Subject: [squid-users] Dynamic Content
>>>> Caching/Windowsupdate/Facebook/youtube
>>>>
>>>>
>>>>
>>>> Dear All,
>>>>
>>>>
>>>> I require your help and guidance regarding dynamic content caching.
>>>> Following are the quries.
>>>>
>>>>
>>>> 1. I am running squid in multiple instances mode (For Cache Disk
>>>> Failure Protection). I dont think that it has any effect on internet
>>>> object caching? I am confused that if connect methods are to be
>>>> duplicate on both of the instances or i have configured it right
>>>> specially in perspective of windows update.
>>>>
>>
>> Depends on whether the port the cache instance is listening on is
>> reachable to external people, if it is then its Squid will definitely
>> need the http_access security settings turned on as well.
>>
>>>>
>>>> 2. As rewrite_url is not exported in new versions(version 3 and
above)
>>>> of squid is it still possible for squid to cache facebook/youtube
>>>> videos? Have i configured it correctly? As i have seen no TCP_HIT for
>>>> mp3,mp4 etc so i think caching is not done.
>>>>
>>
>> If you meant to write "storeurl_rewrite"? then yes. That particular
>> method of caching them is not possible yet in 3.x. YouTube will still
>> cache using the low-efficiency duplicate-object way it does most
places.
>>
>>>>
>>>> 3. Please can u please check my configuration for windows updates? is
>>>> there anything else which i have missed there? How could i assure
that
>>>> windows update is being cached properly?
>>>>
>>
>> You don't show any http_access rules from the cache instance.
>> The default is to block all access through that instance.
>>
>> The main instance is okay.
>>
>>>>
>>>>
>>>> Through studying online tutorials mailarchive support and best of my
>>>> understanding i have configured squid as follows. Please peruse and
>>>> guide.
>>>>
>>>> --------------------------
>>>> Squid Cache Instance:
>>>>
>>>> visible_hostname squidlhr.v.local
>>>> unique_hostname squidcacheinstance
>>>> pid_filename /var/run/squidcache.pid
>>>>
>>>>
>>>> cache_dir aufs /cachedisk1/var/spool/squid 50000 128 256
>>>> coredump_dir /cachedisk1/var/spool/squid
>>>>
>>>> cache_swap_low 75
>>>> cache_mem 1000 MB
>>>> range_offset_limit -1
>>>> maximum_object_size 4096 MB
>>>> minimum_object_size 10 KB
>>>> quick_abort_min -1
>>>> cache_replacement_policy heap
>>>>
>>>> refresh_pattern ^ftp: 1440 20% 10080
>>>> refresh_pattern ^gopher: 1440 0% 1440
>>>> refresh_pattern . 0 20% 4320
>>>>
>>>> #specific for youtube belowone....
>>>> refresh_pattern (get_video\?|videoplayback\?|videodownload\?) 5259487
>>>> 99999999% 5259487
>>
>> The youtube pattern and all other custom refresh_patterns' need to be
>> configured above the default set (ftp:, gopher:, cgi-bin, and . ).
>>
>>>>
>>>> # For any dynamic content caching.
>>>> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
>>
>> This dynamic content needs to be between the refresh_pattern ^gopher:
>> and the refresh_pattern . patterns.
>>
>>>>
>>>> --------------------------------------------------
>>>> Squid Main Instance:
>>>> visible_hostname squidlhr
>>>> unique_hostname squidmain
>>>> cache_peer 127.0.0.1 parent 3128 0 default no-digest no-query
>>>> prefer_direct off
>>>>
>>>> cache_dir aufs /var/spool/squid 10000 16 256
>>>> coredump_dir /var/spool/squid
>>>> cache_swap_low 75
>>>> cache_replacement_policy lru
>>>>
>>>> refresh_pattern ^ftp: 1440 20% 10080
>>>> refresh_pattern ^gopher: 1440 0% 1440
>>
>> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
>>
>> (should be set on all squid caching or handling dynamic objects, even
in
>> memory-only mode).
>>
>>>> refresh_pattern . 0 20% 4320
>>>>
>>>>
>>>> #Defining & allowing ports section....
>>>> acl SSL_ports port 443 # https
>>>> acl Safe_ports port 80 # http
>>>> acl Safe_ports port 21 # ftp
>>>> acl Safe_ports port 443 # https
>>>> acl Safe_ports port 70 # gopher
>>>> acl Safe_ports port 210 # wais
>>>> acl Safe_ports port 1025-65535 # unregistered ports
>>>> acl Safe_ports port 280 # http-mgmt
>>>> acl Safe_ports port 488 # gss-http
>>>> acl Safe_ports port 591 # filemaker
>>>> acl Safe_ports port 777 # multiling http
>>>> acl CONNECT method CONNECT
>>>>
>>>> # Only allow cachemgr access from localhost
>>>> http_access allow manager localhost
>>>> http_access deny manager
>>>>
>>>> # Deny request to unknown ports
>>>> http_access deny !Safe_ports
>>>>
>>>> # Deny request to other than SSL ports
>>>> http_access deny CONNECT !SSL_ports
>>>>
>>>> #Allow access from localhost
>>>> http_access allow localhost
>>>>
>>>>
>>>> # Windows Update Section...
>>>> acl windowsupdate dstdomain windowsupdate.microsoft.com
>>>> acl windowsupdate dstdomain .update.microsoft.com
>>>> acl windowsupdate dstdomain download.windowsupdate.com
>>>> acl windowsupdate dstdomain redir.metaservices.microsoft.com
>>>> acl windowsupdate dstdomain images.metaservices.microsoft.com
>>>> acl windowsupdate dstdomain c.microsoft.com
>>>> acl windowsupdate dstdomain www.download.windowsupdate.com
>>>> acl windowsupdate dstdomain wustat.windows.com
>>>> acl windowsupdate dstdomain crl.microsoft.com
>>>> acl windowsupdate dstdomain sls.microsoft.com
>>>> acl windowsupdate dstdomain productactivation.one.microsoft.com
>>>> acl windowsupdate dstdomain ntservicepack.microsoft.com
>>>> acl wuCONNECT dstdomain www.update.microsoft.com
>>>> acl wuCONNECT dstdomain sls.microsoft.com
>>>> http_access allow CONNECT wuCONNECT all
>>>> http_access allow windowsupdate all
>>>>
>>
>>
>> Amos
>> --
>> Please be using
>> Current Stable Squid 2.7.STABLE9 or 3.1.3
> _________________________________________________________________
> Hotmail: Powerful Free email with security by Microsoft.
> https://signup.live.com/signup.aspx?id=60969
Received on Tue May 18 2010 - 01:05:18 MDT