Re: [squid-users] TProxy Issues

these were just samples of URLs ... ALL traffic is not being cached ...
I am talking about 24 hours of traffic on a busy network. If i switch
back to the old transparent setup everything works in fine. I am at a
loss why tproxy is broken

Amos Jeffries wrote:
> Jamie Orzechowski wrote:
>> I have made the recommended changes but still everything is a
>> TCP_MISS and going DIRECT ... I removed the direct-urls for now just
>> in case. Any further suggestions?
> <snip config: it does permit caching **if allowed**>
>>
>>
>> http://www.google.com/calendar/feeds/8vm1eh2d7rcltngl0omsa4qc3g%40group.calendar.google.com/private/full?
>> - DIRECT/208.67.217.230 text/html
>> http://www.google.com/calendar/feeds/8vm1eh2d7rcltngl0omsa4qc3g%40group.calendar.google.com/private/full?
>> - DIRECT/208.67.217.230 text/xml
>> 1236865766.338 991 66.78.98.25 TCP_MISS/200 33924 GET
>> http://www.maximumsignal.ca/store/product.php? -
>> DIRECT/209.200.251.37 text/html
>> 1236865766.993 546 66.78.98.25 TCP_MISS/200 2435 GET
>> http://www.maximumsignal.ca/store/antibot_image.php? -
>> DIRECT/209.200.251.37 image/png
>
> Well, thanks for these. I ran a few of the pages through the
> cacheability engine:
> http://www.ircache.net/cgi-bin/cacheability.py
>
> And got this most of the time:
> Expires 606 weeks 5 days ago (Sat, 26 Jul 1997 05:00:00 GMT)
> Cache-Control post-check=0, must-revalidate, no-store,
> no-cache, pre-check=0
> Last-Modified now (Fri, 13 Mar 2009 01:08:54 GMT)
>
> "
> * This object has been deliberately marked stale.
> * The object had changed when validation was attempted.
> * It will be revalidated on every hit, because it has a
> Cache-Control: no-cache header.
> * It won't be cached at all, because it has a Cache-Control: no-store
> header.
> * Because of the must-revalidate header, all caches will strictly
> adhere to any freshness information you set.
> * It doesn't have a Content-Length header present, so it can't be
> used in a HTTP/1.0 persistent connection.
> "
>
> Sometimes they also have:
> "
> * This object requests that a Cookie be set; this makes it and other
> pages affected automatically stale; clients must check them upon every
> request.
> "
>
> Only the images seem to be cachable and only by cache with reasonable
> refresh_patterns set.
>
> Amos
>
>>
>>
>> Amos Jeffries wrote:
>>>> Here is the config ... it does work fine in "transparent" mode just
>>>> not
>>>> tproxy mode
>>>>
>>>> acl manager proto cache_object
>>>> acl localhost src 127.0.0.1/32
>>>> acl to_localhost dst 127.0.0.0/8
>>>>
>>>
>>>
>>>> acl localnet src 66.78.96.0/19
>>>> acl localnet src 64.235.192.0/19
>>>> acl localnet src 72.0.192.0/19
>>>> acl localnet src 192.168.1.0/24
>>>> acl localnet src 192.168.254.0/24
>>>>
>>>> acl QUERY urlpath_regex cgi-bin \?
>>>> cache deny QUERY
>>>>
>>>
>>> You can drop the above. It's handled by the refresh_patterns.
>>>
>>>
>>>> hierarchy_stoplist cgi-bin ?
>>>>
>>>> acl directurls url_regex -i "/etc/squid3/direct-urls"
>>>> cache deny directurls
>>>>
>>>
>>> You are not testing with one of these URLs are you?
>>>
>>>
>>>> cache deny localnet
>>>>
>>>
>>> This is definately causing MISS. Your http_access rules below prevent
>>> non-localnet access. Therefore every received request will hit this
>>> cache
>>> prevention.
>>>
>>> You want "cache allow all" at the end of the cache list.
>>>
>>>
>>>> always_direct allow directurls
>>>> always_direct allow localnet
>>>>
>>>> acl SSL_ports port 443
>>>> acl Safe_ports port 80 # http
>>>> acl Safe_ports port 21 # ftp
>>>> acl Safe_ports port 443 # https
>>>> acl Safe_ports port 70 # gopher
>>>> acl Safe_ports port 210 # wais
>>>> acl Safe_ports port 1025-65535 # unregistered ports
>>>> acl Safe_ports port 280 # http-mgmt
>>>> acl Safe_ports port 488 # gss-http
>>>> acl Safe_ports port 591 # filemaker
>>>> acl Safe_ports port 777 # multiling http
>>>> acl CONNECT method CONNECT
>>>>
>>>> http_access allow manager localhost
>>>> http_access deny manager
>>>> http_access deny !Safe_ports
>>>> http_access deny CONNECT !SSL_ports
>>>> http_access deny to_localhost
>>>> http_access allow localnet
>>>> http_access allow localhost
>>>> http_access deny all
>>>> icp_access allow localnet
>>>> htcp_access allow localnet
>>>> icp_access deny all
>>>> htcp_access deny all
>>>> htcp_clr_access deny all
>>>> ident_lookup_access deny all
>>>>
>>>> http_port 66.78.102.2:3128
>>>> http_port 66.78.102.2:3129 tproxy
>>>>
>>>> cache_mgr support_at_ripnet.com
>>>>
>>>> acl snmp snmp_community s64hf2
>>>> snmp_access allow snmp all
>>>>
>>>> snmp_port 3401
>>>> snmp_incoming_address 192.168.1.8
>>>> snmp_outgoing_address 192.168.1.8
>>>>
>>>> shutdown_lifetime 10 seconds
>>>> pid_filename /var/run/squid3.pid
>>>> mime_table /usr/share/squid3/mime.conf
>>>> icon_directory /usr/share/squid3/icons
>>>>
>>>
>>>
>>>> error_directory /usr/share/squid3/errors/en
>>>>
>>>
>>> :( I would recommend removing that for a better experience by your
>>> users.
>>> (allows errors to appear in the visitors own language)
>>>
>>>
>>>> cache_effective_user proxy
>>>> ignore_unknown_nameservers on
>>>> dns_nameservers 66.78.99.4 66.78.99.5
>>>>
>>>>
>>> <snip>
>>>
>>>> refresh_pattern -i \.index.(html|htm)$ 0 40% 10080
>>>>
>>>
>>> Double check this pattern you have files ending in:
>>> '.index' any-character 'htm' optional-'l'
>>>
>>> I think you maybe meant:
>>> -i .index\.(html|htm)$
>>>
>>>
>>>> refresh_pattern -i \.(html|htm|css|js)$ 1440 40% 40320
>>>> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
>>>> refresh_pattern . 0 40% 40320
>>>>
>>>>
>>>>
>>>> Amos Jeffries wrote:
>>>>
>>>>>> I am using squid 3.1.0.6
>>>>>>
>>>>>> If I check the disk free while the cache is running I do not see
>>>>>> any of
>>>>>> my cache directories incrementing at all.
>>>>>>
>>>>>> Any ideas?
>>>>>>
>>>>>>
>>>>> Okay, so much for the easy answer. We will have to see your config to
>>>>> tell
>>>>> why its not caching.
>>>>>
>>>>> Amos
>>>>>
>>>>>
>>>>>
>>>>>> Amos Jeffries wrote:
>>>>>>
>>>>>>
>>>>>>> Jamie Orzechowski wrote:
>>>>>>>
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> My post does not seem to be going to the list. Wondering if
>>>>>>>> you have
>>>>>>>> any ideas?
>>>>>>>>
>>>>>>>> I think I have TPROXY working but running into some issues.
>>>>>>>> Checking my logs all my traffic shows up as a TCP_MISS
>>>>>>>>
>>>>>>>>
>>>>>>> Squid 3.1.0.3? it has a storage problem that can show like this.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> 1236698452.579 79 66.78.98.194 TCP_MISS/200 542 GET
>>>>>>>> http://l1.zedo.com//log/p.gif? - DIRECT/72.247.244.10 image/gif
>>>>>>>> 1236698452.634 293 66.78.98.194 TCP_MISS/200 4972 GET
>>>>>>>> http://blstb.msn.com/i/9B/DDD13A38CB8B34F4DFA3F7BFFF71.jpg -
>>>>>>>> DIRECT/192.221.114.124 image/jpeg
>>>>>>>> 1236698452.878 100 66.78.98.194 TCP_MISS/200 1076 GET
>>>>>>>> http://h.foxsports.com/HG? - DIRECT/64.154.81.231 image/gif
>>>>>>>> 1236698453.367 252 66.78.98.194 TCP_MISS/200 1368 GET
>>>>>>>> http://www.myinternetservices.com/live/visitor/index.php? -
>>>>>>>> DIRECT/72.232.167.111 image/gif
>>>>>>>> 1236698454.087 13 66.78.98.194 TCP_MISS/200 812 GET
>>>>>>>> http://weyedata.pelmorex.com/WeatherEye/ObsData/CAON0090.xml -
>>>>>>>> DIRECT/207.96.160.37 text/xml
>>>>>>>> 1236698455.251 116 66.78.98.194 TCP_MISS/200 1368 GET
>>>>>>>> http://www.myinternetservices.com/live/visitor/index.php? -
>>>>>>>> DIRECT/72.232.167.111 image/gif
>>>>>>>> 1236698456.570 6451 66.78.98.194 TCP_MISS/200 45898 GET
>>>>>>>> http://www.facebook.com/profile.php? - DIRECT/69.63.176.140
>>>>>>>> text/html
>>>>>>>> 1236698456.876 77 66.78.98.194 TCP_MISS/200 2765 GET
>>>>>>>> http://profile.ak.facebook.com/v227/2005/50/q638320646_36.jpg -
>>>>>>>> DIRECT/209.170.91.178 image/jpeg
>>>>>>>>
>>>>>>>> My iptables is the following
>>>>>>>>
>>>>>>>> echo 1 > /proc/sys/net/ipv4/ip_forward
>>>>>>>> /sbin/iptables -t mangle -N DIVERT
>>>>>>>> /sbin/iptables -t mangle -A DIVERT -j MARK --set-mark 1
>>>>>>>> /sbin/iptables -t mangle -A DIVERT -j ACCEPT
>>>>>>>> /sbin/iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
>>>>>>>> /sbin/iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY
>>>>>>>> --tproxy-mark 0x1/0x1 --on-port 3129
>>>>>>>> //
>>>>>>>> any idea why I am not getting any TCP_HITS? ...
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Amos
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> =-=-=-=-=-=-=-=-=-=-=-=-=
>>>>>> Jamie Orzechowski - CCNA
>>>>>> RipNET Ltd. System/Network Administrator
>>>>>> Tel.: 613-342-3946 x294
>>>>>>
>>>>>>
>>>>>> THIS MESSAGE IS INTENDED ONLY FOR THE ADDRESSEE,
>>>>>> IT MAY CONTAIN PRIVILEGED OR CONFIDENTIAL INFORMATION.
>>>>>> ANY UNAUTHORIZED DISCLOSURE IS STRICTLY PROHIBITED.
>>>>>> IF YOU HAVE RECEIVED THIS MESSAGE IN ERROR,
>>>>>> PLEASE NOTIFY ME IMMEDIATELY SO THAT I MAY CORRECT MY
>>>>>> INTERNAL RECORDS. PLEASE THEN DELETE THE ORIGINAL MESSAGE.
>>>>>> =-=-=-=-=-=-=-=-=-=-=-=-=
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>
>
Received on Fri Mar 13 2009 - 11:55:25 MDT