RE: [squid-users] Squid icap respmod is not caching

Hi!

So maybe some insight into our ICAP application:

It essentially acts as a proxy for mobile carriers (for handling MSISDNs and such) and has a similar functionality as some of those simple anonymizing proxies:
* it takes a request for http://a.b.c/www.squid-cache.org/default.css
* and the icap request-mod translates it to http://www.squid-cache.org/default.css
* then squid does the request
* then back to ICAP for response-modifications - modify the relevant links for images in the css or html,...
* then squid deliver it to the device

So we are running the following request:
$ curl -vvv --proxy 127.0.0.1:80 http://a.b.c/ www.squid-cache.org/default.css
* About to connect() to proxy 127.0.0.1 port 80 (#0)
* Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> GET http://a.b.c/www.squid-cache.org/default.css HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: a.b.c
> Accept: */*
> Proxy-Connection: Keep-Alive
>
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0< HTTP/1.1 200 OK
< Content-Type: text/css
< ETag: "1ea3a1f-e02-4557f2cff9ec0"
< Last-Modified: Thu, 28 Aug 2008 06:23:31 GMT
< Content-Length: 3586
< Accept-Ranges: bytes
< Server: Apache/2.2.17 (FreeBSD) mod_ssl/2.2.17 OpenSSL/0.9.8q DAV/2 SVN/1.6.15 PHP/5.3.3 with Suhosin-Patch
< Date: Tue, 16 Jul 2013 09:46:14 GMT
< Vary: Accept-Encoding
< X-Cache: MISS from a.b.c
< X-Cache-Lookup: MISS from a.b.c:3128
< Via: internet
< Connection: keep-alive
<
{ [data not shown]

We have executed this for the following versions of squid:
* 3.1.19 - works with caching
* 3.1.23 - works with caching
* 3.2.1 - no caching
* 3.2.7 - no caching
* 3.2.13 - no caching

Also the config has 2 ports open:
80 - reverse proxy only
3128 - for use as a general proxy as well

so if we run a request like this first:
$curl -vvv --proxy 127.0.0.1:3128 http://www.squid-cache.org/default.css > /dev/null

Then the subsequent:
$ curl -vvv --proxy 127.0.0.1:80 http://a.b.c/www.squid-cache.org/default.css
Delivers as HIT and does NOT pass thru response-modification phase again.
This means the image URLS are not modified in the css...

Then we have to clean the cache to work again

Here also the trace from ICAP communication (note that some "empty" lines may be missing in this report, due to the way we are extracting it from captures)

Squid->ICAP - Request-Mod:
-->REQMOD icap://127.0.0.1:1344/reqmod ICAP/1.0
-->Host: 127.0.0.1:1344
-->Date: Tue, 16 Jul 2013 09:59:44 GMT
-->Encapsulated: req-hdr=0, null-body=251
-->Allow: 204
-->X-Client-IP: 127.0.0.1
-->GET http://a.b.c/www.squid-cache.org/default.css HTTP/1.1
-->User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
-->Host: a.b.c
-->Accept: */*
ICAP->Squid - Request-Mod-response:
-->ICAP/1.0 200 OK
-->Date: Tue, 16 Jul 2013 09:59:44 GMT
-->Encapsulated: req-hdr=0,null-body=351
-->GET http://www.squid-cache.org/default.css http/1.1
-->host: www.squid-cache.org
-->x-mib-before: 1373968784137
-->x-mib-eventtype: BR
-->accept: */*
-->x-mib-request-uid: d50658b2-5448-4ba1-9be9-5cfcb5892da6
-->user-agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
-->x-mibinstance: BB
-->x-ignore: BB

Squid->ICAP - Response-Mod:
-->RESPMOD icap://127.0.0.1:1344/respmod ICAP/1.0
-->Host: 127.0.0.1:1344
-->Date: Tue, 16 Jul 2013 09:59:44 GMT
-->Encapsulated: req-hdr=0, res-hdr=351, res-body=665
-->Allow: 204
-->X-Client-IP: 127.0.0.1
-->GET http://www.squid-cache.org/default.css HTTP/1.1
-->Host: www.squid-cache.org
-->x-mib-before: 1373968784137
-->x-mib-eventtype: BR
-->Accept: */*
-->x-mib-request-uid: d50658b2-5448-4ba1-9be9-5cfcb5892da6
-->User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
-->x-mibinstance: BB
-->x-ignore: BB
-->
-->HTTP/1.1 200 OK
-->Date: Tue, 16 Jul 2013 09:59:44 GMT
-->Server: Apache/2.2.17 (FreeBSD) mod_ssl/2.2.17 OpenSSL/0.9.8q DAV/2 SVN/1.6.15 PHP/5.3.3 with Suhosin-Patch
-->Last-Modified: Thu, 28 Aug 2008 06:23:31 GMT
-->ETag: "1ea3a1f-e02-4557f2cff9ec0"
-->Accept-Ranges: bytes
-->Content-Length: 3586
-->Content-Type: text/css
-->
-->Design by Free CSS Templates
-->http://www.freecsstemplates.org
-->Released for free under a Creative Commons Attribution 2.5 License
-->font-family: "Trebuchet MS", Arial, Helvetica, sans-serif;
-->body {
...

ICAP->Squid - Response-Mod-response:
-->ICAP/1.0 200 OK
-->Date: Tue, 16 Jul 2013 09:59:44 GMT
-->Encapsulated: res-hdr=0,res-body=337
-->
-->HTTP/1.1 200 OK
-->content-type: text/css
-->etag: "1ea3a1f-e02-4557f2cff9ec0"
-->last-modified: Thu, 28 Aug 2008 06:23:31 GMT
-->content-length: 3586
-->accept-ranges: bytes
-->server: Apache/2.2.17 (FreeBSD) mod_ssl/2.2.17 OpenSSL/0.9.8q DAV/2 SVN/1.6.15 PHP/5.3.3 with Suhosin-Patch
-->date: Tue, 16 Jul 2013 09:59:44 GMT
-->vary: Accept-Encoding
-->
-->Design by Free CSS Templates
-->http://www.freecsstemplates.org
-->Released for free under a Creative Commons Attribution 2.5 License
-->font-family: "Trebuchet MS", Arial, Helvetica, sans-serif;
-->body {

Hope this gives enough information to understand the issue...

Thanks,
        Martin

-----Original Message-----
From: Eliezer Croitoru [mailto:eliezer_at_ngtech.co.il]
Sent: Dienstag, 25. Juni 2013 18:37
To: squid-users_at_squid-cache.org
Subject: Re: [squid-users] Squid icap respmod is not caching

Let say that on 3.2.0.1X it was working for me with ICAP respmod on with
youtube videos so it's not a RESPMOD problem unless...
There was a change in some of squid internals.
on squid 3.3 if i'm not wrong somebody told me that youtube caching
works but I have never seen that someone used my RESPMOD "no-store"
option so I am not 100% sure about it.
can you share the debug_options ALL,3 to understand what is going on?

What are you using ICAP for?

Eliezer

On 06/25/2013 02:17 PM, Paul Browne wrote:
> Hi,
>
> I will work on getting the headers / tcpdump & snoop logs) however just to be clear:
>
> 1) Squid 3.1 was caching our ICAP respmod responses and it was only when we went to the 3.2.7 version of squid did it fail to cache
>
> 2) curl is not an issue here, we tested it without respmod and it cached successfully,
>
> Regards
> -Paul
>
> -----Original Message-----
> From: Amos Jeffries [mailto:squid3_at_treenet.co.nz]
> Sent: 21 June 2013 01:28
> To: squid-users_at_squid-cache.org
> Subject: Re: [squid-users] Squid icap respmod is not caching
>
> On 21/06/2013 8:48 a.m., Eliezer Croitoru wrote:
>> ETAG???Vary??
>> There was a small talk about a bug in the vary thing But I am unsure
>> how it was tested.
>> Can we test this thing??
>
>
> And what HTTP status-line content is there?
>
> Also what headers is curl sending? curl in particular is known to force reload teh cache (which shows up as MISS).
> The request headers also can affect revalidation and the "X-Cache"
> headers do not report REFRESH/revalidate operations accurately.
>
> Unless we have the FULL set of request and respose headers paired together it is diffcult or impossible to identify what Squid is really doing.
>
> Amos
>
>
> This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,
> you may review at http://www.amdocs.com/email_disclaimer.asp
>
Received on Tue Jul 16 2013 - 10:23:40 MDT