[squid-users] Caching files from Amazon S3

Hi,

We are stuck with a unique situation and require some help.

We are trying to use squid as a reverse proxy. I have been able to configure
it to cache normal images such as abc.gif residing on the local hard drive.

I run the webserver on a virtual ip and have squid running our internet
facing ip. The request comes to squid from the client and it sees if it has
the relevant files in cacche otherwise it talks to the webserver on the
backend and fetches it.

The issue is that we want to hide where the images are coming from since
they can come from S3 or our local system. So we used mod rewrite and wrote
a rule which would pass this URL
http://oursite.com/Utilities/feeds/getimages/display/thumb/142358.jpg from
the client to squid and from squid to apache. Apache then passes this string
to getimages.php? where we break this string, decode it and then return the
jpg file.

Our response is an http response given below that the client receives

!!!! ----Header Recv----
HTTP/1.0 206 Partial Content
Date: Sat, 15 Mar 2008 16:42:21 GMT
Server: Apache/2.2.6 (Fedora)
X-Powered-By: PHP/5.2.4
Cache-Control: private, must-revalidate, max-age=0
Last-Modified: Sat, 15 Mar 2008 16:42:21 GMT
Accept-Ranges: bytes
ETag: "9518df2e0531022b1413d21978d6fb80"
Content-Type: image/jpeg
Content-Range: bytes 0-14673899/14673900
Content-Length: 14673900
X-Cache: MISS from ratchet.vistaclick.lahore
X-Cache-Lookup: MISS from ratchet.vistaclick.lahore:80
Via: 1.0 ratchet.vistaclick.lahore:80 (squid/2.6.STABLE17)
Connection: close

the issue is that when I look at the squid access log against this response
it is as follows when we press F5 :

1205517662.101 81 192.168.0.241 TCP_MISS/200 16932 GET
http://oursite.com/Utilities/feeds/getimages/display/thumb/142361.jpg -
FIRST_UP_PARENT/192.168.0.237 image/jpeg.

we get a TCP_MISS/304 when we try with CTRL F5

Now the issue is that we have manipulated the http headers as much as we can
using the php http library however we cannot get squid to cache these
images. It caches normal images just fine however when we process the image
though our getimages.php script we only get the entries above in the log
file.

Please let me know if there is anyway to make squid images that we process
through PHP.

Any help you can offer would be greatly appreciated by a squid newbie :)

--
Abidoon
SQUID CONFIGURATION FILE BELOW
------------------------------------------------------------------------------------------------------------------
# $Id: squid.conf,v 1.1 2005/07/16 22:24:57 jmates Exp $
#visible_hostname
#http_port 80  vhost
http_port 192.168.0.236:80  defaultsite=www.mysite.com vhost
cache_peer 192.168.0.237  parent 80 0 no-query originserver
maximum_object_size 15360 KB
# minimum_object_size 0 KB
maximum_object_size_in_memory 1500 KB
cache_replacement_policy lru
memory_replacement_policy lru
logfile_rotate 3
cache_log none
cache_store_log none
cache_access_log /var/log/squid/access_log
# use local caching name server, avoid /etc/hosts
dns_nameservers 208.67.222.222
hosts_file none
#auth_param basic children 5
#auth_param basic realm Squid proxy-caching web server
#auth_param basic credentialsttl 2 hours
request_header_max_size 16 KB
negative_ttl 1 minutes
negative_dns_ttl 1 minutes
# connect_timeout 1 minutes
# peer_connect_timeout 30 seconds
# read_timeout 10 minutes
# request_timeout 5 minutes
# persistent_request_timeout 2 minute
# half_closed_clients on
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443 563     # https, snews
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
#http_access deny manager
# Deny requests to unknown ports
#http_access deny !Safe_ports
# Deny CONNECT to other than SSL ports
#http_access deny CONNECT !SSL_ports
#http_access deny to_localhost
#acl okdomains dstdomain www2.noog.com
#http_access deny !to_localhost
http_access allow all
# And finally deny all other access to this proxy
#http_access deny all
http_reply_access allow all
#icp_access deny all
miss_access allow all
ident_lookup_access deny all
cache_mgr webmaster@mysite.com
cache_effective_user squid
#httpd_accel_host 192.168.0.5
#httpd_accel_port 81
#httpd_accel_uses_host_header off
#httpd_accel_single_host on
#httpd_accel_with_proxy on
forwarded_for on
log_icp_queries on
#snmp_port 0
#snmp_access deny all
# offline_mode off
coredump_dir none
pipeline_prefetch on
mime_table /etc/squid/mime.conf