[squid-users] Large-scale Reverse Proxy for serving images FAST

All,

I'm new to Squid and I have been given the task of optimizing the delivery of photos from our website. We have 1 main active image server which serves up the images to the end user via 2 chained CDNs. We want to drop the middle CDN as it's not performing well and is a waste of money; in it's stead we plan to place a few reverse proxy web accelerators between the primary CDN and our image server.

We currently recieve 152 hits/sec on average with about 550hps max to our secondary CDN from cache misses at the Primary.
I would like to serve a lot of this content straight from memory to get it out there as fast as possible.

I've read around that there are memory and processing limitations in Squid in the magnitude of 2-4GB RAM and 1 core/1 thread, respectively. So, my solution was to run multiple instances, as we don't have the rackspace to scale this out otherwise.

I've managed to build a working config of 1:1 squid:origin, but I am having trouble scaling this up and out.

Here is what I have attempted to do, maybe someone can point me in the right direction:

Current config:
User Browser -> Prim CDN -> Sec CDN -> Our Image server @ http port 80

New config idea:
User -> Prim CDN -> Squid0 @ http :80 -> round-robin to "parent" squid instances on same machine @ http :81, :82, etc -> Our Image server @ http :80

Squid0's (per diagram above) squid.conf:

acl Safe_ports port 80
acl PICS_DOM_COM dstdomain pics.domain.com
acl SQUID_PEERS src 127.0.0.1
http_access allow PICS_DOM_COM
icp_access allow SQUID_PEERS
miss_access allow SQUID_PEERS
http_port 80 accel defaultsite=pics.domain.com
cache_peer localhost parent 81 3130 name=imgCache1 round-robin proxy-only
cache_peer localhost parent 82 3130 name=imgCache2 round-robin proxy-only
cache_peer_access imgCache1 allow PICS_DOM_COM
cache_peer_access imgCache2 allow PICS_DOM_COM
cache_mem 8192 MB
maximum_object_size_in_memory 100 KB
cache_dir aufs /usr/local/squid0/cache 1024 16 256 -- This one isn't really relevant, as nothing is being cached on this instance (proxy-only)
icp_port 3130
visible_hostname pics.domain.com/0

Everything else is per the defaults in squid.conf.

"Parent" squids' (from above diagram) squid.conf:

acl Safe_ports port 81
acl PICS_DOM_COM dstdomain pics.domain.com
acl SQUID_PEERS src 127.0.0.1
http_access allow PICS_DOM_COM
icp_access allow SQUID_PEERS
miss_access allow SQUID_PEERS
http_port 81 accel defaultsite=pics.domain.com
cache_peer 192.168.0.223 parent 80 0 no-query originserver name=imgParent
cache_peer localhost sibling 82 3130 name=imgCache2 proxy-only
cache_peer_access imgParent allow PICS_DOM_COM
cache_peer_access imgCache2 allow PICS_DOM_COM
cache_mem 8192 MB
maximum_object_size_in_memory 100 KB
cache_dir aufs /usr/local/squid1/cache 10240 16 256
visible_hostname pics.domain.com/1
icp_port 3130
icp_hit_stale on

Everything else per defaults.

So, when I run this config and test I see the following happen in the logs:

From "Squid0" I see that it resolves to grab the image from one of it's parent caches. This is great! (some show as "Timeout_first_up_parent" and others as just "first_up_parent")

1237253713.769 62 127.0.0.1 TCP_MISS/200 2544 GET http://pics.domain.com:81/thumbnails/59/78/45673695.jpg - TIMEOUT_FIRST_UP_PARENT/imgParent image/jpeg

From the parent cache that it resolves to, I see that it grabs the image from IT'S parent, originserver (our image server). Subsequent requests are 'TCP_HIT' or mem hit. Great stuff!

1237253713.769 62 127.0.0.1 TCP_MISS/200 2694 GET http://pics.domain.com/thumbnails/59/78/45673695.jpg - FIRST_PARENT_MISS/imgCache1 image/jpeg

Problem is, it doesn't round-robin the requests to both of my "parent" squids and you end up with a very 1-sided cache. If I stop the "parent" instance that is resolving the items, the second "parent" doesn't take over either. If I then proceed to restart the "Squid0" instance, it will then direct the requests to the second "parent", but then the first wont recieve any requests. So I know both "parent" configs work, but I must be doing something wrong somewhere, or is this all just a silly idea...?

Can anyone comment on the best way to run a high-traffic set of accel cache instances similar to this, or how to fix what i've tried to do? Or another way to put a LOT of data into a squid instance's memory. (We have ~150Million x 2KB images that are randomly requested).
I'd like to see different content cached on each instance with little or no overlap with round-robin handling which squid gets to cache an item and icp handling which squid has that item.

I'm open to other ideas too..

Sorry for the loooooong email.

Thanks all!

David

      __________________________________________________________________
Instant Messaging, free SMS, sharing photos and more... Try the new Yahoo! Canada Messenger at http://ca.beta.messenger.yahoo.com/
Received on Tue Mar 17 2009 - 04:31:17 MDT