Re: [squid-users] store_url_rewriting on squid 2.7 completed. thanks all (there is much more to do)

On 22/02/11 16:41, Eliezer wrote:
> Thank amos and everyone that helped me.
>
> today i finished my youtube caching project using squid 3.1.11 and squid2.7
>
> it took me awhile to understand how everything works and all the weird
> stuff that i got while rewriting the address and other stuff.
> André Luiz dos Santos helped me very very much seeing some things that i
> haven't seen about some end-points\states on youtube and other video sites.
>
>
> i had a lot of trouble cause the urls for the videos are hell and i dont
> like regular expressions.
>
> i had an exam on java programming so to prepare for the test i played
> with java and made the store_url_rewrite program on java.
> the nice thing about this software is the simplicity of that i could
> make things understandable and make the debugging steps for problems
> much more effective.
>
> André gave me the nice link to his work on:
>
> http://code.google.com/p/youtube-cache/
>
> i prefer the store_url_rewrite cause it's a cache server and not video
> storage one so nginx and other stuff are nice but not exactly what i and
> many other want to achieve.
>
> problems i encounter on the way:
> - identifying youtube dynamic urls (there are many of the) patterns.
>
> - return =NULL logged as return from the storeurl_program
> the culprit was too much storeurl_rewrite_children
> using java storeurl_rewrite_children 2 is more then anyone can need and
> more then 15 causing the problem.
>
> - i got weird things like "rewritten url part one" + IP/- - GET -
> myip=IP myport=3128"second end of the url"
> until now i really dont know the cause for it and the only option is the
> my store_rewrite program was analysing wrong the input url
> and for that i have changed my code to make sure that the output is fixed.

Looks like debugging trace text being sent to stdout instead of stderr.

The NULL result may be related to this problem. Squid routinely handles
many hundreds of simultaneous helpers for some networks.

>
> - sometimes the youtube links are not the videos them-self but a
> redirection link to other domains and to avoid gettings "error loading"
> i needed to change the minimum object size to cache.
>
> - the best way i have found till now to make the cache reliable is to
> actually KNOW what the refresh rules are and all the acls system works.
> if you will add the correct rules you will benefit a lot.

:)

>
> - in the squid-cache site\wiki some of the explanations are old\not
> understable.
>
> - i didnt look that much but i am looking for a documented VISUAL IMAGE
> of the squid system such as in iptables in here:
> http://www.linuxhomenetworking.com/wiki/images/f/f0/Iptables.gif
> i would be glad to make one if there isn't.
>
> - i am looking for a way to extend an object cache lease\life span based
> on usage.
> is there a way to "mangle" a single object cache validity without
> releasing and reloading it into the cache?

The HTTP validation requests do this, extending the cached age
dynamically based on the difference between modification time and
validation time. refresh_pattern is how you tune the rate of
re-validation. To work nicely this requires the origin server to answer
validation checks with 304 responses as appropriate instead of always
re-sending a whole new object.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.11
   Beta testers wanted for 3.2.0.5