Re: [squid-users] Re: Should I see a massive slowdown when chaining squid => privoxy

On 05/06/11 11:55, Harry Putnam wrote:
> Amos Jeffries<squid3_at_treenet.co.nz> writes:
>
> [...]
>
> Harry wrote (summarized -ed hp):
> Adding squid and privoxy into a proxy setup seems to really really slow
> down my browsing as compared to browsing with a direct connection. (no proxy)
>
> And asks if this is normal.
>
> [...]
>
>> Speed gain/loss/other depends on what you are moving from.
>>
>> MORE IMPORTANTLY: how you define "slow"!
>>
>> Keep in mind that you also now have around 2x the processing going on
>> with 2 proxies. The difference added by Squid can be at least
>> 10ms. Some people call that noticeable slowdown. Some dont care about
>> anything less than a second.
>
> I'm guessing its more than seconds slower.... but not really sure how
> to gage the difference reliably, so as not to be giving flawed
> information here or have difference due to caching or something.
>
> Can you suggest a method to arrive at a fairly good comparison?

The "developer tools" in any NetKit browser (Safari, Chrome, Gecko etc)
or FireBug in Firefox or Iceweasel browsers shows a wealth of timing
information on the network analysis panel. One forced load with and one
without will tell you pretty clearly what the timing and speed
difference is. And whether its page processing or transfer lag.

>
>> * 3.1 is about 10-20% slower than the latest 2.7 on the same
>> config. With the older versions of 3.1 being on the slower end of that
>> scale as we work to optimize and fix things throughout the series.
>
> Wouldn't 20% be noticeable? So you're saying to back down a few
> versions for now?

Possibly. It varies a lot on where and how you run Squid. The "slow"
here is in CPU time cycles. So if your CPU is maxed out you see it as
real seconds, if you have spare CPU it should not be noticeably different.

>
>> * Moving to Squid from a non-proxy setup can be a major drop down
>> depending on the browser age. The browsers themselves drop the
>> parallel fetch rate from hundreds down to under 10. Browser tweaking
>> is the only way to avoid this.
>
> I'm using firefox 4.X on all home lan machines (that have a gui).
>
> Can you recommend some documentation that might help with what you
> called `Browser tweaking', I've never done anything special to a
> browser other than add or subtract add-on tools.
>
>> * Moving from browser->privoxy to a browser->squid->privoxy setup you
>> should have seen only a small drop. Some possibilities are Squid using
>> slow disks (maybe RAID), or Squid box is swapping, or the bandwidth is
>> being routed down the same physical links to/from Squid.
>
> No raid, and the hardware of the server is P4 Intel(R) Celeron(R) CPU
> 3.06GHz and 2GBram running on oldish IDE discs
>
> ------- --------- ---=--- --------- --------
>
> Probably should have included some questions about the squid.conf and
> privoxy/config in the first post. Maybe there are things in there
> that are not good left as default.
>
> I realize that both tools have several config files. I've left all
> but the two main ones in default state and have posted my working
> /etc/squid/squid.conf and /etc/privoxy/config
>
> There is a lot of debris in the comments but still I thought it might
> be useful to leave it all in for jogging memories...But also included
> a way to prune the comments by changing the name of the cgi script
> at the end of the URL from `disp.cgi' to `strp.cgi'.
>
> Any coaching would be well appreciated.
>

Looks good. :)

The audit results so far:

  * patch is reversed. Please diff the files the other way around on the
next one :)

  * "_SQUID_INLINE_ static" is not as portable as we would like. So we
have to avoid it for now.

  * instead of debugs() level 0 or 1 we have a macro DBG_CRITICAL or
DBG_IMPORTANT to indicate how bad the problem is. Most of what you have
at level 28,0 only needs 28,DBG_IMPORTANT.

  * rather than naming the function all over the place please use the
macro HERE to start the debugs text in debugs level 2 and higher.
   * also, stuff important enough for level 0 and 1 is usually user
visible and should be describing a problem+solution clearly enough not
to need the internal function name.

compileRE():
  * the bad pattern error message seems to work better like this:

   debugs(28, DBG_IMPORTANT, "WARNING: Skipping invalid regex in " <<
          cfg_filename << " line " << config_lineno << ": '" <<
          config_input_line << "': " << errbuf);
   debugs(28, 7, HERE << "compiled regex:'" << RE << "': " << errbuf);

compileOptimisedREs():
  * the if (RElen > BUFSIZ-1) case is done by aclParseRegexList(). It is
redundant here.

  * you follow "if (RElen + largeREindex + 3 < BUFSIZ-1)" with four ++
operations and assignments. Can the fourth overflow the buffer?

  * at the end of the if-elif-else sequence you state "do the loop again
to add the RE to largeRE" then continue;.
   What is that comment meaning? the compiled pattern is used as prefix
for further appended patterns?

compileUnoptimisedREs():
  * the if (RElen > BUFSIZ-1) case is done by aclParseRegexList(). It is
redundant here.

aclParseRegexList():
  * s/or use rules without expressions like 'dstdomain'/
       or use urlpath_regex with 'dstdomain' ACL/

Some further optimizations can be done:
  easy: ".?" can also be stripped along with ".*"

  less easy: this strips the prefixes, but what about the suffixes?

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.12
   Beta testers wanted for 3.2.0.8 and 3.1.12.2