Re: [squid-users] Squid proxy is very slow for web browsing in "near default" config

Dave T wrote:
> Thank you. Comments inline.
>
> On Sun, Jan 10, 2010 at 5:49 PM, Amos Jeffries <squid3_at_treenet.co.nz> wrote:
>> Dave T wrote:
>>> I just set up squid for the first time. It is on a Ubuntu box hosted
>>> on Linode.com. I have zero experience with proxy servers. I used this
>>> guide:
>>> http://news.softpedia.com/news/Seting-Up-a-HTTP-Proxy-Server-with-Authentication-and-Filtering-52467.shtml
>> Eeek! That tutorial is advising people to create open proxies for global public access (allow all).
>
> I think that is just for initial testing. The tutorial actually
> changes that in the second step.
>
>>
>>> (I also looked at a few other guides such as this one:
>>> http://ubuntuforums.org/showthread.php?t=320733. However, I wanted to
>>> most barebones config to start with and the link I used was the
>>> simplest I found.)
>> The simplest and safest documentation is in:
>> /usr/share/doc/squid-common/QUICKSTART
>> or
>> /usr/share/doc/squid3-common/QUICKSTART
>>
>> ... which outlines the minimal config changes to go from a clean install of your particular version to a working proxy.
>
> Thanks. Amazing that I looked everywhere else but on my local HDD. :)
>>
>>> So now that I have it set up, I'm testing it with FoxyProxy. It is not
>>> working well. Many web pages do not load completely. Some load very
>>> slowly. A few load fast (but even then, some images are often
>>> missing). Many times I have to try an address several times before a
>>> page will even start to load.
>>>
>>> I am using iptables. When I turn the firewall off, I have slightly
>>> less problems, but nothing significantly changes. I don't want to
>>> leave the firewall off, so I took a few ideas from here:
>>> http://www.cyberciti.biz/tips/linux-setup-transparent-proxy-squid-howto.html
>>> But the changes I put in actually made the performance a little worse
>>> than before. And like I said, even with the firewall off, the problems
>>> I described remain.
>>>
>>> What should I look at next to begin to understand my problem? Thanks.
>> Coming here was a good start.
>>
>> We are going to need to known the version of Squid you are using, there are a dozen or more available on Ubuntu.
>>
> I assume this will give more than enough info:
>
> $ dpkg -s squid
<snip>
> Version: 2.6.18-1ubuntu3
<snip>
>
> Linux Linode01 2.6.18.8-linode19 #1 SMP Mon Aug 17 22:19:18 UTC 2009
> i686 GNU/Linux
>

Excellent.

A little old, there are some recent config alterations we recommend. I'm
adding the ones 2.6 can use inline with your config below.

>
>> Also, we are going to have to see what squid.conf you have ended up working with. Minus the documentation comments and empty lines please.
>
> Here is what I am using for TESTING only. I was getting TCP_DENIED/407
> errors in the log, so I made an attempt to test it with no auth
> required at all. (Not sure if I achieved that with this config or not,
> but the problems didn't go away.)
>
> acl all src 0.0.0.0/0.0.0.0

all src all

> acl manager proto cache_object
> acl localhost src 127.0.0.1/255.255.255.255

acl localhost src 127.0.0.1

> acl to_localhost dst 127.0.0.0/8

acl to_localhost dst 127.0.0.0/8 0.0.0.0/32

> acl purge method PURGE
> acl CONNECT method CONNECT

NP: For non-testing use you will need to re-add the Safe_ports and
SSL_ports security controls here.
They are the safety nets that prevent people, particularly infected
clients, from opening tunnels via the proxy and sending spam or worse.

> http_access allow all

replace the above http_access line with:

  # alter to match your LAN range(s) currently allowed to use the proxy.
  acl localnet src 192.168.0.0/16
  http_access allow localnet
  http_access deny all

> icp_access allow all

NP: you probably want icp_access to be limited to local LAN same as
http_access is above.

> http_port 3128
> hierarchy_stoplist cgi-bin ?
> access_log /var/log/squid/access.log squid
> acl QUERY urlpath_regex cgi-bin \?
> cache deny QUERY
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440

refres_pattern -i (/cgi-bin/|\?) 0 0% 0

> refresh_pattern . 0 20% 4320
> acl apache rep_header Server ^Apache
> broken_vary_encoding allow apache
> extension_methods REPORT MERGE MKACTIVITY CHECKOUT
> hosts_file /etc/hosts
> coredump_dir /var/spool/squid

 From the above config the only thing I can see that would potentially
cause your problems are:
  * bad choice of test website being loaded. If its cache unfriendly (ie
youtube, facebook, google) you can see those symptoms.

  * available storage memory is full (cache_mem), the fast objects you
see are from there or disk. slow ones fetched from the Internet.

  * 2.6 without a cache_dir specified uses a 100MB storage cache in /var
somewhere. If the HDD is slow or full that can cause lag.
   --> this is solved by explicitly adding a cache_dir, for memory-only
use "cache_dir null /tmp" and allocate a larger cache_mem value

  * DNS server issues. If one of the servers configured for the squid
box is overloaded or dead it can delay Squid fetches badly.

  * someone has already found the proxy and is abusing it. This visibly
manifests as slow traffic with clients favourite objects being shoved
out of storage early.

I'm assuming you removed the transparent proxy settings from iptables?
If they are still there they will be causing traffic loops which can lag
the times and prevent some items from loading at all.

Look to access.log for what requests are passing through the proxy and
where they are being fetched from.

Look to cache.log with "debug_options ALL,1" in squid.conf to see if
there are any warnings that might be relevant. Ideally that should run
completely silent, but occasional warnings over a period of days or
weeks are to be expected.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE21
   Current Beta Squid 3.1.0.15