Re: [squid-users] transparent integration with proxy on router

Hi Amos,
  If some of these entries are incorrect other than the redirect rule,
it's unfortunately a bug with the pfSense module. pfSense uses a nice
little web GUI to create the squid.conf. I don't have the ability to
edit anything you see below except the rules I've defined are created
automatically by pfsense. If I overwrite it, I'll lose my changes when
it restarts. I'll take a look at the code as it's written in PHP and
submit a feature request with a patch to clean up a lot of the
boilerplate configuration. Thanks for all the help, you've been a great
resource to get this working quickly.

Todd

Amos Jeffries wrote:
> On Tue, 29 Sep 2009 13:22:58 +1300, Todd Nine <todd_at_spidertracks.co.nz>
> wrote:
>
>> Hi Amos,
>> Here is my squid.conf. I've just used the defaults and added a single
>> rule. We're pushing a lot of throughput (several gigs a day). I've
>> disabled writing to disk as we actually run from a USB appliance, and
>> set the cache size to 1 GB (1024M) of RAM. My main use of squid is not
>> caching, but rather http redirection to save us money on our usage fees
>> from our ISPs.
>>
>
> In which case you probably do want to look at caching. Since that can have
> between 20% and 40% reduction in HTTP traffic going to your ISPs.
>
>
> What version of squid is this?
>
>
>> Thanks again for the help!
>>
>> File:
>> # Do not edit manually !
>> http_port 10.0.1.1:3128
>> http_port 10.0.2.1:3128
>> http_port 127.0.0.1:80 transparent
>> icp_port 0
>>
>> pid_filename /var/run/squid.pid
>> cache_effective_user proxy
>> cache_effective_group proxy
>> error_directory /usr/local/etc/squid/errors/English
>> icon_directory /usr/local/etc/squid/icons
>> visible_hostname router
>> cache_mgr admin_at_localhost
>> access_log /var/squid/log/access.log
>> cache_log /var/squid/log/cache.log
>> cache_store_log none
>> shutdown_lifetime 3 seconds
>> # Allow local network(s) on interface(s)
>> acl localnet src 10.0.1.0/255.255.255.0 10.0.2.0/255.255.255.0
>>
>
> Please use CIDR masks:
> acl localnet src 10.0.1.0/24 10.0.2.0/24
>
>
>> uri_whitespace strip
>>
>> cache_dir aufs /var/squid/cache 100 16 256
>>
>
> Huh? you said you disabled writing to disk. That above is using a 100MB
> cache on the disk.
>
> To disable disk caching use the 'cache_dir null ' storage type in Squid
> older than 3.1, or remove all cache_dir from Squid-3.1+.
>
>
>> cache_mem 1024 MB
>> maximum_object_size 4 KB
>>
>
> 1GB worth of 4KB objects is a LOT of objects. If you have the memory to
> spare 1GB for caching its probably best to allow moderate sized objects to
> be cached in RAM. Setting Max size to 1MB should do. Though depending on
> the popularity of video sites with your users they may also benefit from
> 10MB max object size (video causes a bump at 2MB-8MB apparently).
>
>
>> minimum_object_size 0 KB
>> cache_replacement_policy heap LFUDA
>> memory_replacement_policy heap GDSF
>> offline_mode off
>> dns_children 32
>>
>
> The above is only relevant with the obsolete 'dnsserver' helper. If you are
> still using that you would get a great deal of performance boots by
> changing to the internal DNS (requires a recompile).
>
>
>> cache_swap_low 90
>> cache_swap_high 95
>> acl donotcache dstdomain "/var/squid/acl/donotcache.acl"
>> cache deny donotcache
>> # No redirector configured
>>
>>
>>
>> # Setup some default acls
>> acl all src 0.0.0.0/0.0.0.0
>> acl localhost src 127.0.0.1/255.255.255.255
>>
>
> Some more CIDR benefits:
> acl all src all
> acl localhost src 127.0.0.1
>
>
>> acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 1111
>> 3128 1025-65535
>> acl sslports port 443 563 1111
>> acl manager proto cache_object
>> acl purge method PURGE
>> acl connect method CONNECT
>> acl dynamic urlpath_regex cgi-bin \?
>> cache deny dynamic
>>
>
> More speed boost by dropping the above QUERY
>
>
>> http_access allow manager localhost
>>
>> http_access deny manager
>> http_access allow purge localhost
>> http_access deny purge
>> http_access deny !safeports
>> http_access deny CONNECT !sslports
>>
>> # Always allow localhost connections
>> http_access allow localhost
>>
>> request_body_max_size 0 KB
>> reply_body_max_size 0 allow all
>> delay_pools 1
>> delay_class 1 2
>> delay_parameters 1 -1/-1 -1/-1
>> delay_initial_bucket_level 100
>> delay_access 1 allow all
>>
>> # Allow local network(s) on interface(s)
>> http_access allow localnet
>> # Custom options
>> #Set up our ACL for high throughput sites
>> acl high_throughput dstdomain .amazonaws.com .rapidshare.com
>> #Bind high throughput to the wireless interface
>> tcp_outgoing_address 116.90.140.xx high_throughput
>>
>> # Default block all to be sure
>> http_access deny all
>>
>>
>>
>> Amos Jeffries wrote:
>>
>>> On Tue, 29 Sep 2009 09:32:49 +1300, Todd Nine <todd_at_spidertracks.co.nz>
>>> wrote:
>>>
>>>
>>>> Thanks for the help! I read over the rules and it was quite easy to
>>>>
> set
>
>>>> up what I needed once I had the right directive. I simply set up the
>>>> following.
>>>>
>>>> #Set up our ACL for high throughput sites
>>>> acl high_throughput dstdomain .amazonaws.com
>>>>
>>>> #Bind high throughput to the wireless interface
>>>> tcp_outgoing_address 116.90.140.xx high_throughput
>>>>
>>>> However we're having a side effect issue. Our router box is a bit old
>>>> (an old P4), and we can't keep up with the squid demands due to the
>>>> number of users with 2 GB of ram. Is there a directive that I can tell
>>>>
>
>
>>>> squid not to proxy connections unless they meet the "high_throughput"
>>>> acl? I looked and couldn't find any bypass directives that met what I
>>>> needed.
>>>>
>>>> Thanks,
>>>> Todd
>>>>
>>>>
>>> Once connections have already entered Squid its too late to not send
>>>
> them
>
>>> to Squid.
>>>
>>> I have run Squid on P4s routers with 256MB of RAM for hundreds of
>>>
> domains
>
>>> and dozens of clients without having the box run up much of a sweat.
>>>
> What
>
>>> is your load (both CPU box load, and visitor rates, bandwidth) like?
>>> Also check your other configuration and access controls are using
>>> efficient
>>> methods, if you don't know what those are already I'm happy to give
>>> configs
>>> an audit and point things that need adjusting out.
>>>
>>> Amos
>>>
>>>
>>>
>>>> Amos Jeffries wrote:
>>>>
>>>>
>>>>> On Mon, 28 Sep 2009 16:21:16 +1300, Todd Nine
>>>>>
> <todd_at_spidertracks.co.nz>
>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>> Hi all,
>>>>>> I'm using squid on a pfSense router we've built. We have 2
>>>>>> connections, one we pay for usage (DSL) and one we do not (Wireless).
>>>>>>
>
>
>>>>>> We use Amazon S3 extensively at work. We've been attempting to route
>>>>>>
>
>
>>>>>> all traffic over the wireless via an IP range, but as S3 can change
>>>>>>
>>>>>>
>>> IPs,
>>>
>>>
>>>>>> this doesn't work and we end up with a large bill for our DSL. Is it
>>>>>>
>
>
>>>>>> possible to have squid route connections via a specific interface if
>>>>>>
> a
>
>>>>>> hostname such as "amazonaws.com" is in the HTTP request header?
>>>>>>
>>>>>> Thanks,
>>>>>> Todd
>>>>>>
>>>>>>
>>>>>>
>>>>> Yes you can.
>>>>>
>>>>> Find an IP assigned to the interface you want traffic to go out. Use
>>>>> the
>>>>> tcp_outgoing_addr directive and ACLs that match the requests to make
>>>>>
>>>>>
>>> sure
>>>
>>>
>>>>> all the requests to that domain are assigned that outgoing address.
>>>>>
>>>>>
>>> Then
>>>
>>>
>>>>> make sure the OS sends traffic from that IP out the right interface.
>>>>>
>>>>> Amos
>>>>>
>>>>>
>>>>>

-- 
todd
SENIOR SOFTWARE ENGINEER
todd nine | spidertracks ltd |  117a the square
po box 5203 | palmerston north 4441 | new zealand
P: +64 6 353 3395 | M: +64 210 255 8576 
E: todd_at_spidertracks.co.nz <mailto:todd_at_spidertracks.co.nz> 
 W: www.spidertracks.com <http://www.spidertracks.com>