Re: [squid-users] Not routing from tproxy to internet

On 6/02/2013 5:51 a.m., Paris Charalampou wrote:
> Hi,
>
>
> On 5/2/2013 16:39, Amos Jeffries wrote:
>> On 6/02/2013 1:02 a.m., Paris Charalampou wrote:
>>> Hi,
>>>
>>> I have the following problem while trying to setup a squid in tproxy
>>> mode for a centralized solution. Our topology is:
>>>
>>> client -> R1 -> squid -> R1 -> Internet
>>>
>>> Note that server uses a public ip address which is the same for incoming
>>> and outgoing connections
>> Note that TPROXY hides the Squid server from both client and Internet
>> servers. So its IP is irrelevant here. Case in point is that its
>> receiving IP is 127.0.0.1 in your configuration (how could clients or
>> servers send/receive traffic directly to another machines 127.0.0.1?).
>>
>>> I am using squid 3.2.6 [tried3.2.7 without luck] in Debian Wheezy with
>>> linux kernel 3.2 and iptables 1.4.14
>>>
>>> Squid.conf
>>> --
>>> workers 1
>>> debug_options ALL,4
>>>
>>> client_db off
>>>
>>> http_port 80 tproxy
>> 80 is not a good port for this. Make is something random and firewall
>> REJECT all external traffic to that port in the mangle table. That
>> will prevent some traffic loops caused by external routing, and
>> several types of malicious attacks.
>>
> I modified configuration and made squid listen on 22122 and the public
> ip address
>>> http_port 8080
>>> icp_port 0
>>>
>>>
>>> iptables
>>> ---
>>> iptables -t mangle -F
>>> iptables -t mangle -N DIVERT
>>> iptables -t mangle -A DIVERT -i lo -j ACCEPT
>>> iptables -t mangle -A DIVERT -j MARK --set-mark 1
>>> iptables -t mangle -A DIVERT -j ACCEPT
>>> iptables -t mangle -A PREROUTING -s <server_ip_address> -j ACCEPT
>>> iptables -t mangle -A PREROUTING -i lo -j ACCEPT
>>> iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
>>> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY
>>> --tproxy-mark 0x1/0x1 --on-ip 127.0.0.1 --on-port 80
>>>
>>>
> Change the iptables to use the real ip address
>
> iptables-save
> # Generated by iptables-save v1.4.14 on Tue Feb 5 20:05:23 2013
> *mangle
> :PREROUTING ACCEPT [13:813]
> :INPUT ACCEPT [421:27597]
> :FORWARD ACCEPT [0:0]
> :OUTPUT ACCEPT [631:222775]
> :POSTROUTING ACCEPT [631:222775]
> :DIVERT - [0:0]
> -A PREROUTING -i eth1 -p tcp -m socket -j DIVERT
> -A PREROUTING -i eth1 -p tcp -m tcp --dport 80 -j TPROXY --on-port 22122
> --on-ip 194.xx.xx.xx --tproxy-mark 0x1/0x1
> -A DIVERT -j MARK --set-xmark 0x1/0xffffffff
> -A DIVERT -i eth1 -j ACCEPT
> COMMIT
> # Completed on Tue Feb 5 20:05:23 2013
> # Generated by iptables-save v1.4.14 on Tue Feb 5 20:05:23 2013
> *filter
> :INPUT ACCEPT [111856:6310952]
> :FORWARD ACCEPT [3096:2131970]
> :OUTPUT ACCEPT [227190:44369859]
> COMMIT
> # Completed on Tue Feb 5 20:05:23 2013
> # Generated by iptables-save v1.4.14 on Tue Feb 5 20:05:23 2013
> *nat
> :PREROUTING ACCEPT [3482:448137]
> :INPUT ACCEPT [3477:447757]
> :OUTPUT ACCEPT [62:6605]
> :POSTROUTING ACCEPT [62:6605]
> COMMIT
> # Completed on Tue Feb 5 20:05:23 2013
>>> Do I have to use WCCP on the router standing between theproxy andthe
>>> internet??
>> No. WCCP is separate to TPROXY. Why do you ask? is it involved elsewhere?
>>
> No,we are not using WCCP at the time being. R1 in our topology uses
> cisco IP-policy with route-maps to redirect http traffic to the squid boxes.
>
>>> I cannot route the packets from the proxy tothe edge router
>>> while I can see incoming http packets from the clients. All incoming
>>> connection timeout while waiting for the reply from destination server
>> Why not? route all packets normally on the Squid box as if it were a
>> simple relay. The TPROXY rules above are the only abnormal part.
>>
>> PS. take care that rp_filter and similar filtering limitations are
>> permitting the Squid box to emit packets from external machines IPs.
> rp_filter is disabled. I have the following output in cache.log when
> running in debug mode
>
> 2013/02/05 19:49:15.600 kid1| neighbors.cc(288) getFirstUpParent:
> getFirstUpParent: returning NULL
> 2013/02/05 19:49:15.600 kid1| neighbors.cc(464) getDefaultParent:
> getDefaultParent: returning NULL
> 2013/02/05 19:49:15.600 kid1| neighbors.cc(464) getDefaultParent:
> getDefaultParent: returning NULL
> 2013/02/05 19:49:15.600 kid1| peer_select.cc(298) peerSelectDnsPaths:
> Found sources for 'http://www.<some_site>.gr/'
> 2013/02/05 19:49:15.600 kid1| peer_select.cc(299) peerSelectDnsPaths:
> always_direct = 0
> 2013/02/05 19:49:15.600 kid1| peer_select.cc(300) peerSelectDnsPaths:
> never_direct = 0
> 2013/02/05 19:49:15.600 kid1| peer_select.cc(303)
> peerSelectDnsPaths: DIRECT = local=81.186.x.x
> remote=85.25.x.x:80 flags=25
> 2013/02/05 19:49:15.600 kid1| peer_select.cc(311)
> peerSelectDnsPaths: timedout = 0
> 2013/02/05 19:49:15.600 kid1| forward.cc(337) startConnectionOrFail:
> http://www.<some_site>.gr/
> 2013/02/05 19:49:15.600 kid1| forward.cc(858) connectStart:
> fwdConnectStart: http://www.<some_site>.gr/
> 2013/02/05 19:49:15.600 kid1| pconn.cc(435) pop: lookup for key
> {85.25.X.X:80/www.<some_site>.gr/} failed.
>
> None of the above being the ip address of the squid box.

Exactly. This is how TPROXY works. Fully Transparent right down to the
IP address level.

The NAT systems you are more familiar with change the packet then let
the system routing deliver it to the altered service which is then
required to figure out what NAT changed and undo it. With TPROXY the
packets are delivered straight to the Squid socket without being
altered. Squid receiving IP:port are only involved as labels used to
open the sockt, and inform iptables TPROXY target which socket to
deliver to.

The outgoing traffic from Squid->server spoofs the client IP address as
received on the client->Squid connection. As you can see in the log line:
      DIRECT = local=81.186.x.x remote=85.25.x.x:80 flags=25

Squid is using the client IP 81.186.x.x to open connections to server
85.25.x.x port 80. This is what your R2 router will be getting.

Your R1 router need to route packets with --dport 80 to the Squid box.
Your R2 router needs to route packets with --sport 80 to the Squid box.
All other packets can be left for normal routing between client and
Internet.

Amos
Received on Wed Feb 06 2013 - 06:15:39 MST