[squid-users] Help! one more time on on Squid3.HEAD(20110307), TPROXY4 and Iptables 1.4.9 + ebtables

From: Jim Binder <jbinder_at_cyphort.com>
Date: Tue, 15 Mar 2011 00:22:05 -0700

Trying this one more time to see if anyone might know what's wrong in getting my transparent bridging with squid to work.
Config... pings work thought the box (the bridge is working however; the 3129 socket never pops with an HTTP request)

Admin on Eth1, Internet on eth0 and Inside (client) interface on eth2. Br0 used as the bridge.

Running Fedora core 14 (but went back as fare as 12 and couldn't get it to work)

Squid Cache: Version 3.HEAD-20110307
configure options: '--enable-ecap' '--enable-icap-client' '--enable-linux-netfilter' --enable-ltdl-convenience

iptables-1.4.9-1.fc14.i686
kernel-2.6.35.11-83.fc14.i686
ebtables-2.0.9-5.fc13.i686

Went as far to turn on dynamic debug logging and I don't see what's wrong but the connect never seems to get made to the 3129 socket.

[ 214.914113] TRACE: mangle:PREROUTING:rule:2 IN=eth2 OUT= MAC=00:40:f4:cd:01:70:00:50:56:36:df:78:08:00 SRC=192.168.1.91 DST=192.168.1.88 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=3380 DF PROTO=TCP SPT=48255 DPT=80 SEQ=1363486620 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A02522AA80000000001030306)
[ 214.914155] xt_TPROXY: redirecting: proto 6 c0a80158:80 -> 00000000:3129, mark: 1
[ 217.920783] TRACE: raw:PREROUTING:policy:3 IN=eth2 OUT= MAC=00:40:f4:cd:01:70:00:50:56:36:df:78:08:00 SRC=192.168.1.91 DST=192.168.1.88 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=3381 DF PROTO=TCP SPT=48255 DPT=80 SEQ=1363486620 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A025236680000000001030306)
[ 217.920846] TRACE: mangle:PREROUTING:rule:2 IN=eth2 OUT= MAC=00:40:f4:cd:01:70:00:50:56:36:df:78:08:00 SRC=192.168.1.91 DST=192.168.1.88 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=3381 DF PROTO=TCP SPT=48255 DPT=80 SEQ=1363486620 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A025236680000000001030306)
[ 217.920891] xt_TPROXY: redirecting: proto 6 c0a80158:80 -> 00000000:3129, mark: 1

[root_at_fw01 ~]# iptables -t raw -L -v; echo '------'; iptables -t mangle -L -v
Chain PREROUTING (policy ACCEPT 13966 packets, 5291K bytes)
 pkts bytes target prot opt in out source destination
   14 840 TRACE tcp -- any any anywhere anywhere tcp dpt:http
    4 208 TRACE tcp -- any any anywhere anywhere tcp spt:http

Chain OUTPUT (policy ACCEPT 11445 packets, 5781K bytes)
 pkts bytes target prot opt in out source destination
    0 0 TRACE tcp -- any any anywhere anywhere tcp dpt:http
    0 0 TRACE tcp -- any any anywhere anywhere tcp spt:http
------
Chain PREROUTING (policy ACCEPT 3843 packets, 4678K bytes)
 pkts bytes target prot opt in out source destination
10086 586K DIVERT tcp -- any any anywhere anywhere socket
   14 840 TPROXY tcp -- any any anywhere anywhere tcp dpt:http TPROXY redirect 0.0.0.0:3129 mark 0x1/0xffffffff

Chain INPUT (policy ACCEPT 10284 packets, 622K bytes)
 pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 19 packets, 25784 bytes)
 pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 11443 packets, 5780K bytes)
 pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 11483 packets, 5810K bytes)
 pkts bytes target prot opt in out source destination

Chain DIVERT (1 references)
 pkts bytes target prot opt in out source destination
10086 586K MARK all -- any any anywhere anywhere MARK set 0x1
10086 586K ACCEPT all -- any any anywhere anywhere
[root_at_fw01 ~]#

[root_at_fw01 ~]# ebtables -t broute -L --Lc
Bridge table: broute

Bridge chain: BROUTING, entries: 2, policy: ACCEPT
-p IPv4 -i eth2 --ip-proto tcp --ip-dport 80 -j redirect --redirect-target DROP, pcnt = 19 -- bcnt = 1140
-p IPv4 -i eth0 --ip-proto tcp --ip-sport 80 -j redirect --redirect-target DROP, pcnt = 0 -- bcnt = 0

2011/03/14 23:59:02.029 kid1| The AsyncCall clientListenerConnectionOpened constructed, this=0x9115470 [call8]
2011/03/14 23:59:02.029 kid1| comm_openex: Attempt open socket for: [::]:3128
2011/03/14 23:59:02.029 kid1| comm_openex: Opened socket FD 15 : family=10, type=1, protocol=6
2011/03/14 23:59:02.030 kid1| StartListening.cc(52) will call clientListenerConnectionOpened(FD 15, err=0, port=0x8e993b8) [call8]
2011/03/14 23:59:02.030 kid1| The AsyncCall clientListenerConnectionOpened constructed, this=0x9115558 [call10]
2011/03/14 23:59:02.030 kid1| comm_openex: Attempt open socket for: 0.0.0.0:3129
2011/03/14 23:59:02.030 kid1| comm_openex: Opened socket FD 16 : family=2, type=1, protocol=6
2011/03/14 23:59:02.030 kid1| comm: com_set_transparent() port mode for addr:'0.0.0.0:3129'.
2011/03/14 23:59:02.030 kid1| comm_open: set (IP_TRANSPARENT) on FD 16
2011/03/14 23:59:02.030 kid1| StartListening.cc(52) will call clientListenerConnectionOpened(FD 16, err=0, port=0x8e99440) [call10]
2011/03/14 23:59:02.030 kid1| HTCP Disabled.
2011/03/14 23:59:02.030 kid1| Squid plugin modules loaded: 0
2011/03/14 23:59:02.030 kid1| Adaptation support is off.
2011/03/14 23:59:02.030 kid1| Config.cc(134) FinalizeEach: Initialized 0 message adaptation services
2011/03/14 23:59:02.030 kid1| Config.cc(134) FinalizeEach: Initialized 0 message adaptation service groups
2011/03/14 23:59:02.030 kid1| Config.cc(134) FinalizeEach: Initialized 0 message adaptation access rules
2011/03/14 23:59:02.030 kid1| Ready to serve requests.
2011/03/14 23:59:02.031 kid1| entering clientListenerConnectionOpened(FD 15, err=0, port=0x8e993b8)
2011/03/14 23:59:02.031 kid1| AsyncCall.cc(32) make: make call clientListenerConnectionOpened [call8]
2011/03/14 23:59:02.031 kid1| AcceptingHTTP Socket connections at FD 15 on [::]:3128
2011/03/14 23:59:02.031 kid1| leaving clientListenerConnectionOpened(FD 15, err=0, port=0x8e993b8)
2011/03/14 23:59:02.031 kid1| entering clientListenerConnectionOpened(FD 16, err=0, port=0x8e99440)
2011/03/14 23:59:02.031 kid1| AsyncCall.cc(32) make: make call clientListenerConnectionOpened [call10]
2011/03/14 23:59:02.031 kid1| Accepting spoofingHTTP Socket connections at FD 16 on 0.0.0.0:3129
2011/03/14 23:59:02.031 kid1| leaving clientListenerConnectionOpened(FD 16, err=0, port=0x8e99440)
2011/03/14 23:59:02.689 kid1| logfile_mod_daemon_append: daemon:/usr/local/squid/var/logs/access.log: appending 2 bytes
2011/03/14 23:59:02.689 kid1| logfile_mod_daemon_append: current buffer has 7 of 32768 bytes before append
2011/03/14 23:59:02.689 kid1| logfileHandleWrite: daemon:/usr/local/squid/var/logs/access.log: write returned 9
2011/03/14 23:59:02.689 kid1| storeLateRelease: released 0 objects

[root_at_fw01 ~]#
[root_at_fw01 ~]# ip route list table all
local default dev lo table 100 scope host
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.78
192.168.1.0/24 dev br0 proto kernel scope link src 192.168.1.66
169.254.0.0/16 dev eth1 scope link metric 1003
default via 192.168.1.254 dev br0
local 192.168.1.66 dev br0 table local proto kernel scope host src 192.168.1.66
broadcast 192.168.1.0 dev eth1 table local proto kernel scope link src 192.168.1.78
broadcast 192.168.1.0 dev br0 table local proto kernel scope link src 192.168.1.66
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.1.255 dev eth1 table local proto kernel scope link src 192.168.1.78
broadcast 192.168.1.255 dev br0 table local proto kernel scope link src 192.168.1.66
local 192.168.1.78 dev eth1 table local proto kernel scope host src 192.168.1.78
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
unreachable ::/96 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 0
unreachable ::ffff:0.0.0.0/96 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 0
unreachable 2002:a00::/24 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 0
unreachable 2002:7f00::/24 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 0
unreachable 2002:a9fe::/32 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 0
unreachable 2002:ac10::/28 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 0
unreachable 2002:c0a8::/32 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 0
unreachable 2002:e000::/19 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 0
unreachable 3ffe:ffff::/32 dev lo metric 1024 error -101 mtu 16436 advmss 16376 hoplimit 0
fe80::/64 dev eth1 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 0
fe80::/64 dev eth2 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 0
fe80::/64 dev eth0 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 0
fe80::/64 dev br0 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 0
unreachable default dev lo table unspec proto kernel metric 4294967295 error -101 hoplimit 255
local ::1 via :: dev lo table local proto none metric 0 mtu 16436 advmss 16376 hoplimit 0
local fe80::207:e9ff:fee5:ac7a via :: dev lo table local proto none metric 0 mtu 16436 advmss 16376 hoplimit 0
local fe80::240:f4ff:fecd:170 via :: dev lo table local proto none metric 0 mtu 16436 advmss 16376 hoplimit 0
local fe80::240:f4ff:fecd:170 via :: dev lo table local proto none metric 0 mtu 16436 advmss 16376 hoplimit 0
local fe80::2a0:c9ff:fe08:4c26 via :: dev lo table local proto none metric 0 mtu 16436 advmss 16376 hoplimit 0
ff00::/8 dev eth1 table local metric 256 mtu 1500 advmss 1440 hoplimit 0
ff00::/8 dev eth2 table local metric 256 mtu 1500 advmss 1440 hoplimit 0
ff00::/8 dev eth0 table local metric 256 mtu 1500 advmss 1440 hoplimit 0
ff00::/8 dev br0 table local metric 256 mtu 1500 advmss 1440 hoplimit 0
unreachable default dev lo table unspec proto kernel metric 4294967295 error -101 hoplimit 255
[root_at_fw01 ~]# ip rule list
0: from all lookup local
32763: from all fwmark 0x1 iif eth2 lookup 100
32764: from all fwmark 0x1 iif eth0 lookup 100
32765: from all fwmark 0x1 iif lo lookup 100
32766: from all lookup main
32767: from all lookup default

[root_at_fw01 ~]# cat /proc/sys/net/bridge/*
0
0
0
0
0

[root_at_fw01 ~]# cat /proc/sys/net/ipv4/ip_forward
1
[root_at_fw01 ~]# cat /proc/sys/net/ipv4/conf/all/rp_filter
0

Lastly -- the script I used to do the config.

#!/bin/bash

set -x

CLIENT_IFACE=eth2
INET_IFACE=eth0

ifconfig $CLIENT_IFACE down
ifconfig $INET_IFACE down
ifconfig $CLIENT_IFACE 0.0.0.0 up
ifconfig $INET_IFACE 0.0.0.0 up

ifconfig br0 down
brctl delbr br0

brctl addbr br0
brctl addif br0 $CLIENT_IFACE
brctl addif br0 $INET_IFACE
brctl stp br0 on
# ifconfig br0 up

dhclient br0

#ip route flush table 100
#ip rule add fwmark 1 lookup 100
#ip route add local 0.0.0.0/0 dev lo table 100
#echo 0 > /proc/sys/net/ipv4/conf/lo/rp_filter
#echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
#echo 1 > /proc/sys/net/ipv4/ip_forward

ip rule del dev lo fwmark 1 lookup 100
ip rule del dev eth0 fwmark 1 lookup 100
ip rule del dev eth2 fwmark 1 lookup 100

ip rule add dev lo fwmark 1 lookup 100
ip rule add dev eth0 fwmark 1 lookup 100
ip rule add dev eth2 fwmark 1 lookup 100

ip route del local 0.0.0.0/0 dev lo table 100
ip route add local 0.0.0.0/0 dev lo table 100

ip route flush cache

echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
echo 1 > /proc/sys/net/ipv4/conf/all/forwarding
echo 1 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter

iptables -t filter -F

iptables -t raw -F
iptables -t raw -A PREROUTING -p tcp --dport 80 -j TRACE
iptables -t raw -A OUTPUT -p tcp --dport 80 -j TRACE

iptables -t raw -A PREROUTING -p tcp --sport 80 -j TRACE
iptables -t raw -A OUTPUT -p tcp --sport 80 -j TRACE

#iptables-restore < /root/squid-iptables-tproxy.save
iptables -t mangle -F
iptables -t mangle -N DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT

iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 1 --on-port 3129

modprobe ipt_LOG

#ebtables-restore < /root/squid-ebtables-tproxy.save
ebtables -t broute -F
ebtables -t broute -A BROUTING -i $CLIENT_IFACE -p IPv4 --ip-proto tcp --ip-dport 80 -j redirect --redirect-target DROP
ebtables -t broute -A BROUTING -i $INET_IFACE -p IPv4 --ip-proto tcp --ip-sport 80 -j redirect --redirect-target DROP

# turn off filtering of bridged traffic in the forward stage of iptables
#
cd /proc/sys/net/bridge/
for i in *
do
echo 0 > $i
done
unset i

James S. Binder
Vice President, Engineering
Cyphort Inc.,

jbinder_at_cyphort.com
408.761.1403 (cell)

This information contained in this e-mail message and any attachments thereto, is intended only for the personal and confidential use of the recipient(s) named above. This message may be under the terms of a Mutual Non-Disclosure Agreement communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify use immediately by e-mail and delete this original message.
Received on Tue Mar 15 2011 - 07:22:11 MDT

This archive was generated by hypermail 2.2.0 : Tue Mar 15 2011 - 12:00:01 MDT