[squid-users] Squid 2.6 + WCCPv2 + Linux 2.6 + cisco

I am having odd problems getting squid+wccpv2+Linux working with our
cisco router.
I apologize in advance for the relative deluge of information, but I
want to be sure to cover all the bases.

Here is the list of ingredients:

Hardware: IBM eServer something or other.
OS: Fedora Core 6 sub-version?
Kernel: 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:54:53 EST 2006 i686 i686
i386 GNU/Linux (from uname -a)
Squid: 2.6.STABLE5-20061116
Router: Cisco 3745, IOS 12.3.6a

General description:

We have two test workstations whose default routes point to the
interception router:
10.160.100.10 workstation A
10.160.100.38 workstation B
10.160.100.2 interception router, workstation side
10.160.123.17 interception router, loopback IP
192.168.251.19 interception router, interface closest to squid
172.16.2.93 squid server

The interception router has an interface in the workstation subnet as
well as another interface pointing to the internal network in which the
squid proxy is found. The squid proxy is not directly connected to the
interception router. There are two intervening routers between the
squid and the interception router.
The squid proxy has 4 ethernet interfaces, eth0 through eth3. Only eth2
is connected.
The squid proxy accesses the Internet through a Checkpoint firewall
which takes care of NATting the private IP to a static public IP.

Configruations:

#####
## Cisco:
ip wccp web-cache redirect-list WCCP-USERS group-list WCCP-PROXY-LIST
interface Loopback0
 ip address 10.160.123.17 255.255.255.255
interface GigabitEthernet1/0.100
 ip wccp web-cache redirect in
ip access-list standard WCCP-PROXY-LIST
 permit 172.16.2.93 <== IP address of Squid WCCPv2 proxy
ip access-list standard WCCP-USERS
 permit 10.160.100.10 <== test workstation
 permit 10.160.100.38 <== test workstation

#####
## Linux Setup and configuration:

## Ethernet is plugged in to eth2 only. eth0,1,3 are not connected.
## eth2 is 172.16.2.93 mask:255.255.255.248 bcast: 172.16.2.95
## default gateway is 172.16.2.89

## GRE setup
## For [remote], we are using loopback IP of router.
modprobe ip_gre
iptunnel add gre1 mode gre remote 10.160.123.17 local 172.16.2.93 dev eth2
ifconfig gre1 127.0.0.2 up

## IPTables setup
iptables -A INPUT -i gre1 -j ACCEPT
iptables -t nat -A PREROUTING -i gre1 \
         -d ! 10.160.0.0/17 -p tcp --dport 80 \
         -j DNAT --to-destination 172.16.2.93:8080

## kernel parameters.
## Is this the same as going to write stuff in /proc/sys/net/...?
## It seems to be.
sysctl net.ipv4.ip_forward=1
sysctl net.ipv4.conf.eth2.forwarding=1
sysctl net.ipv4.conf.gre1.forwarding=1
sysctl net.ipv4.conf.eth2.rp_filter=0
sysctl net.ipv4.conf.gre1.rp_filter=0
sysctl net.ipv4.conf.all.rp_filter=0

#####
## Squid build parameters:
./configure \
        --disable-wccp \
        --enable-err-languages="English French" \
        --enable-linux-netfilter \
        --disable-ident-lookups \
        --enable-auth="basic digest ntlm" \
        --enable-basic-auth-helpers="getpwnam LDAP MSNT
multi-domain-NTLM NCSA PAM SASL SMB YP" \
        --enable-ntlm-auth-helpers="fakeauth no_check
SMB" \
        --enable-digest-auth-helpers="ldap
password" \
        --enable-external-acl-helpers="ip_user ldap_group session
unix_group wbinfo_group" \
        --with-large-files
make
make install

#####
## Deviations from stock squid.conf.default:

http_port 172.16.2.93:8080 transparent
http_port localhost:8080
tcp_outgoing_address 172.16.2.93

## Disable ICP since we only have the one cache
icp_port 0

#####
## Set all the pathname stuff here.
## Easier to change.
## -- Jason Taylor, 2006-11-16
cache_dir ufs /var/squid/cache-test1 2048 16 256
coredump_dir /var/squid/cache-prod1
pid_filename /var/log/squid/squid-test1.pid
cache_log /var/log/squid/cache-test1.log
cache_store_log /var/log/squid/store-test1.log
cache_access_log /var/log/squid/access-test1.log
access_log syslog
#####

#####
## Set the ftp user.
ftp_user proxy@dev.nl

#####
## Start local acl entries
acl MyNets src 10.0.0.0/8 172.16.0.0/12
http_access allow MyNets

#####
## Use the cache effective username of "squid"
cache_effective_user squid

#####
## Set the proxy hostname
visible_hostname ProxyWCCP

#####
## This setting might possibly be related to WCCP. We shall see.
httpd_accel_no_pmtu_disc on

#####
## Turn off snmp. It could prove useful down the road, but now now.
## snmp_port 0

#####
## Let's turn on the WCCP.
wccp2_address 172.16.2.93
wccp2_router 192.168.251.19
wccp2_rebuild_wait on
wccp2_forwarding_method 1
wccp2_return_method 1
wccp2_assignment_method 1
wccp2_service standard 0
wccp2_weight 10000

===============end of configs=====================

So with that out of the way, here is what I am seeing...
When I start squid, it connects with the router:
    [root@localhost etc]# netstat -an | egrep -i "listen|estab"
    Active Internet connections (servers and established)
    tcp 0 0 127.0.0.1:8080
0.0.0.0:* LISTEN
    tcp 0 0 172.16.2.93:8080
0.0.0.0:* LISTEN
    tcp 0 0 :::22
:::* LISTEN
    tcp 0 0 ::ffff:172.16.2.93:22
::ffff:10.160.100.18:51955 ESTABLISHED
    tcp 0 48 ::ffff:172.16.2.93:22
::ffff:10.160.100.38:3338 ESTABLISHED
    udp 0 0 172.16.2.93:2048
192.168.251.19:2048 ESTABLISHED

and a tcpdump on eth2 shows the udp:2048 traffic.
I had originally tried defining the wccp2_router as the router's
loopback address, but it would respond with the IP of the closest
interface to the squid, so I changed it to that.
    [root@localhost etc]# tcpdump -ni eth2 port 2048
    tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode
    listening on eth2, link-type EN10MB (Ethernet), capture size 96 bytes
    11:56:14.029205 IP 172.16.2.93.2048 > 192.168.251.19.2048: UDP,
length 144
    11:56:14.029684 IP 192.168.251.19.2048 > 172.16.2.93.2048: UDP,
length 140
    11:56:24.029542 IP 172.16.2.93.2048 > 192.168.251.19.2048: UDP,
length 144
    11:56:24.030043 IP 192.168.251.19.2048 > 172.16.2.93.2048: UDP,
length 140
    11:56:34.029909 IP 172.16.2.93.2048 > 192.168.251.19.2048: UDP,
length 144
    11:56:34.030413 IP 192.168.251.19.2048 > 172.16.2.93.2048: UDP,
length 140

I see traffic over the GRE interface. Basically the redirected traffic
from a test workstation. This is the workstation trying to load
"squid-cache.org"
[root@localhost ~]# tcpdump -ni gre1
tcpdump: WARNING: arptype 778 not supported by libpcap - falling back to
cooked socket
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gre1, link-type LINUX_SLL (Linux cooked), capture size 96 bytes
12:17:04.712823 IP 10.160.100.38.3582 > 12.160.37.9.http: S
3102010967:3102010967(0) win 16384 <mss 1260,nop,nop,sackOK>
12:17:04.714861 IP 10.160.100.38.3582 > 12.160.37.9.http: . ack
2210366199 win 17640
12:17:04.716128 IP 10.160.100.38.3582 > 12.160.37.9.http: P 0:378(378)
ack 1 win 17640
12:17:04.741874 IP 10.160.100.38.3582 > 12.160.37.9.http: . ack 2 win 17640
12:17:04.742200 IP 10.160.100.38.3582 > 12.160.37.9.http: F 378:378(0)
ack 2 win 17640

And I see the squid respond to the user over eth2. Below is a tcpdump
of the exact same session.
[root@localhost squid]# tcpdump -ni eth2 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 96 bytes
12:17:04.712820 IP 12.160.37.9.http > 10.160.100.38.3582: S
2210366198:2210366198(0) ack 3102010968 win 5840 <mss 1460,nop,nop,sackOK>
12:17:04.716153 IP 12.160.37.9.http > 10.160.100.38.3582: . ack 379 win 6432
12:17:04.739317 IP 12.160.37.9.http > 10.160.100.38.3582: F 1:1(0) ack
379 win 6432
12:17:04.742224 IP 12.160.37.9.http > 10.160.100.38.3582: . ack 380 win 6432

So if I understand the process correctly, the workstation requests come
over the GRE tunnel, and the squid server spoofs the server address and
send the reply directly to the workstation, leaving the workstation to
believe that it has established a connection with the origin server.

So far so good. But now to the crux of the problem. It doesn't work.
Looking in the logs, I see nothing in access.log.
However, cache.log is a whole other story. Here is what I see, with
annotations where appropriate.

=========<start cache.log extract>==========
==> Here is the startup of squid, using "squid -s -D"

2006/11/17 12:36:57| Starting Squid Cache version 2.6.STABLE5-20061116
for i686-pc-linux-gnu...
2006/11/17 12:36:57| Process ID 8832
2006/11/17 12:36:57| With 1024 file descriptors available
2006/11/17 12:36:57| Using poll for the IO loop
2006/11/17 12:36:57| DNS Socket created at 0.0.0.0, port 32817, FD 5
2006/11/17 12:36:57| Adding domain ads.vmd.ca from /etc/resolv.conf
2006/11/17 12:36:57| Adding nameserver 172.20.2.197 from /etc/resolv.conf
2006/11/17 12:36:57| Adding nameserver 172.24.2.197 from /etc/resolv.conf
2006/11/17 12:36:57| Adding nameserver 172.20.2.196 from /etc/resolv.conf
2006/11/17 12:36:57| Unlinkd pipe opened on FD 10
2006/11/17 12:36:57| Swap maxSize 2097152 KB, estimated 161319 objects
2006/11/17 12:36:57| Target number of buckets: 8065
2006/11/17 12:36:57| Using 8192 Store buckets
2006/11/17 12:36:57| Max Mem size: 8192 KB
2006/11/17 12:36:57| Max Swap size: 2097152 KB
2006/11/17 12:36:57| Rebuilding storage in /var/squid/cache-prod1 (CLEAN)
2006/11/17 12:36:57| Using Least Load store dir selection
2006/11/17 12:36:57| Set Current Directory to /var/squid/cache-prod1
2006/11/17 12:36:57| Loaded Icons.
2006/11/17 12:36:57| Accepting transparently proxied HTTP connections at
172.16.2.93, port 8080, FD 12.
2006/11/17 12:36:57| Accepting proxy HTTP connections at 127.0.0.1, port
8080, FD 13.
2006/11/17 12:36:57| Accepting WCCPv2 messages on port 2048, FD 14.
2006/11/17 12:36:57| Initialising all WCCPv2 lists
2006/11/17 12:36:57| Ready to serve requests.
2006/11/17 12:36:58| Done reading /var/squid/cache-prod1 swaplog (322
entries)
2006/11/17 12:36:58| Finished rebuilding storage from disk.
2006/11/17 12:36:58| 322 Entries scanned
2006/11/17 12:36:58| 0 Invalid entries.
2006/11/17 12:36:58| 0 With invalid flags.
2006/11/17 12:36:58| 322 Objects loaded.
2006/11/17 12:36:58| 0 Objects expired.
2006/11/17 12:36:58| 0 Objects cancelled.
2006/11/17 12:36:58| 0 Duplicate URLs purged.
2006/11/17 12:36:58| 0 Swapfile clashes avoided.
2006/11/17 12:36:58| Took 0.7 seconds ( 438.7 objects/sec).
2006/11/17 12:36:58| Beginning Validation Procedure
2006/11/17 12:36:59| Completed Validation Procedure
2006/11/17 12:36:59| Validated 322 Entries
2006/11/17 12:36:59| store_swap_size = 2120k
2006/11/17 12:36:59| storeLateRelease: released 0 objects

==>I try to load http://squid-cache.org through the wccp proxy.

FATAL: Received Segment Violation...dying.
2006/11/17 12:37:54| storeDirWriteCleanLogs: Starting...
2006/11/17 12:37:54| WARNING: Closing open FD 12
2006/11/17 12:37:54| Finished. Wrote 322 entries.
2006/11/17 12:37:54| Took 0.0 seconds (858666.7 entries/sec).
CPU Usage: 0.023 seconds = 0.010 user + 0.013 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 0
Memory usage for squid via mallinfo():
        total space in arena: 2132 KB
        Ordinary blocks: 2064 KB 5 blks
        Small blocks: 0 KB 0 blks
        Holding blocks: 236 KB 1 blks
        Free Small blocks: 0 KB
        Free Ordinary blocks: 67 KB
        Total in use: 2300 KB 97%
        Total free: 67 KB 3%

==> A small pause in logging (about 10 seconds)

2006/11/17 12:37:57| Starting Squid Cache version 2.6.STABLE5-20061116
for i686-pc-linux-gnu...
2006/11/17 12:37:57| Process ID 8838
2006/11/17 12:37:57| With 1024 file descriptors available
2006/11/17 12:37:57| Using poll for the IO loop
2006/11/17 12:37:57| DNS Socket created at 0.0.0.0, port 32817, FD 5
2006/11/17 12:37:57| Adding domain ads.vmd.ca from /etc/resolv.conf
2006/11/17 12:37:57| Adding nameserver 172.20.2.197 from /etc/resolv.conf
2006/11/17 12:37:57| Adding nameserver 172.24.2.197 from /etc/resolv.conf
2006/11/17 12:37:57| Adding nameserver 172.20.2.196 from /etc/resolv.conf
2006/11/17 12:37:57| Unlinkd pipe opened on FD 10
2006/11/17 12:37:57| Swap maxSize 2097152 KB, estimated 161319 objects
2006/11/17 12:37:57| Target number of buckets: 8065
2006/11/17 12:37:57| Using 8192 Store buckets
2006/11/17 12:37:57| Max Mem size: 8192 KB
2006/11/17 12:37:57| Max Swap size: 2097152 KB
2006/11/17 12:37:57| Rebuilding storage in /var/squid/cache-prod1 (CLEAN)
2006/11/17 12:37:57| Using Least Load store dir selection
2006/11/17 12:37:57| Set Current Directory to /var/squid/cache-prod1
2006/11/17 12:37:57| Loaded Icons.
2006/11/17 12:37:57| Accepting transparently proxied HTTP connections at
172.16.2.93, port 8080, FD 12.
2006/11/17 12:37:57| Accepting proxy HTTP connections at 127.0.0.1, port
8080, FD 13.
2006/11/17 12:37:57| Accepting WCCPv2 messages on port 2048, FD 14.
2006/11/17 12:37:57| Initialising all WCCPv2 lists
2006/11/17 12:37:57| Ready to serve requests.
2006/11/17 12:37:58| Done reading /var/squid/cache-prod1 swaplog (322
entries)
2006/11/17 12:37:58| Finished rebuilding storage from disk.
2006/11/17 12:37:58| 322 Entries scanned
2006/11/17 12:37:58| 0 Invalid entries.
2006/11/17 12:37:58| 0 With invalid flags.
2006/11/17 12:37:58| 322 Objects loaded.
2006/11/17 12:37:58| 0 Objects expired.
2006/11/17 12:37:58| 0 Objects cancelled.
2006/11/17 12:37:58| 0 Duplicate URLs purged.
2006/11/17 12:37:58| 0 Swapfile clashes avoided.
2006/11/17 12:37:58| Took 0.7 seconds ( 455.4 objects/sec).
2006/11/17 12:37:58| Beginning Validation Procedure
2006/11/17 12:37:58| Completed Validation Procedure
2006/11/17 12:37:58| Validated 322 Entries
2006/11/17 12:37:58| store_swap_size = 2120k
2006/11/17 12:37:58| storeLateRelease: released 0 objects

=========< end of cache.log extract > =======

Any ideas a to why I get the "FATAL: Received Segment
Violation...dying." message?

Any help is appreciated.

Cheers,

/Jason
Received on Fri Nov 17 2006 - 10:40:39 MST