Re: [squid-users] Updated CentOS/Squid/Tproxy Transparency steps.

Would be great if you could dump this in a wiki article to make it
easier to find (and update if needed).

Regards
Henrik

tor 2009-06-25 klockan 12:30 -0500 skrev Ritter, Nicholas:
> Some assumptions:
>
> 1) You are using a Cisco Router to redirect traffic to the squid box via
> WCCP
>
> 2) 12.4(15)T8 or higher IOS on the router
>
> 3) In my setups, the squid box is always Layer 2 adjacent to the Cisco
> router, either through a dedicated interface, or a sub-interface.
>
> 4) The ability to compile and install a Linux kernel. Please note that
> in these steps, I am NOT using a redhat kernel, nor am I using the
> RedHat method of building a kernel.
>
> 5) Some steps outlined here can be achieved through several different
> means, follow the steps exactly before emailing me or the list, as I
> have tested other methods, and they don't always work (case in point:
> GRE tunnel interface creation.)
>
> 6) This setup assumes a separate WCCP service group for each direction
> of the HTTP connection, this is not needed, but makes the setup more
> scalable. If you choose to do it a different way, then YMMV.
>
> In the kernel build specific steps, I actually include possibly to much
> information, as well as tell you to enable things that are not always
> needed for TPROXY related functionality, or never related to TPROXY
> functionality. I included them because they fit more environments, and
> thus less time wasted by people asking me questions, not that I mind but
> I don't have enough time to answer all the emails I get. I tried to
> prepare this information out without errors, if the steps don't work,
> email me with the details of where you had problems so that I can adjust
> the steps below. At the end the steps below are some common things to
> watch for in the steps that can cause the setup not to work.
>
>
> Steps
>
> 1) Install CentOS 5.3, make sure you install nothing other than the base
> packages, and trim even those down. I tend to install specific packages
> from the distro later.
>
> Note: I suggest that you make separate partition(s) for where squid will
> actually store its caches. Later mount these partitions with specific
> options (like "noatime") that will help increase performance.
>
> 2) In the initial ncurses-based setup screen, turn off services that you
> don't need, and turn off selinux compeletely.
>
> 3) After install and initial bootup and configuration, run "yum update"
> to update the system for fixes, etc. Then reboot.
>
> 4) After step 2, issue this yum command:
>
> yum install libcap libcap-devel gcc gcc-c++ bison flex yacc autoconf
> automake ncurses ncurses-devel rpm-devel libpcap tcpdump
>
> Note: let it install other dependency packages. The command above
> installs compiles, utilities, etc.
>
>
> 5) Download iptables-1.4.3.2 from netfilter.org
>
> 6) Download kernel 2.6.30 from kernel.org
>
> 7) Download squid-3.1.0.8 from squid.org
>
> 8) Decompress the kernel source, I decompress it to /usr/src/, although
> I have read all over the place that this is a bad thing to do. The
> location really does not have to be /usr/src/
>
> 9) Go into the kernel source directory, issue the following command:
> cp /boot/config-2.6.18-128.1.14.el5 ./RH-config-boxed.config
>
> 10) Issue this command: make menuconfig
>
> 11) When the ncurses-based kernel config screen loads, select the "Load
> an Alternate Configuration File" and type in the full path to the
> RH-config-boxed.config. This will load the current kernel config, and
> there may be some errors, all of which can be ignored.
>
>
> 12) Configure the kernel as you normally would, but be sure to enable
> the following:
>
> In "Networking support -> Networking options"
>
> Enable (not as modules):
> Packet socket
> Packet socket: mmapped IO
> TCP/IP networking
> IP: advanced router
> IP: policy routing
>
> Enable (as modules):
> IP: tunneling
>
> Enable (not as modules):
> IP: GRE tunnels over IP
> IP: broadcast GRE over IP
> Network packet filtering framework (Netfilter)
>
>
> In "Networking support -> Networking options -> Network packet filtering
> framework (Netfilter)"
>
> Enable (not as modules):
> Advanced netfilter configuration
>
>
> In "Networking support -> Networking options -> Network packet filtering
> framework (Netfilter) -> Core Netfilter
>
> Configuration"
>
> Enable (as modules):
> Netfilter connection tracking support
>
> Enable (not as modules):
> Connection tracking security mark support
> Connection tracking events
>
> Enable (as modules):
> Connection tracking netlink interface
> Transparent proxying support (EXPERIMENTAL)
> Netfilter Xtables support (required for ip_tables)
> "CONNMARK" target support
> "MARK" target support
> "TPROXY" target support (EXPERIMENTAL)
> "connmark" connection mark match support
> "conntrack" connection tracking match support
> "mark" match support
> "socket" match support (EXPERIMENTAL)
> "state" match support
>
>
> In "Networking support -> Networking options -> Network packet filtering
> framework (Netfilter) -> IP: Netfilter Configuration"
>
> Enable (as modules):
> IPv4 connection tracking support (required for NAT)
> IP tables support (required for filtering/masq/NAT)
> Full NAT
> MASQUERADE target support
> REDIRECT target support
> Packet mangling
>
>
> 13) After setting the above options, and any other items you want, exit
> out of the kernel config, saving your changes. It will save the kernel
> compile config to RH-config-boxed.config so issue the following command
> to put the new config in the right
>
> place:
>
> cp RH-config-boxed.config config-centos
>
> Then do the make, make_install_modules, make install, if no errors,
> adjust grub.conf to boot to the new kernel. The reboot to
>
> the new kernel.
>
> 14) Assuming the kernel compiled, installed and booted properly, it is
> time to update iptables. Decompress the iptables
>
> source, and use the following configure command:
>
> ./configure --enable-devel --enable-libipq --bindir=/bin --sbindir=/sbin
> --sysconfdir=/etc --with-kernel=<path to new kernel source dir>
> --with-kbuild=<path to new kernel source dir> --with-ksource=<path to
> new kernel source dir>
>
>
> Then do the make, make install
>
>
>
> 15) Edit ld.so.conf to add a library path:
>
> vim /etc/ld.so.conf
>
> add a line to the end of the file: /usr/local/lib
>
>
> 16) I suggest rebooting, just to make sure that iptables upgrade is
> working. Do a reboot, then do a "service iptables status" to make sure
> that iptables is running fine. If it is not running ok, it will show
> either a status failed, or an empty rule set.
>
> 17) Assuming that the iptables upgrade is working fine, the next steps
> are to add the rules and interfaces needed for WCCP and TPROXY
> functionality. First, issue a "insmod ip_gre", then do a "dmesg | tail"
> and make sure that you see "GRE over IPv4 tunneling driver" if you don't
> see anything, or you get an error from insmod, you either compiled GRE
> not as a kernel module, or not at all.
>
> 18) Create the GRE tunnel interface, issue "ifconfig gre0 <ip address of
> squid server> netmask <netmask of squid server ip> up"
>
> 19) Issue an "ifconfig" and make sure you see a gre0 interface.
>
> 20) Next we need to add iptables rules to allow traffic to the gre
> interface itself, gre protocol traffic across the Ethernet interface,
> and WCCP traffic from the router. We will do this by editing the
> iptables save file directly. Do the following:
>
> a.) service iptables save
> b.) vim /etc/sysconfig/iptables
> c.) just after the line that says ":RH-Firewall-1-INPUT - [0:0]" add the
> following:
> -A INPUT -i gre0 -j ACCEPT
> -A INPUT -p gre -j ACCEPT
> -A INPUT -i eth0 -p gre -j ACCEPT
> d.) Somewhere lower in the file where the "-A RH-Firewall-1-INPUT" rules
> are, add the following:
> -A RH-Firewall-1-INPUT -s <address of cisco router>/32 -p udp -m
> udp --dport 2048 -j ACCEPT
>
> NOTE: The "<address of cisco router>" in this instance is the ip
> address of the router that the squid box itself uses.
>
> Port 2048 is the port that WCCP traffic uses.
>
> e.) save the file and exit vim. Do a "service iptables restart" to make
> sure you don't get any errors.
>
>
> 21) Next we add the TPROXY related iptables rules, from the command line
> prompt issue the following commands:
>
> iptables -t mangle -N DIVERT
> iptables -t mangle -A DIVERT -j MARK --set-mark 1
> iptables -t mangle -A DIVERT -j ACCEPT
> iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY
> --tproxy-mark 0x1/0x1 --on-port <port squid listens to> --on-ip
>
> <squid server ip>
>
> 22) Assuming the commands in step 21 didn't give errors, do a "service
> iptables save"
>
> 23) We need to edit the iptables rule order, to make sure it is in the
> correct order: "vim /etc/sysconfig/iptables" and make sure the section
> at the bottom of the file, from the ":DIVERT - [0:0]" onward looks
> something like:
>
> -A DIVERT -j MARK --set-xmark 0x1/0xffffffff
> -A DIVERT -j ACCEPT
> -A PREROUTING -p tcp -m socket -j DIVERT
> -A PREROUTING -p tcp -m tcp --dport 80 -j TPROXY --on-port <squid port>
> --on-ip <squid server ip> --tproxy-mark 0x1/0x1
>
> When doing the cli commands in step 21, iptables puts the rules in the
> wrong order for them to work for TPROXY.
>
> Save the file and exit vim.
>
>
> 24) do a "service iptables restart"
>
> 26) Issue the following commands, per the TPROXY wiki article:
>
> ip rule add fwmark 1 lookup 100
> ip route add local 0.0.0.0/0 dev lo table 100
>
> If no errors from the two commands, add them to the end of
> /etc/rc.d/rc.local
>
>
> 27) I'm not sure this is needed anymore, but it doesn't seem to break
> anything, so add the following line to the end of /etc/rc.d/rc.local:
> echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
>
> 28) In the beginning of the rc.local file, add the following two lines:
>
> modprobe ip_gre
> <the ifconfig gre0 command line from step 18 above>
>
>
> 29) Edit /etc/sysctl.conf, make sure there is a line somewhere that
> says:
>
> # Controls IP packet forwarding
> net.ipv4.ip_forward = 1
>
> 30) Reboot the server
>
> 31) Next, build you squid 3.1.x.x, make sure that pass
> "--enable-linux-netfilter" to the configure directive, build and install
> squid.
>
> 32) You will need to edit the squid config to make sure that it has what
> it needs to use TPROXY and WCCP. For tproxy, you need one item, the
> "http_port" directive needs tproxy appended to it, I do the following:
> http_port <ip of squid server>:<port to bind to, "squid port" from step
> 23 above> tproxy disable-pmtu-discovery=always
>
> 33) Configure the WCCP specific related items in the squid config file,
> specifically:
>
> a) wccp2_router <router address used in the steps above>
> b) wccp_version 2
> c) wccp2_rebuild_wait on
> d) wccp2_fowarding_method 1
> e) wccp2_return_method 1
> f) wccp2_assigment_method 1
> g) wccp2_service dynamic 80
> h) wccp2_service dynamic 90
> i) wccp2_service_info 80 protocol=tcp flags=src_ip_hash
> priority=240 ports=80
> j) wccp2_service_info 90 protocol=tcp
> flags=dst_ip_hash,ports_source priority=240 ports=80
>
> 34) There are other squid configuration items that are need which are
> not included here because they are not specific to WCCP/TPROXY
> functionality and the specifics vary by environment.
>
> 35) In the router configuration, you need to do at least the following:
> a.) enable wccp globally with (this might actually be optional):
> ip wccp web-cache
> b.) enable the specific services:
> 1.) ip wccp 80
> 2.) ip wccp 90
> Note: I use a redirect list ACL with the two commands
> above so that the router doesn't WCCP redirect specific web sites. The
> command would look like "ip wccp 80 redirect-list 122", and access-list
> 122 would be a list of denies for sites to not be redirected, and a
> permit any any at the end of the access-list to allow all other websites
> to be wccp redirected.
> c.) On the interface that is adjacent to the squid box, do a "ip
> wccp redirect exclude in" this command makes it so that the router does
> not redirect the squid traffic as well as other client traffic.
>
>
>
> Some things to check along the way:
>
> 1) Make sure that the GRE interface on the squid box is seeing packets
> coming in. You should never see any packets going out the GRE interface,
> and you will only see packets coming in the GRE interface after the WCCP
> process on the router redirects them to the squid box.
>
> 2) Keep in mind that you should separate the idea in your mind between
> WCCP traffic and tunneled HTTP traffic. The router and the squid process
> talk to each other for status and service information with WCCP. HTTP
> traffic received from the client by the router for redirect is
> encapsulated in GRE (to preserve it) and then forwarded to the GRE
> interface.
>
> 3) The global command "show ip wccp" is a useful router command. In the
> output of this command, you should see two "Service Identifier" sections
> (one for 80, one for 90 if you use my setup steps.) Within each "Service
> Identifier" group, the number of service group clients and routers
> should each be 1. If they are not, them some facet of WCCP conversation
> between the router and the squid server is not working. Check the
> iptables port 2048 setup step 20 part d above.
>
> 4) wccp event debug commands are useful. Use "debug ip wccp events" and
> set you router logging to debug level to see "Hello" and "Here_I_am"
> packets. You need to make sure you see both to insure that the router
> and squid box are WCCP talking.
>
> 5) Surf the Web from a client and see if you get to a web site. If it
> doesn't work, check the items above, recheck the steps.
>
> It if does work, go to a site that tells you your IP to make sure TPROXY
> is working and using the client IP and not the squid IP. It is entirely
> possible that the whole setup will work, but not the client IP spoofing.
> I also suggest that you burn the setup in with web surfing to make sure
> it doesn't break.
>
> I am interested to here people's feedback so that I can improve the
> steps above, as well as share optimizations.
>
> Nick
>
>
>
>
Received on Mon Jun 29 2009 - 21:01:32 MDT