Re: tcp proxy hackery

On Sat, Mar 08, 2008, Adrian Chadd wrote:
> Nothing terribly fancy; very, very ugly and not very well laid out; but
> cut #1 of a simple TCP proxy that I'm using for toying around with network
> IO benchmarking:
>
> http://www.creative.net.au/diffs/test1-320.tar.gz
>
> Edit Makefile.defaults to point to where libevent is hiding, and then 'make'.
> The destination host/port is hardcoded in tcpproxy-1/test.c ; I point it at a
> lighttpd/thttpd server and run apachebench against the proxy.

I'll post an updated tarball in an hour or so. However, this is on a 3GHz
Pentium-D running my single-threaded test loop under FreeBSD, from a local
apachebench (that does connect() right, grumble) on an Athlon 1800 XP to a
1.4ghz Opteron running thttpd (which is pegged at 100% CPU):

Server Software: thttpd/2.25b
Server Hostname: 192.168.1.25
Server Port: 6666

Document Path: /test.c
Document Length: 93 bytes

Concurrency Level: 1000
Time taken for tests: 12.375 seconds
Complete requests: 100000
Failed requests: 0
Broken pipe errors: 0
Total transferred: 33408016 bytes
HTML transferred: 9302232 bytes
Requests per second: 8080.81 [#/sec] (mean)
Time per request: 123.75 [ms] (mean)
Time per request: 0.12 [ms] (mean, across all concurrent requests)
Transfer rate: 2699.64 [Kbytes/sec] received

And:

Server Software: thttpd/2.25b
Server Hostname: 192.168.1.25
Server Port: 6666

Document Path: /test8k
Document Length: 8192 bytes

Concurrency Level: 1000
Time taken for tests: 19.671 seconds
Complete requests: 100000
Failed requests: 0
Broken pipe errors: 0
Total transferred: 844314473 bytes
HTML transferred: 819971219 bytes
Requests per second: 5083.63 [#/sec] (mean)
Time per request: 196.71 [ms] (mean)
Time per request: 0.20 [ms] (mean, across all concurrent requests)
Transfer rate: 42921.79 [Kbytes/sec] received

I can't bump the concurrency level up at the moment as apachebench uses select().

This is after being careful and (ab)using the zero-copy socket magic that FreeBSD
uses but obviously isn't applicable 100% everywhere.

The server CPU usage is:

  PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
 8019 adrian 1 117 0 34956K 23216K CPU0 0 13:43 93.36% test
   25 root 1 -68 - 0K 8K CPU0 0 11:29 69.97% irq256: bge0
   12 root 1 171 ki31 0K 8K RUN 0 15:49 27.59% idle: cpu0
   11 root 1 171 ki31 0K 8K RUN 1 12:43 10.11% idle: cpu1

With systat saying:

39.4%Sys 31.8%Intr 5.8%User 0.0%Nice 23.0%Idle

So at the moment there isn't all that much to try and optimise in the userland; not a
lot is going on.

I'm trying to use FreeBSD's performance monitoring stuff but I'm having absolutely no
luck getting any performance measurements from userland. I may have to do userland
profiling under Linux and system profiling under FreeBSD :)

I'll see what else I can do with this. I may look at using kqueue directly rather
than using libevent, as libevent does a lot that we may not need. The next move will
be to thread this and see how to distribute the load between the two CPUs.
It won't be a 50/50 split as you can see; one of these CPUs will be doing the
network IO. I'll then try to fix apachebench to use libevent and support >1000
connections; I'd like to retry this with up to 50,000 concurrent connections.

Adrian

-- 
- Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support -
- $25/pm entry-level VPSes w/ capped bandwidth charges available in WA -