Re: [squid-users] Filedescriptors

Dave Raven wrote:
>
> I actually do have no doubt it will go faster than 350reqs/per
> sec.

Keep hoping, Dave. It aint gonna happen on IDE hardware. (And even the
majority of SCSI hardware isn't up to the task.)

> I think you underestimate it perhaps, although I'm cautious with
> my words as I'm sure you know it much better than I do; but you
> dont feel thats its mearly limited because of hardware?
> As I've faith in FreeBSD (smile).

FreeBSD is a fine OS. But it aint magic. It is probably the second
most performant OS for Squid, after Linux, given equal hardware.
Solaris also might be a contender, as it can use Squid's aufs threaded
disk I/O.

> How fast is it generally assumed squid will go with a massive
> amount of hardware - if it was to be used as an enterprise
> solution as such.

We've shipped boxes (huge, honking, monster boxes--as big as
off-the-shelf hardware goes) that could sustain around 300 reqs/sec on a
Polymix-4 workload. 350 on a DataComm-1. I think it was Kinkie who had
some boxes in the real world pushing up around that during peak periods.
  Monster boxes in his case as well. This info tends to reinforce my
belief that Polymix-4 is actually harder than the real world. Polymix-3
seemed to be a closer estimation of how Squid will behave in
reality--but it lacks DNS and some other cool stuff.

I've mentioned how much you can expect to get from your single IDE box
with an 800MHz processor and 512MB of RAM. I wasn't joking. 100/sec is
gonna be your limit (if that fast).

> Some stuff on my tests:
>
> [CUT]
> [... all +/- 150 per sec ...]
> 001.62| i-wait 14081 151.38 1511 50.92 0 241
> 001.70| i-wait 14841 151.93 1521 52.50 0 281
> 001.79| i-wait 15648 161.15 1649 49.57 0 254
> 001.87| i-wait 16315 133.13 2103 50.07 0 324
> 001.95| i-wait 16736 84.02 3149 49.64 0 636
> 002.04| i-wait 17197 92.18 4900 48.37 0 937
> 002.12| i-wait 17615 83.59 6456 55.50 0 1284
> 002.20| i-wait 17988 74.60 8375 49.06 0 1665
> [CUT]
>
> Thats a small sample when it was set to do 150 per sec on the
> datacomm-1.pg testfile distributed with polygraph.
> After that it starts to slow down, as the disk gets more load.

Heheh... Yep, I would think so. ;-)

You do notice that you're less than two minutes into the test when Squid
falls apart right? I would expect you'd want to use the cache for more
than two minutes at a time.

> I should point out however, that I'm running with straight ufs,
> and
> the drives are NOT mounted async. Its clearly the drives that are
> limiting me. I get the exact same results with 200 hits, and 250.
> (time wise - when it slows down)...
>
> At 75 it goes fine. up to 125 its fine really. as soon as I go
> higher
> it dies out after about a while.

Given the above result on the 150 run, I doubt you'll get a sustaind 75,
even.

You're not giving these runs a chance to really whip up on your box,
Dave. Give it four hours at 75. Given the above 150 results, I'd bet a
full DataComm-1 at 75 will take this box down a notch or two. Also,
don't forget to pre-fill the cache before running
benchmarks...Benchmarks are meaningless on an empty cache. Squid will
scream when she's on empty.

> Now this is something interesting. its using simple.pg; and
> automatically going to 240 odd requests. but then it dies
> with a ton of those messages above. And it happens at
> the same stage the whole time.

simple.pg is not a benchmark. It is a connectivity test. If you get
connectivity with simple.pg, you move on to something more realistic
immediately. Polymix 4 is the preferred workload, but I still use
DataComm-1 for quick ball-park tests just because it only takes four
hours after-fill to give some reasonably accurate results.

> I'm not sure if this is something that should be discussed on the
> list.
> But I'd be keen to talk privately with you (Joe), or anyone else
> interested in helping me better the performance (on a low end
> unit).

For FreeBSD:

Mount noatime, give it 4096 file descriptors or so, use Soft Updates,
use DiskD. Read about Duane's configurations at the cacheoffs, as he
ran them on FreeBSD and knows better than me how to tune a BSD+Squid.
The Polygraph BSD tuning tips are probably directly applicable to Squid
as well. They're on the cacheoff pages somewhere, in the preparation
tips for the cacheoffs.

> Otherwise, thanks for the help so far everyone.
> If I make any startling discoveries I'll be sure to say.

I'd be startled if you could top 100 with that hardware and FreeBSD on a
four hour DataComm-1. ;-)

I hate to be such a downer, Dave. Your enthusiasm for Squid performance
is nice. But check the existing data. Duane and I each have been at
three of the last four cacheoffs with pretty well-tuned Squid
machines--you're not going to do much better than we did, given equal
hardware. And we maxed at 140 and 130, respectively, I think. Neither
hardware nor Squid has improved that much in the months since Boulder.

Of course, I now have to ask the question: Just how much bandwidth does
your enterprise have? 75 requests a second is more than ample for
3Mbps. Takes a lot of casual office browsers to fill two T1 links, I think.

Hope this helps.

-- 
Joe Cooper <joe@swelltech.com>
Web caching appliances and support.
http://www.swelltech.com