Re: Squid performance on the bake-off

On Apr 11, 11:39pm, Glenn Chisholm (possibly) wrote:
> > Ok, there's the Unix FS handicap to account for, but we are talking
> > about _more_than_six_times_ the performance, and in any case the tradition
> > says that a well-tuned FFS (as is the case with FreeBSD) approaches 50%
> > of the hardware maximum transfer rate.
>
> The problem is not so much the transfer rate, it is the fact that
> we are dealing with massive numbers of small files. They are the cause of
> the bad performance with a normal file system, just as the would be with
> any multipurpose file system.
>
> > Based on my experience with Squid, neither that CPU nor the disks can
> > explain a 545% diference, but just maybe (depending on the total size
> > and distribution of the cacheable data), perhaps 100% more RAM can
> > explain at least a part of it. And the cost for twice (or even four
> > times more) the RAM would be negligible, specially if compared to other
> > alternatives (more/faster disks, etc). I'm ignorant enough about the
> > Polygraph benchmark to be unable to elaborate further.
>
> The amount of RAM can explain none of it. Doubling the RAM in the
> machine was just not an option, it would have required a entirly new
> machine (512MB is all that fits on that motherboard).
>
> I have rerun the same tests that were run at the Bake-off back at
> our lab on alternate hardware with twice the RAM and a great deal more
> disk and CPU power and gained nothing, as long as we are using UFS. The
> machine at the Bake-Off was not thrashing at all. Not that I am claming
> that Squid did its best at the Bake-Off, I believe that Duane had about 10
> minutes to set up the machine and decide what request rates to run. We
> were all rather busy and it was not a priority.
>
> A few simple tests have shown that with alternative file systems
> we have done a great deal better. Profiling the machine while runing with
> UFS shows the thrashing that occurs in the file system. As we have said
> before a custom file system will allow us to remove the major bottleneck
> from Squid. Which will then start showing where Squid is inefficient and
> other UNIX issues that are slowing us down. At the moment Duane is working
> on SquidFS and when he has the time to get that finished we should start
> being able to make squid much faster.

Anybody have any experience with running Squid using an MFS file
system (on FreeBSD or a similar system)? (I might mention that more
RAM would be even more helpful in that case, to keep from having to
swap in and out quite as much)

        -Allen

-- 
Allen Smith				easmith@beatrice.rutgers.edu