Re: moving back to 32-bit swap_filens ?

On Mon, Aug 20, 2001, Chemolli Francesco (USI) wrote:

> > Are you sure its alignment? I'm dead sure its malloc overhead.
> > 16 bytes is a big much for struct alighment. Also, when you
> > set it to 32 bits, did you drop the sdirno type down to a char
> > when you did the size comparison?
>
> It's not unheard of. Some CPU architectures (mainly alpha IIRC) really
> like to have handy alignments.

Oh, I know for the alpha that things might become a little twisted.
But one byte throwing the struct size off by 16 bytes on the i386
platform sounds different. I'll go and check with sizeof() sometime
today.

> Another thing: I think that COSS might be a wonderful test of
> Linux's O_DIRECT open flag. Only caveat, you have to read in PAGE_SIZE
> blocks, at PAGE_SIZE boundaries. But it could increase performance, at
> least this is what Andrea Arcangeli promises, since it will use zero-copy
> DMA from disk to userspace (IIRC) and bypass all kernel caches.

Yup. The BSDs let you get at the raw char devices too which can
implement this kind of thing. FreeBSD can do this kind of tricks
with files and page-aligned RAM.

> > COSS works quite fine with a 2gb spool.
>
> We could leave this as a limit, and use multiple spools...
> Also, I think it would be a good idea to add documentation saying
> that it's best to dd if=/dev/zero possibly after mkfs to decrease
> defragmentation...

Oh, for the files. Good point. When it becomes useful enough to
use I'll put that in there.

> > * finish up async_io.c (add open/close wrappers which handle flushing
> > pending IO, just in case)
> > * Fix up the read code to read back the entire object in one
> > system read,
> > rather than 4k chunks
>
> Uhm.. I'm not very convinced of this. Can improve the memory pressure
> considerably. Maybe you could make it 100k chunks or something, but
> I'd prefer to keep the chunking.

OH, perhaps I didn't explain it properly. Instead of the higher
layer giving storeRead() a buffer to copy into, the storeRead()
callback returns a pointer to a cbdata buffer containing the read
results. The buffer should be variable size, but yeah there'll be
an upper limit of say 128k rather than 4k. For large objects
in a ufs-style FS, the data may come back in chunks since the object
is big, but if the object is a small image it'll come back
in one chunk.

There's some magic with the SM_PAGE_SIZE size in the code, and there
was even before I got to it. the CLIENT_MEM_SOCK_BUF is also 4096
bytes and when I tried upping it a while back things fell over.

> > - This involves changing storeRead() to return a
> > cbdataFree'able buf,
> > rather than taking a buffer to copy into
> > * Up the COSS FS size to be >2gb
> > * Look at adding the swaplog stuff to the beginning of each stripe
> >
> Can't we just use two files? That way we can fsync() only metadata or
> something.

A seperate metadata file could work.

> > So, I'll leave the last step until the rest of them are done, and then
> > work on it. That might require the most squid internal reworking.
> >
> > I'm getting there. :-)
> >
> > So, here's a question: should we put the slab allocator into
> > squid-HEAD
> > now?
>
> Uhm... I'm supposed to deploy 2.5 today. Can't you wait a sec before
> breaking
> things?
> Pretty please?

Tsk. I don't like how people are using squid-HEAD as production code.
This is really starting to irk me somewhat. I'd suggest either creating
a 2.5 branch for you guys to keep doing NTLM stuff on, or force you
two to MFC the NTLM and auth code back to squid-2.4.

I note that more and more people on squid-users are using squid-2.5
in production, and its rather scary for the developers. Well, at
least me. :-)

Adrian
Received on Mon Aug 20 2001 - 08:25:59 MDT