Re: storage manager ideas

On Wed, 2007-01-31 at 21:57 +0800, Adrian Chadd wrote:
> I've done a quick brain-dump about the current storage manager state.
>
> http://wiki.squid-cache.org/DevelopmentIdeas/StorageStuff
>
> I'd appreciate some comments before I run off and start coding this stuff
> up..

I agree that read-parse-dump-write-read-parse-dump-write sequence is
inefficient for message headers and other metadata. IIRC, we talked
about optimizing this at least since 1998. Stored binary metadata has
its drawbacks, but overall it is probably a win for a
performance-sensitive proxy.

For many other problems and changes you are talking about, I would try
reworking MemBuf or providing a similar object that would allow
higher-level code to "copy" and "concatenate" chunks of memory without
the actual copy taking place. Visualize a MemBuf with an offset and
subsize fields. Now add support for a chain of such buffers that looks
like a single buffer for higher-level code.

With such low-level support and perhaps vector I/O, most performance
problems with the high level code doing to much copying would either
disappear or would be much easier to fix (by using appropriate buffers
instead of char* manipulations).

As an added bonus, it may be possible to avoid a lot of the copying when
parsing headers because string-based headers would be able to refer to
portions of the original I/O buffer.

Proper reference counting of true/allocated buffers would be required to
keep overall memory consumption comparable to current Squid, of course.

Finally, I am not sure I agree regarding storage decision making time.
An optimized storage system (the interesting case) would probably
buffer/merge small chunks and would probably not store object chunks
sequentially on disk, so the issue of the total object size becomes
unimportant.

Needless to say, I believe this work should be done in Squid3.1 code
base or later :-/.

HTH,

Alex.
Received on Wed Jan 31 2007 - 09:52:41 MST