Re: squid-2.5 and coss

On Wed, Aug 13, 2003, Adrian Chadd wrote:
> On Thu, Aug 14, 2003, Henrik Nordstrom wrote:
>
> > It would be very undesireable from a release management perspective if
> > COSS in Squid-2.5 is newer than COSS in Squid-3.0, and neither me or
> > Robert will be very happy if this happens.
>
> Let me try to make it work first. heh.

Actually, I've just hit a snag I should have remembered from a while ago.
The objects don't actually have any size information stored in the
metadata. Which makes sense at the moment, the store layer opens and
outputs to files _before_ their size is absolutely known.

There's no way to do a dirty rebuild unless:

* we store the object size in the metadata, if its known
  (which it will be for COSS. heh.)

* we try to add the list of object sizes at the front of
  the stripe.

Now, the former might be neater but the latter could work out.
The only problem is if there's not enough space to store the sizes
of each of the objects.

For example:

a 1 megabyte COSS stripe will hold, on average, 85 objects (at 12k an object).
340 bytes will be needed to store 4-byte sizes for those 85 objects.
I personally am doing all my tests with a 64k blocksize. Works a treat.
We can always throw a warning if we can't pack all the object sizes into
a block. However, I may need to store a byte of flags (especially if
the object has been deleted) _AND_ I'm not entirely sure the current
squid index system will be COSS-friendly (read: I may have to populate
a stack of object sizes when the stripe is being filled and then
handle the situation where an object is purged in the active stripe.)

(Yes, another possibility is to actually store the metadata at the end of
the stripe - if we start tracking how many objects are being stored in
the stripe I can calculate how much space we need each time through
storeCossAllocate() to store the objects AND all their state/offsets...)

The second object is a bit hackish _BUT_ its just another verification
technique. I also think it'd make the dirty rebuilds quite fast - especially if
implementing using asyncio.

Another issue - stale objects. We can't jump in and delete an object from a COSS
stripe once its written to disk - the only time this happens is when it comes time
to delete said stripe and all the objects (which translates to deleting from the
tail of the LRU). If, at a later date, someone PURGEs an object it won't be deleted
from the COSS store until its stripe is up for re-allocation. Obviously this will
only occur during a dirty rebuild but its a real problem.

(Hm, thinking about it, how about keeping a stack of deleted storeentry's and
 write it out as part of writing a stripe to disk? Then I'd know which
 entry to delete..)

Comments? I'd like to start hacking on this tonight.

adrian
Received on Thu Aug 14 2003 - 00:31:59 MDT