Re: doublecheck

On 14 Nov 2000, at 0:35, Duane Wessels <wessels@squid-cache.org> wrote:

> On Tue, 14 Nov 2000, Andres Kroonmaa wrote:
>
> > stat() would be much faster call if every L2 dir was touched only
> > once, that is we wish to sort the files we stat. I think this also
>
> I'm really skeptical that it will make much difference. UFS is slow and
> stat() is slow, unlink is worse.

 there is a vast difference. swap.state is pretty random, if we stat all
 files as we go, we get random disk access. If we read in full swap.state,
 set all filemap bits, and then sequentially stat all files, we get 1
 disk access per L2 directory. As all the process for few Mil of files
 takes quite long time, we have enourmously higher probability of having
 no additional disk access for consequtive files with sequential pattern.
 We also don't overload disk subsystem and directory caches.

 below is a sample run on one of my real caches, L1=10, L2=384, 1M files.
 in one case random access to 66 out of 1M files, in second case sequential
 access to 66 files starting from random point. This isn't of course very
 scientific, but you'd see consistently less disk traffic and smaller
 times for sequential stuff. (2nd column is time spent in syscall, in secs)

./a.out -g -l 384 -L 10 -n 66 -m 1234234 -p /www/cidera2 | ptime truss -lD -t stat ./a.out

/1: 0.0115 xstat(2, "/www/cidera2/05/22/000B73A5", 0x08061E60) Err#2 ENOENT
/1: 0.0098 xstat(2, "/www/cidera2/00/A6/001779BF", 0x08061E60) = 0
/1: 0.0141 xstat(2, "/www/cidera2/02/175/001D30AC", 0x08061E60) = 0
/1: 0.0092 xstat(2, "/www/cidera2/07/A7/0010BB74", 0x08061E60) Err#2 ENOENT
/1: 0.0003 xstat(2, "/www/cidera2/04/F7/000A738C", 0x08061E60) Err#2 ENOENT
/1: 0.0101 xstat(2, "/www/cidera2/01/8D/001993B1", 0x08061E60) = 0
/1: 0.0222 xstat(2, "/www/cidera2/00/146/0001E9C3", 0x08061E60) = 0
/1: 0.0088 xstat(2, "/www/cidera2/06/20/000DB08F", 0x08061E60) Err#2 ENOENT
/1: 0.0189 xstat(2, "/www/cidera2/00/52/00007C21", 0x08061E60) = 0
/1: 0.0108 xstat(2, "/www/cidera2/03/40/0007204E", 0x08061E60) = 0
/1: 0.0103 xstat(2, "/www/cidera2/05/2C/000B8337", 0x08061E60) Err#2 ENOENT
/1: 0.0002 xstat(2, "/www/cidera2/05/8E/000C158E", 0x08061E60) Err#2 ENOENT
/1: 0.0051 xstat(2, "/www/cidera2/02/AE/001C0621", 0x08061E60) = 0
/1: 0.0067 xstat(2, "/www/cidera2/09/5B/0014C88A", 0x08061E60) Err#2 ENOENT
/1: 0.0140 xstat(2, "/www/cidera2/03/123/001EF487", 0x08061E60) = 0
/1: 0.0087 xstat(2, "/www/cidera2/05/114/000CDE02", 0x08061E60) Err#2 ENOENT
/1: 0.0102 xstat(2, "/www/cidera2/04/48/00096D69", 0x08061E60) Err#2 ENOENT
/1: 0.0084 xstat(2, "/www/cidera2/04/172/0021ABC1", 0x08061E60) Err#2 ENOENT
/1: 0.0004 xstat(2, "/www/cidera2/00/147/00186B57", 0x08061E60) Err#2 ENOENT
/1: 0.0077 xstat(2, "/www/cidera2/04/A1/002072FC", 0x08061E60) Err#2 ENOENT
/1: 0.0049 xstat(2, "/www/cidera2/06/5B/000E09BC", 0x08061E60) Err#2 ENOENT
/1: 0.0096 xstat(2, "/www/cidera2/07/162/0011D32B", 0x08061E60) Err#2 ENOENT
/1: 0.0101 xstat(2, "/www/cidera2/04/3D/00095B8D", 0x08061E60) Err#2 ENOENT
/1: 0.0095 xstat(2, "/www/cidera2/02/E1/001C527E", 0x08061E60) = 0
/1: 0.0044 xstat(2, "/www/cidera2/08/FB/00137988", 0x08061E60) Err#2 ENOENT
/1: 0.0039 xstat(2, "/www/cidera2/01/DE/00038E41", 0x08061E60) = 0
/1: 0.0167 xstat(2, "/www/cidera2/02/62/00051371", 0x08061E60) = 0
/1: 0.0108 xstat(2, "/www/cidera2/01/0D/000254BC", 0x08061E60) = 0
/1: 0.0081 xstat(2, "/www/cidera2/00/D2/0017BB39", 0x08061E60) = 0
/1: 0.0055 xstat(2, "/www/cidera2/03/8D/00079496", 0x08061E60) = 0
/1: 0.0152 xstat(2, "/www/cidera2/00/164/0002172F", 0x08061E60) = 0
/1: 0.0182 xstat(2, "/www/cidera2/02/B3/001C0DEA", 0x08061E60) = 0
/1: 0.0169 xstat(2, "/www/cidera2/02/A3/000574EC", 0x08061E60) = 0
/1: 0.0108 xstat(2, "/www/cidera2/06/25/000DB81A", 0x08061E60) Err#2 ENOENT
/1: 0.0095 xstat(2, "/www/cidera2/00/10D/000193E0", 0x08061E60) = 0
/1: 0.0150 xstat(2, "/www/cidera2/01/123/001A7498", 0x08061E60) = 0
/1: 0.0081 xstat(2, "/www/cidera2/07/CE/0010F5B5", 0x08061E60) Err#2 ENOENT
/1: 0.0144 xstat(2, "/www/cidera2/00/55/001700E0", 0x08061E60) = 0
/1: 0.0100 xstat(2, "/www/cidera2/05/113/000CDCF0", 0x08061E60) Err#2 ENOENT
/1: 0.0094 xstat(2, "/www/cidera2/00/CF/0017B734", 0x08061E60) = 0
/1: 0.0064 xstat(2, "/www/cidera2/04/14F/000AF752", 0x08061E60) Err#2 ENOENT
/1: 0.0066 xstat(2, "/www/cidera2/04/121/0021326B", 0x08061E60) Err#2 ENOENT
/1: 0.0108 xstat(2, "/www/cidera2/06/E7/000EDB75", 0x08061E60) Err#2 ENOENT
/1: 0.0002 xstat(2, "/www/cidera2/09/D8/001584E5", 0x08061E60) Err#2 ENOENT
/1: 0.0089 xstat(2, "/www/cidera2/07/C5/0010E80E", 0x08061E60) Err#2 ENOENT
/1: 0.0070 xstat(2, "/www/cidera2/08/33/00124CE8", 0x08061E60) Err#2 ENOENT
/1: 0.0068 xstat(2, "/www/cidera2/07/8C/0010935A", 0x08061E60) Err#2 ENOENT
/1: 0.0072 xstat(2, "/www/cidera2/04/120/00213118", 0x08061E60) Err#2 ENOENT
/1: 0.0132 xstat(2, "/www/cidera2/02/96/0005613E", 0x08061E60) = 0
/1: 0.0003 xstat(2, "/www/cidera2/03/12B/000881FD", 0x08061E60) Err#2 ENOENT
/1: 0.0076 xstat(2, "/www/cidera2/08/24/00123617", 0x08061E60) Err#2 ENOENT
/1: 0.0076 xstat(2, "/www/cidera2/08/49/00126DF7", 0x08061E60) Err#2 ENOENT
/1: 0.0089 xstat(2, "/www/cidera2/08/60/00129100", 0x08061E60) Err#2 ENOENT
/1: 0.0043 xstat(2, "/www/cidera2/03/F6/001EB24F", 0x08061E60) = 0
/1: 0.0075 xstat(2, "/www/cidera2/05/F6/000CB27D", 0x08061E60) Err#2 ENOENT
/1: 0.0002 xstat(2, "/www/cidera2/07/D6/0011011B", 0x08061E60) Err#2 ENOENT
/1: 0.0188 xstat(2, "/www/cidera2/01/7D/0002FBC6", 0x08061E60) = 0
/1: 0.0003 xstat(2, "/www/cidera2/00/1D/0016AC3A", 0x08061E60) Err#2 ENOENT
/1: 0.0102 xstat(2, "/www/cidera2/04/109/000A8EE2", 0x08061E60) Err#2 ENOENT
/1: 0.0042 xstat(2, "/www/cidera2/03/E8/00081CFC", 0x08061E60) = 0
/1: 0.0116 xstat(2, "/www/cidera2/03/78/001DF420", 0x08061E60) Err#2 ENOENT
/1: 0.0076 xstat(2, "/www/cidera2/00/170/00022842", 0x08061E60) = 0
/1: 0.0040 xstat(2, "/www/cidera2/05/12F/000D0710", 0x08061E60) Err#2 ENOENT
/1: 0.0003 xstat(2, "/www/cidera2/02/12C/001CC245", 0x08061E60) Err#2 ENOENT
/1: 0.0067 xstat(2, "/www/cidera2/02/16/0004A1C6", 0x08061E60) = 0
/1: 0.0092 xstat(2, "/www/cidera2/09/1D/00146C4B", 0x08061E60) Err#2 ENOENT
Done Reading Filenames: 66

real 0.588
user 0.006
sys 0.015

./a.out -g -l 384 -L 10 -n 66 -m 1234234 -p /www/cidera2 -s | ptime truss -lD -t stat ./a.out
/1: 0.0140 xstat(2, "/www/cidera2/03/79/001DF606", 0x08061E60) = 0
/1: 0.0007 xstat(2, "/www/cidera2/03/79/001DF607", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF608", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF609", 0x08061E60) = 0
/1: 0.0003 xstat(2, "/www/cidera2/03/79/001DF60A", 0x08061E60) = 0
/1: 0.0001 xstat(2, "/www/cidera2/03/79/001DF60B", 0x08061E60) = 0
/1: 0.0006 xstat(2, "/www/cidera2/03/79/001DF60C", 0x08061E60) = 0
/1: 0.0006 xstat(2, "/www/cidera2/03/79/001DF60D", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF60E", 0x08061E60) = 0
/1: 0.0001 xstat(2, "/www/cidera2/03/79/001DF60F", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF610", 0x08061E60) = 0
/1: 0.0003 xstat(2, "/www/cidera2/03/79/001DF611", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF612", 0x08061E60) = 0
/1: 0.0004 xstat(2, "/www/cidera2/03/79/001DF613", 0x08061E60) = 0
/1: 0.0048 xstat(2, "/www/cidera2/03/79/001DF614", 0x08061E60) = 0
/1: 0.0007 xstat(2, "/www/cidera2/03/79/001DF615", 0x08061E60) = 0
/1: 0.0005 xstat(2, "/www/cidera2/03/79/001DF616", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF617", 0x08061E60) = 0
/1: 0.0010 xstat(2, "/www/cidera2/03/79/001DF618", 0x08061E60) = 0
/1: 0.0006 xstat(2, "/www/cidera2/03/79/001DF619", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF61A", 0x08061E60) = 0
/1: 0.0005 xstat(2, "/www/cidera2/03/79/001DF61B", 0x08061E60) = 0
/1: 0.0006 xstat(2, "/www/cidera2/03/79/001DF61C", 0x08061E60) = 0
/1: 0.0015 xstat(2, "/www/cidera2/03/79/001DF61D", 0x08061E60) = 0
/1: 0.0006 xstat(2, "/www/cidera2/03/79/001DF61E", 0x08061E60) = 0
/1: 0.0010 xstat(2, "/www/cidera2/03/79/001DF61F", 0x08061E60) = 0
/1: 0.0003 xstat(2, "/www/cidera2/03/79/001DF620", 0x08061E60) Err#2 ENOENT
/1: 0.0008 xstat(2, "/www/cidera2/03/79/001DF621", 0x08061E60) = 0
/1: 0.0003 xstat(2, "/www/cidera2/03/79/001DF622", 0x08061E60) Err#2 ENOENT
/1: 0.0003 xstat(2, "/www/cidera2/03/79/001DF623", 0x08061E60) Err#2 ENOENT
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF624", 0x08061E60) Err#2 ENOENT
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF625", 0x08061E60) Err#2 ENOENT
/1: 0.0003 xstat(2, "/www/cidera2/03/79/001DF626", 0x08061E60) Err#2 ENOENT
/1: 0.0006 xstat(2, "/www/cidera2/03/79/001DF627", 0x08061E60) = 0
/1: 0.0010 xstat(2, "/www/cidera2/03/79/001DF628", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF629", 0x08061E60) = 0
/1: 0.0008 xstat(2, "/www/cidera2/03/79/001DF62A", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF62B", 0x08061E60) = 0
/1: 0.0001 xstat(2, "/www/cidera2/03/79/001DF62C", 0x08061E60) = 0
/1: 0.0005 xstat(2, "/www/cidera2/03/79/001DF62D", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF62E", 0x08061E60) = 0
/1: 0.0001 xstat(2, "/www/cidera2/03/79/001DF62F", 0x08061E60) = 0
/1: 0.0003 xstat(2, "/www/cidera2/03/79/001DF630", 0x08061E60) Err#2 ENOENT
/1: 0.0006 xstat(2, "/www/cidera2/03/79/001DF631", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF632", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF633", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF634", 0x08061E60) = 0
/1: 0.0007 xstat(2, "/www/cidera2/03/79/001DF635", 0x08061E60) = 0
/1: 0.0001 xstat(2, "/www/cidera2/03/79/001DF636", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF637", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF638", 0x08061E60) = 0
/1: 0.0001 xstat(2, "/www/cidera2/03/79/001DF639", 0x08061E60) = 0
/1: 0.0007 xstat(2, "/www/cidera2/03/79/001DF63A", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF63B", 0x08061E60) = 0
/1: 0.0001 xstat(2, "/www/cidera2/03/79/001DF63C", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF63D", 0x08061E60) = 0
/1: 0.0006 xstat(2, "/www/cidera2/03/79/001DF63E", 0x08061E60) = 0
/1: 0.0001 xstat(2, "/www/cidera2/03/79/001DF63F", 0x08061E60) = 0
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF640", 0x08061E60) = 0
/1: 0.0001 xstat(2, "/www/cidera2/03/79/001DF641", 0x08061E60) = 0
/1: 0.0003 xstat(2, "/www/cidera2/03/79/001DF642", 0x08061E60) Err#2 ENOENT
/1: 0.0002 xstat(2, "/www/cidera2/03/79/001DF643", 0x08061E60) Err#2 ENOENT
/1: 0.0007 xstat(2, "/www/cidera2/03/79/001DF644", 0x08061E60) Err#2 ENOENT
/1: 0.0003 xstat(2, "/www/cidera2/03/79/001DF645", 0x08061E60) Err#2 ENOENT
/1: 0.0004 xstat(2, "/www/cidera2/03/79/001DF646", 0x08061E60) Err#2 ENOENT
/1: 0.0006 xstat(2, "/www/cidera2/03/79/001DF647", 0x08061E60) Err#2 ENOENT
Done Reading Filenames: 66

real 0.057
user 0.006
sys 0.014

------------------------------------
 Andres Kroonmaa <andre@online.ee>
 Delfi Online
 Tel: 6501 731, Fax: 6501 708
 Pärnu mnt. 158, Tallinn,
 11317 Estonia
Received on Tue Nov 14 2000 - 02:55:59 MST