Re: [PATCH] Generated SBuf::find() test cases

On 02/25/2013 03:45 PM, Amos Jeffries wrote:
> On 26/02/2013 11:06 a.m., Alex Rousskov wrote:
>> The attached patch adds about 700'000 test cases for eight
>> SBuf::find() methods. Needless to say, they are generated rather than
>> manually written. They take a couple of seconds to run. Configuration
>> options allow us to increase or decrease the number of test cases if
>> needed.

> Why do we need so many tests?

I do not know how many tests are needed. Somebody can study that (if
they feel that reducing the number of generated tests is important).
More on that below.

> Surely a selection of lengths from the middle, and specific scans of all
> the edge-boundary tests would be enough?

That is what the random tests are doing. One could massage them into
using shorter strings (already a parameter), selecting fewer lengths
from the middle (a simple code modification), and/or doing fewer
edge-boundary tests (a simple code modification), of course.

> That can't be more than a few hundred - which is few enough to run each
> as a separate CppUnit case.

I have not studied whether doing so will reduce the number of bugs
found. I only studied that going in the opposite direction (more test
cases) does not seem to help much. Please note that it is difficult to
know for sure because while many test cases find bugs, it is difficult
to isolate independent/different bugs without actually fixing SBuf.

Why is "a few hundred" is OK as a separate CppUnit case but "a few
hundred thousands" is not OK? I am _not_ trying to argue either way,
just curious what the criteria for sizing CppUnit cases are, since I
know so little about CppUnit...

If more CppUnit cases are needed, I am sure it would be possible to
parameterize the current code to generate more specific test cases.
After repeating such "focused" generation inside each appropriate
CppUnit case, one would end up with multiple CppUnit test cases covering
approximately the same input space.

> Also, using fuzzing with a random input leaves us open to the rare
> occasions when a single byte octet value at a specific location is the
> cause of a bug. Think \0 at octet 5 when doing 32-bit chunk comparisions
> in some buggy implementation checking against !*p. That type of failure
> appears intermittently and inspires confusion rather than a fix.

Bugs notwithstanding, all random input is 100% reproducible so there
should not be any intermittent behavior originating from the test cases
themselves (at least as far as the same execution environment is
concerned -- I do not know whether random() implementations yield
different sequences on different platforms).

HTH,

Alex.
Received on Tue Feb 26 2013 - 00:08:16 MST