Re: Meta-test suite

[Donovan Kolbly <donovan@xxxxxxxxxxx>]

On Wed, 9 Mar 2005, Per Bothner wrote:

> Donovan Kolbly wrote:
> 
> > More crucially, there does not seem to be any way for a portable runner to
> > keep track of the nesting of test suites.
> 
> The API should probably have a way to get the value of (test-runner-group-path).
> One detail: the latter has the suite-names in reverse order, since the
> implementation uses a stack; a public function should probably return the
> names in top-to-botton order.  That's a minor tweak.

So that, suppose:

   (test-begin "a")
   (test-begin "b")
   (test-assert "x" #t)

then in an on-test hook executing for the test-assert, we have:

   (test-runner-test-name runner) ==> "x"
   (test-runner-group-path runner) ==> ("a" "b")

That seems reasonable.

Although, then a custom runner would have no way to know about an empty 
group.

Would you intend that, for a custom runner, the following are 
indistinguishable:

   (test-begin "a")
   (test-assert "x" #t)
   (test-end)
   (test-begin "a")
   (test-assert "y" #t)
   (test-end)

vs:

   (test-begin "a")
   (test-assert "x" #t)
   (test-assert "y" #t)
   (test-end)

My tendency goes more to object identity, but I doubt it matters much 
here.

> 
> > I'm not sure yet what to
> > suggest for SRFI-64 to track suite nesting, but something along the lines
> > of a `test-runner-on-suite-begin!' and `test-runner-on-suite-end!' would
> > probably do it (*).
> 
> That might be useful for some things, though I don't think it should be needed
> to keep track of the "group-path", which the framework should do.
> 
> > [*] do you want to call these things "suites" or "groups"?  The
> >     terminology seems to vary from place to place, or am I still missing
> >     something....
> 
> Hm.  I think a "test suite" is a collection of tests, which consists of
> individualt "test cases", which are usually grouped into "test groups".
> A test-begin/test-end-pair together with whatever's between them is an
> implicit/informal test group, while a test-group is an explicit test group.
> 
> A test suite is whatever gets run user a test runner.
> 
> Does that sound reasonable?  If so, "suite-name" should be "group-name".

Sounds good.

To say it another way:

A "test suite" is a collection of "test cases" optionally organized into a
hierarchy of "test groups".  A "test group" may be either implict, as
introduced by test-begin and terminated by test-end, or explicit, as
introduced by test-group.  Furthermore, note that the tests within an
explicit test group may be skipped using test-skip, but this is not true
for tests within an implicit test group.

[I think it's wierd that you can't skip an implicit test group, just for 
symmetry.  Although I can't see implementing it so that all the forms are 
skipped, it might make sense to actually skip *tests* inside a skippable
implicit group.]

For the purposes of test counting as checked by test-end, a test group (of
either variety) counts as a single test.  Note that the registered on-test
procedure is not involved in test group boundaries, which means that the
sum of (test-runner-*-count) may be larger than the number of calls to the
on-test procedure.  Furthermore, if an explicit test group is skipped, the
group as a whole is skipped, not each test within it, which means that
(test-runner-skip-count) may be *smaller* than if the tests were
individually skipped.

-- 
-- Donovan Kolbly                    (  d.kolbly@xxxxxxxxxxx
				     (  http://www.rscheme.org/~donovan/