[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Meta-test suite

This page is part of the web mail archives of SRFI 64 from before July 7th, 2015. The new archives for SRFI 64 contain all messages, not just those from before July 7th, 2015.

On Wed, 9 Mar 2005, Per Bothner wrote:

> Donovan Kolbly wrote:
> > More crucially, there does not seem to be any way for a portable runner to
> > keep track of the nesting of test suites.
> The API should probably have a way to get the value of (test-runner-group-path).
> One detail: the latter has the suite-names in reverse order, since the
> implementation uses a stack; a public function should probably return the
> names in top-to-botton order.  That's a minor tweak.

So that, suppose:

   (test-begin "a")
   (test-begin "b")
   (test-assert "x" #t)

then in an on-test hook executing for the test-assert, we have:

   (test-runner-test-name runner) ==> "x"
   (test-runner-group-path runner) ==> ("a" "b")

That seems reasonable.

Although, then a custom runner would have no way to know about an empty 

Would you intend that, for a custom runner, the following are 

   (test-begin "a")
   (test-assert "x" #t)
   (test-begin "a")
   (test-assert "y" #t)


   (test-begin "a")
   (test-assert "x" #t)
   (test-assert "y" #t)

My tendency goes more to object identity, but I doubt it matters much 

> > I'm not sure yet what to
> > suggest for SRFI-64 to track suite nesting, but something along the lines
> > of a `test-runner-on-suite-begin!' and `test-runner-on-suite-end!' would
> > probably do it (*).
> That might be useful for some things, though I don't think it should be needed
> to keep track of the "group-path", which the framework should do.
> > [*] do you want to call these things "suites" or "groups"?  The
> >     terminology seems to vary from place to place, or am I still missing
> >     something....
> Hm.  I think a "test suite" is a collection of tests, which consists of
> individualt "test cases", which are usually grouped into "test groups".
> A test-begin/test-end-pair together with whatever's between them is an
> implicit/informal test group, while a test-group is an explicit test group.
> A test suite is whatever gets run user a test runner.
> Does that sound reasonable?  If so, "suite-name" should be "group-name".

Sounds good.

To say it another way:

A "test suite" is a collection of "test cases" optionally organized into a
hierarchy of "test groups".  A "test group" may be either implict, as
introduced by test-begin and terminated by test-end, or explicit, as
introduced by test-group.  Furthermore, note that the tests within an
explicit test group may be skipped using test-skip, but this is not true
for tests within an implicit test group.

[I think it's wierd that you can't skip an implicit test group, just for 
symmetry.  Although I can't see implementing it so that all the forms are 
skipped, it might make sense to actually skip *tests* inside a skippable
implicit group.]

For the purposes of test counting as checked by test-end, a test group (of
either variety) counts as a single test.  Note that the registered on-test
procedure is not involved in test group boundaries, which means that the
sum of (test-runner-*-count) may be larger than the number of calls to the
on-test procedure.  Furthermore, if an explicit test group is skipped, the
group as a whole is skipped, not each test within it, which means that
(test-runner-skip-count) may be *smaller* than if the tests were
individually skipped.

-- Donovan Kolbly                    (  d.kolbly@xxxxxxxxxxx
				     (  http://www.rscheme.org/~donovan/