[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Meta-test suite

Donovan Kolbly wrote:

So that, suppose:

   (test-begin "a")
   (test-begin "b")
   (test-assert "x" #t)

then in an on-test hook executing for the test-assert, we have:

   (test-runner-test-name runner) ==> "x"
   (test-runner-group-path runner) ==> ("a" "b")


Yes.
Although, then a custom runner would have no way to know about an empty group.
Would you intend that, for a custom runner, the following are indistinguishable: 

   (test-begin "a")
   (test-assert "x" #t)
   (test-end)
   (test-begin "a")
   (test-assert "y" #t)
   (test-end)

vs:

   (test-begin "a")
   (test-assert "x" #t)
   (test-assert "y" #t)
   (test-end)


I do think we should have call-back routins for test-begin/test-end.
For example a test-runner might want to write in a log:
[Entering group a]
Test x PASS.
[Exiting group a]
[Entering group a]
Test y PASS.
[Exiting group a]



A "test suite" is a collection of "test cases" optionally organized into a
hierarchy of "test groups".  A "test group" may be either implict, as
introduced by test-begin and terminated by test-end, or explicit, as
introduced by test-group.


Agreed.


Furthermore, note that the tests within an
explicit test group may be skipped using test-skip, but this is not true
for tests within an implicit test group.
[I think it's wierd that you can't skip an implicit test group, just for symmetry. Although I can't see implementing it so that all the forms are skipped, it might make sense to actually skip *tests* inside a skippable 
implicit group.]


I'm inclined to agree.  I don't think the reference implementation does
it this way, but it should be difficult to fix.

For test-match-nth, should be count both the test-group/test-begin *and*
(assuming the test-group/test-begin is not skipped) the tests in it?

(test-skip 2) ;; Define this to mean skipping the 2nd following test.
(test-begin "a")
(test-assert "x1")
(test-assert "x2")
(test-end "a")
(test-assert "x3")

Should we skip "x2" or "x3"?  I.e. do we count 1 for "a" as a unit, or 1
for "a" and 1 each for "x1"..."x3".  The latter might be a little strange,
but perhaps more convenient - and easier to implement, since we can use
a single global counter.


For the purposes of test counting as checked by test-end, a test group (of
either variety) counts as a single test.


I'm open to discussion on this: perhaps it should count as a single test
*if it is skipped*.

> Note that the registered on-test

procedure is not involved in test group boundaries, which means that the
sum of (test-runner-*-count) may be larger than the number of calls to the
on-test procedure.


Yes, I think so.  I we add test-begin/test-end call-backs, as I think we
should, we might be able to be more specific.

> Furthermore, if an explicit test group is skipped, the

group as a whole is skipped, not each test within it, which means that
(test-runner-skip-count) may be *smaller* than if the tests were
individually skipped.


This seems unavoidable.
--
	--Per Bothner
per@xxxxxxxxxxx   http://per.bothner.com/