[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Meta-test suite

This page is part of the web mail archives of SRFI 64 from before July 7th, 2015. The new archives for SRFI 64 contain all messages, not just those from before July 7th, 2015.

Donovan Kolbly wrote:

So that, suppose:

   (test-begin "a")
   (test-begin "b")
   (test-assert "x" #t)

then in an on-test hook executing for the test-assert, we have:

   (test-runner-test-name runner) ==> "x"
   (test-runner-group-path runner) ==> ("a" "b")


Yes.
Although, then a custom runner would have no way to know about an empty group.
Would you intend that, for a custom runner, the following are indistinguishable: 

   (test-begin "a")
   (test-assert "x" #t)
   (test-end)
   (test-begin "a")
   (test-assert "y" #t)
   (test-end)

vs:

   (test-begin "a")
   (test-assert "x" #t)
   (test-assert "y" #t)
   (test-end)


I do think we should have call-back routins for test-begin/test-end.
For example a test-runner might want to write in a log:
[Entering group a]
Test x PASS.
[Exiting group a]
[Entering group a]
Test y PASS.
[Exiting group a]



A "test suite" is a collection of "test cases" optionally organized into a
hierarchy of "test groups".  A "test group" may be either implict, as
introduced by test-begin and terminated by test-end, or explicit, as
introduced by test-group.


Agreed.


Furthermore, note that the tests within an
explicit test group may be skipped using test-skip, but this is not true
for tests within an implicit test group.
[I think it's wierd that you can't skip an implicit test group, just for symmetry. Although I can't see implementing it so that all the forms are skipped, it might make sense to actually skip *tests* inside a skippable 
implicit group.]


I'm inclined to agree.  I don't think the reference implementation does
it this way, but it should be difficult to fix.

For test-match-nth, should be count both the test-group/test-begin *and*
(assuming the test-group/test-begin is not skipped) the tests in it?

(test-skip 2) ;; Define this to mean skipping the 2nd following test.
(test-begin "a")
(test-assert "x1")
(test-assert "x2")
(test-end "a")
(test-assert "x3")

Should we skip "x2" or "x3"?  I.e. do we count 1 for "a" as a unit, or 1
for "a" and 1 each for "x1"..."x3".  The latter might be a little strange,
but perhaps more convenient - and easier to implement, since we can use
a single global counter.


For the purposes of test counting as checked by test-end, a test group (of
either variety) counts as a single test.


I'm open to discussion on this: perhaps it should count as a single test
*if it is skipped*.

> Note that the registered on-test

procedure is not involved in test group boundaries, which means that the
sum of (test-runner-*-count) may be larger than the number of calls to the
on-test procedure.


Yes, I think so.  I we add test-begin/test-end call-backs, as I think we
should, we might be able to be more specific.

> Furthermore, if an explicit test group is skipped, the

group as a whole is skipped, not each test within it, which means that
(test-runner-skip-count) may be *smaller* than if the tests were
individually skipped.


This seems unavoidable.
--
	--Per Bothner
per@xxxxxxxxxxx   http://per.bothner.com/