Counting [was Re: Meta-test suite]

[Donovan Kolbly <donovan@xxxxxxxxxxx>]

On Wed, 9 Mar 2005, Per Bothner wrote:

> Donovan Kolbly wrote:
> > So that, suppose:
> > 
> >    (test-begin "a")
> >    (test-begin "b")
> >    (test-assert "x" #t)
> > 
> > then in an on-test hook executing for the test-assert, we have:
> > 
> >    (test-runner-test-name runner) ==> "x"
> >    (test-runner-group-path runner) ==> ("a" "b")
> 
> Yes.

I've added `test-runner-group-path' to my implementation and some tests 
for it.

> 
> I do think we should have call-back routins for test-begin/test-end [...]

In consistency with the other callbacks in the spec, I'd guess you imagine
something like `test-runner-on-test-{begin,end}[!]'.  Other ideas?  You
could make test-group have a seperate callback, or define it in terms of
begin/end.


> > Furthermore, note that the tests within an
> > explicit test group may be skipped using test-skip, but this is not true
> > for tests within an implicit test group.
> > 
> > [I think it's wierd that you can't skip an implicit test group, just for 
> > symmetry.  Although I can't see implementing it so that all the forms are 
> > skipped, it might make sense to actually skip *tests* inside a skippable
> > implicit group.]
> 
> I'm inclined to agree.  I don't think the reference implementation does
> it this way, but it should be difficult to fix.
> 
> For test-match-nth, should be count both the test-group/test-begin *and*
> (assuming the test-group/test-begin is not skipped) the tests in it?
> 
> (test-skip 2) ;; Define this to mean skipping the 2nd following test.
> (test-begin "a")
> (test-assert "x1")
> (test-assert "x2")
> (test-end "a")
> (test-assert "x3")
> 
> Should we skip "x2" or "x3"?  I.e. do we count 1 for "a" as a unit, or 1
> for "a" and 1 each for "x1"..."x3".  The latter might be a little strange,
> but perhaps more convenient - and easier to implement, since we can use
> a single global counter.

I could see implementing it either way; I think it would be a matter of
whether you check the "are-we-in-a-skipped-implicit-group?" flag before or
after you call to check the specifiers in the current skip list.

I don't have a good intuition as to the use of test-match-nth -- is this 
something you've used elsewhere before?

In the absence of prior art, I'd suggest consistency with the use of 
test-group.  In other words, I'd like to preserve as much as possible the 
similarity between

   (test-begin "a")
   ...
   (test-end "a")

and

   (test-group "a" ...)

And clarify in the spec that the only reason they aren't equivalent (i.e.,
(1) the evaluation of non-test forms in the implicit-group case, and (2)  
the execution of the test-end in case of an exception) is to preserve the
sanity of the implementor.

In that case, we'd skip "x3", since all of "a" would count as the 1st.

> 
> > For the purposes of test counting as checked by test-end, a test group (of
> > either variety) counts as a single test.
> 
> I'm open to discussion on this: perhaps it should count as a single test
> *if it is skipped*.

I think I'm getting confused by the multiple ways of counting.  So, to 
clarify, there are three places that counting happens:

  (1) the counting of (usually) test cases, as reported by 
      `test-runner-*-count'.

  (2) the counting of substructures, as checked by `test-end' with
      the count option

  (3) the counting of skip candidates, as checked by the 
      stateful predicate `test-match-nth'.

The RScheme implementation notwithstanding, I'm thinking the following 
makes sense:

(2) and (3) should be unified and (1) should be clarified to count only
test cases and never test groups.

Furthermore, (3) does *not* involve short-circuited evaluation, and we
should introduce a new accessor, `test-runner-count-in-group' which
returns the position (1,2,3...) of the current test case in it's group.

Then test-match-nth looks more like:

(define (test-match-nth n #optional (count default: 1))
  (let ((i (test-runner-count-in-group (test-runner-current))))
    (and (>= i n) (<= i (+ n count -1)))))

[Although some tweak is necessary to suspend the processing during 
subgroups; you only want the skip to match at the same structural level.  
Maybe it should be `test-runner-count-in-group-path' and return the path 
of positions.]

This approach implies that test groups (explicit or implicit) should
always count at one structural element and never appear in
(test-runner-skip-count).

-- 
-- Donovan Kolbly                    (  d.kolbly@xxxxxxxxxxx
				     (  http://www.rscheme.org/~donovan/