[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Counting [was Re: Meta-test suite]

This page is part of the web mail archives of SRFI 64 from before July 7th, 2015. The new archives for SRFI 64 contain all messages, not just those from before July 7th, 2015.

On Wed, 9 Mar 2005, Per Bothner wrote:

> Donovan Kolbly wrote:
> > So that, suppose:
> > 
> >    (test-begin "a")
> >    (test-begin "b")
> >    (test-assert "x" #t)
> > 
> > then in an on-test hook executing for the test-assert, we have:
> > 
> >    (test-runner-test-name runner) ==> "x"
> >    (test-runner-group-path runner) ==> ("a" "b")
> Yes.

I've added `test-runner-group-path' to my implementation and some tests 
for it.

> I do think we should have call-back routins for test-begin/test-end [...]

In consistency with the other callbacks in the spec, I'd guess you imagine
something like `test-runner-on-test-{begin,end}[!]'.  Other ideas?  You
could make test-group have a seperate callback, or define it in terms of

> > Furthermore, note that the tests within an
> > explicit test group may be skipped using test-skip, but this is not true
> > for tests within an implicit test group.
> > 
> > [I think it's wierd that you can't skip an implicit test group, just for 
> > symmetry.  Although I can't see implementing it so that all the forms are 
> > skipped, it might make sense to actually skip *tests* inside a skippable
> > implicit group.]
> I'm inclined to agree.  I don't think the reference implementation does
> it this way, but it should be difficult to fix.
> For test-match-nth, should be count both the test-group/test-begin *and*
> (assuming the test-group/test-begin is not skipped) the tests in it?
> (test-skip 2) ;; Define this to mean skipping the 2nd following test.
> (test-begin "a")
> (test-assert "x1")
> (test-assert "x2")
> (test-end "a")
> (test-assert "x3")
> Should we skip "x2" or "x3"?  I.e. do we count 1 for "a" as a unit, or 1
> for "a" and 1 each for "x1"..."x3".  The latter might be a little strange,
> but perhaps more convenient - and easier to implement, since we can use
> a single global counter.

I could see implementing it either way; I think it would be a matter of
whether you check the "are-we-in-a-skipped-implicit-group?" flag before or
after you call to check the specifiers in the current skip list.

I don't have a good intuition as to the use of test-match-nth -- is this 
something you've used elsewhere before?

In the absence of prior art, I'd suggest consistency with the use of 
test-group.  In other words, I'd like to preserve as much as possible the 
similarity between

   (test-begin "a")
   (test-end "a")


   (test-group "a" ...)

And clarify in the spec that the only reason they aren't equivalent (i.e.,
(1) the evaluation of non-test forms in the implicit-group case, and (2)  
the execution of the test-end in case of an exception) is to preserve the
sanity of the implementor.

In that case, we'd skip "x3", since all of "a" would count as the 1st.

> > For the purposes of test counting as checked by test-end, a test group (of
> > either variety) counts as a single test.
> I'm open to discussion on this: perhaps it should count as a single test
> *if it is skipped*.

I think I'm getting confused by the multiple ways of counting.  So, to 
clarify, there are three places that counting happens:

  (1) the counting of (usually) test cases, as reported by 

  (2) the counting of substructures, as checked by `test-end' with
      the count option

  (3) the counting of skip candidates, as checked by the 
      stateful predicate `test-match-nth'.

The RScheme implementation notwithstanding, I'm thinking the following 
makes sense:

(2) and (3) should be unified and (1) should be clarified to count only
test cases and never test groups.

Furthermore, (3) does *not* involve short-circuited evaluation, and we
should introduce a new accessor, `test-runner-count-in-group' which
returns the position (1,2,3...) of the current test case in it's group.

Then test-match-nth looks more like:

(define (test-match-nth n #optional (count default: 1))
  (let ((i (test-runner-count-in-group (test-runner-current))))
    (and (>= i n) (<= i (+ n count -1)))))

[Although some tweak is necessary to suspend the processing during 
subgroups; you only want the skip to match at the same structural level.  
Maybe it should be `test-runner-count-in-group-path' and return the path 
of positions.]

This approach implies that test groups (explicit or implicit) should
always count at one structural element and never appear in

-- Donovan Kolbly                    (  d.kolbly@xxxxxxxxxxx
				     (  http://www.rscheme.org/~donovan/