Re: Skipping selected tests which are groups

[Per Bothner <per@xxxxxxxxxxx>]

Donovan Kolbly wrote:
> I am trying to figure out the intended interaction between "test-skip" and 
> "test-begin".
>
> The spec says that "Before each test (or begin-group [sic: I assume this
> means 'test-begin'])

I think I meant 'test-group'.

the set of active skip-specifiers are applied to the
> active test-runner. If any specifier matches, then the test is skipped."
>
> What is "the test" when it's a test-begin?  Are all the tests inside the 
> group supposed to be skipped, or is all the code evaluable between the 
> start and end of the test group supposed to be skipped.
>
> Consider:
>
>   (test-skip "test-b")
>   (test-begin "test-b")
>   (do-some-stuff)
>   (test-assert "b.1" (whatever))
>   (test-assert "b.2" (foo))
>   (test-end "test-b")
>
> Is (do-some-stuff) supposed to get evaluated?

Yes - there isn't really any way to avoid it.

However, I thik you may be misunderstanding the difference between a
test-name and a suite-name.  The (test-skip "test-b") is short-hand
for (test-skip (test-matched-named "test-b")) - and there are no tests
names "test-b" - only a *test suite* named "test-b".

At least that's the current specification and reference implementation.
There is no (test-match-suite "suite-name") - there probably should be.

Here's an illustration, leaving aside the issue of test names:

  (test-skip (test-match-nth 1 2)) ;; skip the next two tests
  (test-begin "test-b")           ; executed
  (do-some-stuff)                 ; executed
  (test-assert "b.1" (whatever))  ; first test skipped
  (test-assert "b.2" (foo))       ; 2nd test skipped
  (test-assert "b.3" (foo))       ; not skippped??
  (test-end "test-b")

One could ask: should all the tests within the test-begin/test-end
group count as a single tests when deciding which tests to skip?
That's not the case in the current reference implementation, I believe,
but a test-group does count as a single test.

Btw: another reasonable should hand might be to interpret (test-skip N [M])
for integers N/M as short-hand for (test-skip (test-matching N [M]))

> ...or... does "begin-group" in the spec mean "test-group", in which case
> the user is obliged to lexically nest any forms which are to actually be
> skipped?
>
> e.g., this behavior is a little [ :-) ] easier to implement:
>
>   (test-skip "test-b")                                                              
>   (test-group "test-b"
>     (do-some-stuff)                    ; skip
>     (test-assert "b.1" (whatever))     ; skip
>     (test-assert "b.2" (foo)))         ; skip
>   (test-end "test-b")

Right except for the issue of whether the test-name "test-b" matches
the suite-name "test-b".

The intention - and I believe the reference implementation - is that the
decision to skip is made for each test-assert/test-eq*/etc *and* for each
test-group.

> Also, do skipped tests count for the purposes of the `count' argument to
> test-end?  I assume so, or else much pain to any who uses both features!  
> I suppose that is why SKIP is mentioned as a possible test result: to take 
> up a slot in list whose length must be = to `count'.

Yes, that is the intention.  Though it doesn't appear that count is actually
check in the RI.  Sigh.  I made a note ...

By the way: I appreciate you trying out the API, and giving feedback.
It's very valuable that somebody familiar with a Scheme implementation try
to "port" the API and see how well it works.  Plus I haven't had a time to
expierment with it much myself.
--
	--Per Bothner
per@xxxxxxxxxxx   http://per.bothner.com/