[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Skipping selected tests which are groups



Donovan Kolbly wrote:
I am trying to figure out the intended interaction between "test-skip" and "test-begin".

The spec says that "Before each test (or begin-group [sic: I assume this
means 'test-begin'])

I think I meant 'test-group'.

the set of active skip-specifiers are applied to the
active test-runner. If any specifier matches, then the test is skipped."

What is "the test" when it's a test-begin? Are all the tests inside the group supposed to be skipped, or is all the code evaluable between the start and end of the test group supposed to be skipped.

Consider:

  (test-skip "test-b")
  (test-begin "test-b")
  (do-some-stuff)
  (test-assert "b.1" (whatever))
  (test-assert "b.2" (foo))
  (test-end "test-b")

Is (do-some-stuff) supposed to get evaluated?

Yes - there isn't really any way to avoid it.

However, I thik you may be misunderstanding the difference between a
test-name and a suite-name.  The (test-skip "test-b") is short-hand
for (test-skip (test-matched-named "test-b")) - and there are no tests
names "test-b" - only a *test suite* named "test-b".

At least that's the current specification and reference implementation.
There is no (test-match-suite "suite-name") - there probably should be.

Here's an illustration, leaving aside the issue of test names:

  (test-skip (test-match-nth 1 2)) ;; skip the next two tests
  (test-begin "test-b")           ; executed
  (do-some-stuff)                 ; executed
  (test-assert "b.1" (whatever))  ; first test skipped
  (test-assert "b.2" (foo))       ; 2nd test skipped
  (test-assert "b.3" (foo))       ; not skippped??
  (test-end "test-b")

One could ask: should all the tests within the test-begin/test-end
group count as a single tests when deciding which tests to skip?
That's not the case in the current reference implementation, I believe,
but a test-group does count as a single test.

Btw: another reasonable should hand might be to interpret (test-skip N [M])
for integers N/M as short-hand for (test-skip (test-matching N [M]))

...or... does "begin-group" in the spec mean "test-group", in which case
the user is obliged to lexically nest any forms which are to actually be
skipped?

e.g., this behavior is a little [ :-) ] easier to implement:

(test-skip "test-b") (test-group "test-b"
    (do-some-stuff)                    ; skip
    (test-assert "b.1" (whatever))     ; skip
    (test-assert "b.2" (foo)))         ; skip
  (test-end "test-b")

Right except for the issue of whether the test-name "test-b" matches
the suite-name "test-b".

The intention - and I believe the reference implementation - is that the
decision to skip is made for each test-assert/test-eq*/etc *and* for each
test-group.

Also, do skipped tests count for the purposes of the `count' argument to
test-end? I assume so, or else much pain to any who uses both features! I suppose that is why SKIP is mentioned as a possible test result: to take up a slot in list whose length must be = to `count'.

Yes, that is the intention.  Though it doesn't appear that count is actually
check in the RI.  Sigh.  I made a note ...

By the way: I appreciate you trying out the API, and giving feedback.
It's very valuable that somebody familiar with a Scheme implementation try
to "port" the API and see how well it works.  Plus I haven't had a time to
expierment with it much myself.
--
	--Per Bothner
per@xxxxxxxxxxx   http://per.bothner.com/