[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: "Test results"



On Mon, 15 Aug 2005, Per Bothner wrote:

But first I'd like some feedback on an API for "test results":

A test runner maintains a set of "result properties" associated with
the current or most recent test.  (I.e. the properties of the
most recent test are available as long as a new test hasn't started.)

Each property has a name (a symbol) and a value (any value).
Some properties are standard or set by the implementation.

This seems like a good general idea.


(test-result-ref [runner] 'pname [default])

Hmm. I think statically ambiguous interfaces are confusing. If you see the following (admittedly poorly written) code fragment:

   (test-result-ref x y)

you will need much more information (perhaps located arbitrarily far away in the program) to understand what is actually being done. Is the property denoted by y of runner x being accessed, or is property x of the current runner being accessed, with a default value of y?

And, of course, the compiler has to defer the disambiguation to runtime as well.

Returns the property value associated with the pname property name.
If there is no value assocate with 'pname return default,
or #t if default isn't specified.

Why #t? I would think that #f, as the Most Distinguished Value, is a more useful default default.


(test-result-set! [runner] 'pname value)
Sets the property value associated with the pname property name to value.

(test-result-clear [runner] ['pname])

Same ambiguity problem as mentioned above.

If 'pname is specified, remove the property with the name 'pname.
If 'pname is not specified, remove all result properties.
The implementation automatically calls test-result-clear
at the start of a test-assert and similar procedures.

Do we need to say anything about when the standard properties might be "added"? My own implementation will not explicitly store standard properties, so in effect they'll (mostly) always be present.


(test-result-alist [runner])
Returns an association list of the current result properties.
It is unspecified if the result shares state with the test-runner.
The result should not be modified; on the other hand the result
may be inclicitly modified by future test-result-set! or
test-result-clear calls, with one exception: A test-result-clear
without a 'pname doesn't not modify the returned alist.  Thus you
can "archive" result objects from previous runs.
Issue: Or should we just say that test-result-alist returns a
fresh alist?

I like archiving, but requiring a fresh alist is overly constraining. What if my implementation can share structure where possible (sort of a copy-on-write thing)? I'd hate to always have to clone the alist...


We also need a list of standard properties:
'kind
 One of 'pass, 'xpass, 'fail, 'xfail, 'skip

'source-file
'source-line
Location of test-assert etc in test suite source code.

+                  if meaningful/available


'source-form
The source form, if meaningful/available.

'expected-result
The expected result, if meaningful/available.

'actual-result
The actual result, if meaningful/available.

I suggest the addition of (possibly optional) timestamps, including the execution time duration and the absolute start time.

The test-runner call-back functions (such as that bound using
test-runner-on-test) is modified to only take a test-runner,
rather than also take an explicit test-result, since the latter
is now part of the former.

I'll try tweaking my meta-test suite to take these new concepts into account...

-- Donovan Kolbly                    (  d.kolbly@xxxxxxxxxxx
				     (  http://www.rscheme.org/~donovan/