This page is part of the web mail archives of SRFI 64 from before July 7th, 2015. The new archives for SRFI 64 contain all messages, not just those from before July 7th, 2015.
On Mon, 15 Aug 2005, Per Bothner wrote:
But first I'd like some feedback on an API for "test results": A test runner maintains a set of "result properties" associated with the current or most recent test. (I.e. the properties of the most recent test are available as long as a new test hasn't started.) Each property has a name (a symbol) and a value (any value). Some properties are standard or set by the implementation.
This seems like a good general idea.
(test-result-ref [runner] 'pname [default])
Hmm. I think statically ambiguous interfaces are confusing. If you see the following (admittedly poorly written) code fragment:
(test-result-ref x y)you will need much more information (perhaps located arbitrarily far away in the program) to understand what is actually being done. Is the property denoted by y of runner x being accessed, or is property x of the current runner being accessed, with a default value of y?
And, of course, the compiler has to defer the disambiguation to runtime as well.
Returns the property value associated with the pname property name. If there is no value assocate with 'pname return default, or #t if default isn't specified.
Why #t? I would think that #f, as the Most Distinguished Value, is a more useful default default.
(test-result-set! [runner] 'pname value) Sets the property value associated with the pname property name to value. (test-result-clear [runner] ['pname])
Same ambiguity problem as mentioned above.
If 'pname is specified, remove the property with the name 'pname. If 'pname is not specified, remove all result properties. The implementation automatically calls test-result-clear at the start of a test-assert and similar procedures.
Do we need to say anything about when the standard properties might be "added"? My own implementation will not explicitly store standard properties, so in effect they'll (mostly) always be present.
(test-result-alist [runner]) Returns an association list of the current result properties. It is unspecified if the result shares state with the test-runner. The result should not be modified; on the other hand the result may be inclicitly modified by future test-result-set! or test-result-clear calls, with one exception: A test-result-clear without a 'pname doesn't not modify the returned alist. Thus you can "archive" result objects from previous runs. Issue: Or should we just say that test-result-alist returns a fresh alist?
I like archiving, but requiring a fresh alist is overly constraining. What if my implementation can share structure where possible (sort of a copy-on-write thing)? I'd hate to always have to clone the alist...
We also need a list of standard properties: 'kind One of 'pass, 'xpass, 'fail, 'xfail, 'skip 'source-file 'source-line Location of test-assert etc in test suite source code.
+ if meaningful/available
'source-form The source form, if meaningful/available. 'expected-result The expected result, if meaningful/available. 'actual-result The actual result, if meaningful/available.
I suggest the addition of (possibly optional) timestamps, including the execution time duration and the absolute start time.
The test-runner call-back functions (such as that bound using test-runner-on-test) is modified to only take a test-runner, rather than also take an explicit test-result, since the latter is now part of the former.
I'll try tweaking my meta-test suite to take these new concepts into account...
-- Donovan Kolbly ( d.kolbly@xxxxxxxxxxx ( http://www.rscheme.org/~donovan/