[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: "Test results"

Sorry for the long dormancy on SRFI-64.  I'll post an update soon.
But first I'd like some feedback on an API for "test results":

A test runner maintains a set of "result properties" associated with
the current or most recent test.  (I.e. the properties of the
most recent test are available as long as a new test hasn't started.)

Each property has a name (a symbol) and a value (any value).
Some properties are standard or set by the implementation.

(test-result-ref [runner] 'pname [default])
Returns the property value associated with the pname property name.
If there is no value assocate with 'pname return default,
or #t if default isn't specified.

(test-result-set! [runner] 'pname value)
Sets the property value associated with the pname property name to value.

(test-result-clear [runner] ['pname])
If 'pname is specified, remove the property with the name 'pname.
If 'pname is not specified, remove all result properties.
The implementation automatically calls test-result-clear
at the start of a test-assert and similar procedures.

(test-result-alist [runner])
Returns an association list of the current result properties.
It is unspecified if the result shares state with the test-runner.
The result should not be modified; on the other hand the result
may be inclicitly modified by future test-result-set! or
test-result-clear calls, with one exception: A test-result-clear
without a 'pname doesn't not modify the returned alist.  Thus you
can "archive" result objects from previous runs.
Issue: Or should we just say that test-result-alist returns a
fresh alist?

We also need a list of standard properties:
  One of 'pass, 'xpass, 'fail, 'xfail, 'skip

Location of test-assert etc in test suite source code.

The source form, if meaningful/available.

The expected result, if meaningful/available.

The actual result, if meaningful/available.

The test-runner call-back functions (such as that bound using
test-runner-on-test) is modified to only take a test-runner,
rather than also take an explicit test-result, since the latter
is now part of the former.
	--Per Bothner
per@xxxxxxxxxxx   http://per.bothner.com/