[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: "Test results"

This page is part of the web mail archives of SRFI 64 from before July 7th, 2015. The new archives for SRFI 64 contain all messages, not just those from before July 7th, 2015.



Sorry for the long dormancy on SRFI-64.  I'll post an update soon.
But first I'd like some feedback on an API for "test results":

A test runner maintains a set of "result properties" associated with
the current or most recent test.  (I.e. the properties of the
most recent test are available as long as a new test hasn't started.)

Each property has a name (a symbol) and a value (any value).
Some properties are standard or set by the implementation.

(test-result-ref [runner] 'pname [default])
Returns the property value associated with the pname property name.
If there is no value assocate with 'pname return default,
or #t if default isn't specified.

(test-result-set! [runner] 'pname value)
Sets the property value associated with the pname property name to value.

(test-result-clear [runner] ['pname])
If 'pname is specified, remove the property with the name 'pname.
If 'pname is not specified, remove all result properties.
The implementation automatically calls test-result-clear
at the start of a test-assert and similar procedures.

(test-result-alist [runner])
Returns an association list of the current result properties.
It is unspecified if the result shares state with the test-runner.
The result should not be modified; on the other hand the result
may be inclicitly modified by future test-result-set! or
test-result-clear calls, with one exception: A test-result-clear
without a 'pname doesn't not modify the returned alist.  Thus you
can "archive" result objects from previous runs.
Issue: Or should we just say that test-result-alist returns a
fresh alist?

We also need a list of standard properties:
'kind
  One of 'pass, 'xpass, 'fail, 'xfail, 'skip

'source-file
'source-line
Location of test-assert etc in test suite source code.

'source-form
The source form, if meaningful/available.

'expected-result
The expected result, if meaningful/available.

'actual-result
The actual result, if meaningful/available.

The test-runner call-back functions (such as that bound using
test-runner-on-test) is modified to only take a test-runner,
rather than also take an explicit test-result, since the latter
is now part of the former.
--
	--Per Bothner
per@xxxxxxxxxxx   http://per.bothner.com/