This page is part of the web mail archives of SRFI 64 from before July 7th, 2015. The new archives for SRFI 64 contain all messages, not just those from before July 7th, 2015.
On Tue, 5 Apr 2005, Per Bothner wrote: > Donovan Kolbly wrote: > So I'd probably go the other way and make test > results (or, more likely, > > test cases) first class, something like so: > > > > (test-case [runner]) -> current or most recent test case, or #f if none > > > > (test-case-kind tcase) -> one of (pass xpass ...) [*] > > (test-case-passed? tcase) ; same as (memq (test-case-kind tcase) '(pass xpass)) > > (test-case-property tcase 'property [value]) ; property setter/getter > > (test-case-name tcase) -> test name > > > > [* although 'kind' is looking like a strange name here; maybe > > "disposition"?] > > I think you're being inprecise wrt test-case vs test-result. > A test-case is something you execute and the result is a > test-result (or a test-case-result). I might re-run the asme > test-case multiple times, in a different environment. It's then > a single test-case, but multiple results. Perhaps the term "test case occurrence" or "instance" would be more precise. I do mean to refer to the concrete objects at the leaves of the test tree (as opposed to the abstract test cases which may be present in the test suite source code and possible reused in various ways). I also meant to make a distinction from the *result* which is essentially the output of the test case occurrence specifically because you want to hook in before the occurrence executes and affect the result (e.g., by deciding to skip it). BTW, in the subproposal, how are you thinking someone would write a runner that decides to skip some subsequent tests? Would the runner itself, during the on-test hook, invoke (test-skip (test-match-nth 1 999999))? I guess if you want to make that kind of decision before running a test case instead of after, you would make a custom specifier. For example, let's say I decide to skip all tests that have the substring "(only if ABC)" in their name, e.g., because the ABC test failed, I might execute a test-skip with a custom specifier when ABC fails. > Though perhaps you > could argue if it's run with a different environment, it's really > a different test-case. Still, I don't think using test-case to > mean a result is a good name. > > So the question: Is it useful to make test results first-class? > Perhaps. I can see some use cases: a testrunner that just saves > every result. Then perhaps you can have an interactive GUI, > where you can click on a result to see details, for example. Interestingly, that's exactly how my xmlrunner works. I haven't tried to port it to the bare API as a functionality test... A post-mortem live debugger might be another use case. e.g., when you run the test suite the runner captures the context of any error conditions and then you can drill in to debug any case that errored out. Well, anyway, all that is still possible without first-class test case occurrences in the API. A custom runner can wrap things into whatever first-class objects it wants, as long as the hooks are there. And I don't think there's anything in the subproposal to prevent an implementation from using such a representation internally. -- -- Donovan Kolbly ( d.kolbly@xxxxxxxxxxx ( http://www.rscheme.org/~donovan/