Re: Suggestion: include test suite in SRFI

[Donovan Kolbly <donovan@rscheme.org>]

On Tue, 10 Nov 1998, Shriram Krishnamurthi wrote:

> Test suites evolve over time, especially as unforseen behaviors occur
> that need to be included or excluded.  These changes will occur after
> the SRFI has been adopted.  We currently shut down a SRFIs mailing
> list once it has been adopted.  Should we re-open the mailing list to
> discuss modifications to the test suite?  Who gets to decide on
> changes to the test suite at that point?  (Disputed changes will often 
> point to deeper semantic disagreements.)
> 
> 'shriram

I view the test component of a SRFI on a similar footing with the
reference implementation -- essentially an appendix to the specification.
The test component is intended to describe the SRFI by examples of
behavior, rather than imperatively as in the reference implementation. 

Hence, I think your question about reopening discussion to address
deficiencies in the test cases is equally applicable to deficiencies in
the reference implementation.  I intentionally borrowed language from that
section to imply that a SRFI can stand despite a deficient test case,
just as it stands despite an erroneous reference implementation.  Of
course, if it often happens that the test cases are broken, they lose
their value, just as often broken reference implementations would
devalue the process.

In addition, I am definitely thinking that the test cases should drive out
boundary or funny case behaviors.  Hopefully by including the tests in the
SRFI itself, the author can clarify by example what is intended in some
cases.  Note that the specification itself would perhaps include some
examples -- the proposed TEST-IMPLEMENTATION section would merely
formalize somewhat and make readily executable those examples.  (In fact,
I would be happy with a structural formality in the specification itself
from which the test cases could be extracted, but I think that would be a
little harder, and plus we want it to be pure HTML 3.2 anyway.)


The primary purpose for my suggestion, however, is to aid the person
providing and alternative implementation.  The SRFI usually includes a
reference implementation, but if a Scheme system maintainer wants to
provide an alternative implementation (perhaps leveraging off of
implementation-specific features or whatever), the TEST-IMPLEMENTATION
section provides an easy, standard way to get a warm fuzzy that the
implementation sort of works.  Otherwise, the implementor of an
alternative must extract by hand test cases from the specification (which,
granted, can be more effective since they can be white-box test cases and
drive out boundaries in the alternative implementation).

Also, some SRFIs will not be able to provide a reference implementation,
so the TEST-IMPLEMENTATION section serves as an alternate way to provide
an auxillary description when no reference implementation is possible. 

The test cases also provide some comfort that a Scheme implementation is
properly interpreting the reference implementation itself, though I don't
see that as a primary function.

-- Donovan Kolbly                    (  RScheme Development Group
                                     (  d.kolbly@rscheme.org
				     (  http://www.rscheme.org/~donovan/