[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Suggestion: include test suite in SRFI

On Tue, 10 Nov 1998, Kent M Pitman wrote:

> In any event, also, I observe the possibility of tests whose 
> function is not to "prove conformance" but to "demonstrate
> correct behavior on a known set of tests".  By such word games,
> you can relax the set of things the test seems to "prove" and
> so make it possible for a correct answer to mean "passed" and
> not "might have passed" since completing the entire test does
> not have the same weight.  That might be a better way to go.

That is closer to what I was thinking, anyway.  They should provide a
common basis for comfort, but should not be taken to prove anything.

> Also, I do think you should create an appeal procedure because
> a failure to conform may sometimes be due to lack of foresight
> on the part of the designers and (as with a disputed item on a
> credit card bill) might wish not to be counted as a black mark
> during a time while some sort of mediation was invoked.

I see your point, but I sure don't want to create some huge legal system
for this.  With the test case subservient to the specification, I could
forsee confusion in the marketplace between "conforming" and "passes the
tests", if the tests are testing MORE than what is specified.  Of course,
I can forsee just as well confusion between "conforming" and "uses the
reference implementation", since a reference implementations, like a test
case, will likely do more than what is specified.

> However, all that said, I think exactly for the reasons cited
> above that you are best staying out of the business of tightly
> coupling tests with specifications unless you make the tests
> definitional, and at that point you might as well just supply a 
> reference implementation instead.

Except that the reference implementation and tests are complementary, and
a reference implementation is not always possible.

I'm not sure what you mean by "definitional" here.  I'm not suggesting a
formal conformance suite, a la Ada.

> I don't like the idea of screening who can be in this srfi system and
> who can't, nor do i like the idea of saying who complies and who
> doesn't.

My wording made it sound too formal and harsh, I think.  I was hoping to
avoid some duplication of effort in generating test cases.  I was partly
presuming that (1) the author of the SRFI would, in general, be in a
better position to demonstrate the intended behavior by example, and (2) 
the SRFI review process would give people a chance to see the tests and
run them quickly on their own systems, providing an additional check on
the internal consistency of the proposal.

>   I think it should be simply be a place to register something
> for anyone who wants to do it, and that compliance should be judged by
> the market and/or by one OR MORE independent (not uniquely determined)
> agencies, since I may not trust any one set of tests or testers, and
> I'd rather let the marketplace sort out who are good testers than
> build that into the process.  There's no obvious reason to believe
> that just because someone can write a good spec, that they can also
> write a good test.  

True, and I'd forgotten that.  In my industrial experience, the testers
have been completely different groups with different skill sets from the
developers, and that is usually beneficial because the testers don't
subconsciously know how to avoid triggering bugs. 

> I'm worried about other good tests not being 
> given the same status, and I'm worried about bad tests attached to
> good specs getting undue weight.

Good point.  Maybe I should suggest a different, even more decoupled
approach.  According to Shriram's argument, perhaps both the reference
implementation and the test cases should be completely decoupled,
permitting multiple and evolving instances associated with the SRFI. 

That is, instead of imbedding the reference implementation and test cases
in the SRFI, the SRFI should just be the specification, with an extenable
sequence of appendices of either type "reference implemention" or "test
suite". The process could then require a SRFI to have an initial set of
appendices.  The extensibility process would have to be formalized, but at
least it associates the changes with the SRFI object itself rather than
requiring a superceded-by SRFI, when no defect in the specification itself
is found.

> (It's also funny to have to have me be saying that I'd like to see
> less mechanism in the process and to have you guys being the ones
> proposing what I see as excess red tape.  Turnabound is fair play, I
> guess.)

And there I was last PP piling on even more red tape!  (or maybe removing
some, depending on how think of taking the reference implementation OUT of
the core SRFI process)

-- Donovan Kolbly                    (  RScheme Development Group
                                     (  d.kolbly@rscheme.org
				     (  http://www.rscheme.org/~donovan/