Re: return values

[Alex Shinn <foof@xxxxxxxxxxxxx>]

At Mon, 07 Feb 2005 17:08:34 -0800, Per Bothner wrote:
> 
> Alex Shinn wrote:
> 
> > For test cases it can be very
> > useful to return a boolean value so that additional information can be
> > displayed when the test fails (or errors) unexpectedly.
> 
> But why should that be part of the test suite, rather than the
> test runner?  I.e. I would *expect* a test runner to display "additional
> information" when a test fails unexpectedly.
> 
> I can see when debugging either a test-suite or when trying to figure
> out why a test failed you might want to print extra information.
> But in that case you'd typically edit the test suite *temporarily*,
> perhaps simplifing it, perhaps writing the state of variables, etc.
> But that's debugging, not testing: you'd change it back when you've
> figured it out.

And then lose all the debugging steps you worked so hard to make?

Here is a simplified excerpt from a test suite I'm working on right
now to verify a complex data structure.  The procedure takes an
initial state for the data structure, a list of symbolic changes to
apply, and an expected result.  If the test fails, it performs an
instant replay of the test, and on each sub-step of the test it
displays what change it is about to apply and a dump of the resulting
data structure.  Although this part is a little complicated, I can
then add new tests as a simple list of symbols, and if they fail I get
to see instantly what went wrong where, and can often fix the problem
without even needing to manually debug.  [full source available if
you're interested]

(define (run-test init changes expected)
  (let ((ds (make-data-structure init)))
    (for-each
     (lambda args (apply-change ds args))
     changes)
    (let ((result (serialize ds)))
      (cond
        ((equal? result expected)
         (set! *pass* (+ 1 *pass*))
         (display "[ OK ] ") (write init) (display " => ") (write changes)
         (display " => ") (write result) (newline))
        (else
         (set! *fail* (+ 1 *fail*))
         (display "[FAIL] ") (write init) (display " => ") (write changes)
         (display " => ") (write result) (display " [expected ")
         (write expected) (display "]\n")
         (display "  \x1B[36m<<<<< BEGIN INSTANT REPLAY\x1B[0m\n")
         (let ((ds (make-data-structure init)))
           (display "  ") (write (serialize ds)) (newline)
           (for-each
            (lambda args
              (display "  => ") (write args) (newline)
              (apply-change ds args)
              (display "  ") (write (serialize ds)) (newline))
            changes))
         (display "  \x1B[36m>>>>> END INSTANT REPLAY\x1B[0m\n"))))))

This approach requires access to the data the test was run with, not
the name of the test or the source form, so it must be tightly coupled
with the actual call to test-assert.

> > The true values could be just #t or more descriptive symbols like 'pass, 'skip,
> > 'fail, 'error.
> 
> Woould it be more useful to be able to test the most recent result.
> E.g. rather than
>    (if (eq? 'fail (test-assert ...))) ...
> do:
>    (test-assert ...)
>    (if (eq? 'fail (test-result-last)) ...)
> or something like that.

Perhaps, but I would always check the result immediately after the
test call so this is just adding an extra step for my purposes.

Also, I should have been more clear, 'fail was intended to mean an
expected failure (as from test-expect-fail).  An unexpected failure
would return #f, so the code would look like

  (if (not (test-assert ...))
    ...)

-- 
Alex