SRFI 269: Portable Test DefinitionsSRFI 269: Portable Test Definitionsby Andrew Tropin, Ramin Honary
This SRFI is currently in draft status. Here is an explanation of each status that a SRFI can hold. To provide input on this SRFI, please send email to srfi-269@nospamsrfi.schemers.org. To subscribe to the list, follow these instructions. You can access previous messages via the mailing list archive.
This SRFI defines a portable API for test definitions that is
decoupled from test execution and reporting. It provides three
primitives: the universal is macro for
assertions, test for grouping assertions into
independently executable units, and suite for organizing
tests into hierarchies. Tests and suites can carry user-provided
metadata to adjust the behavior of a test runner, for example to
select tests by tags or to enforce timeout values. The API is tiny,
yet capable and flexible. By focusing on the definition and leaving
execution semantics to test runners, this SRFI offers a common ground
that can reduce fragmentation among testing libraries.
Unlike side-effect-driven testing frameworks (e.g. SRFI 64), this API produces first-class runtime entities, making it easy to filter, schedule, wrap them in exception guards and continuation barriers, run in arbitrary order, and re-run dynamically generated test subsets. In addition to the usual CLI test runners, it enables runtime-friendly test runners that integrate well with highly interactive development workflows inside REPLs and IDEs, significantly increasing control over test execution, and shortening the feedback loop.
To bridge the test definitions and test runners, the SRFI specifies a message-passing programming interface and test loading and execution semantics recommendations for test runner implementers.
Most of the Scheme libraries and applications benefit from tests,
and most of test suites benefit from portability.
SRFI 64
(2005) was a valuable first step toward a common testing API, and its
widespread adoption demonstrates the need. Yet the Scheme ecosystem
remains fragmented: implementations maintain their own incompatible
testing libraries
(RackUnit,
Chicken's test
egg,
Chibi's (chibi
test), numerous ad hoc solutions), each with its own
terminology and conventions. There is no shared vocabulary for what
"assertion," "test," and "test suite" mean, making it difficult to
write portable tests, and nearly impossible to build portable testing
tools.
SRFI 64's design couples three distinct concerns: defining tests,
executing them, and reporting results. Writing
a test-assert form is running it: the assertion
fires as a side effect at load time and that leads to multiple
consequences. Tests cannot be defined in one place and run in
another; the execution strategy cannot be changed without rewriting
test code; the test-begin/test-end model is
fragile and provides no first-class grouping. SRFI 64's own rationale
acknowledges this trade-off, noting that the API "may be a little less
elegant and 'compositional' than using explicit test objects."
Because SRFI 64 tests are imperative side effects rather than first-class values, they cannot be filtered by tag, reordered, re-run as a subset, or inspected programmatically. The test runner relies on global mutable state, making tests non-reentrant and difficult to compose. These limitations are especially painful in interactive workflows: at a REPL or inside an IDE. A programmer wants to pick one failing test, re-run it, examine the failure/use a debugger, fix the code, and iterate: a tight feedback loop that is impossible when tests are ephemeral side effects that vanish after execution.
This SRFI addresses these problems through three design decisions:
is, test,
suite. They construct first-class entities (runtime
objects) and deliver them to a pluggable test runner via a
message-passing protocol. Definition code is fully portable;
execution and reporting strategy varies by environment.is macro captures
both the unevaluated source form and a separate argument thunk,
enabling rich failure diagnostics. The result is an API that is
equally at home in CI pipelines and in live REPL sessions.The API surface is deliberately minimal: three definition forms, one parameter, three predicates, and one deferred variant. Yet it covers assertions with rich diagnostics, named tests, nested suites, user-provided metadata, and deferred composable suite loaders. This SRFI is intended to supersede SRFI 64 for the purpose of test definition. By standardizing what a test is and leaving how it is run to test runners, this SRFI provides a stable foundation on which future standards for runners, reporters, and discovery mechanisms can be built.
The API specified by this SRFI is organized into two layers:
is, test,
and suite, and a test-runner* parameter, a
small set of predicates, and deferred
variant suite-loader.The sample implementation section provides informative guidance and materials for test runner implementers on loading, scheduling, and reporting.
The definition primitives do not execute tests themselves. Instead,
each form constructs a first-class entity, an association
list (alist), and delivers it to the current test runner via a
message. A test entity captures the test body as a
one-argument procedure, its source location, description string, and
optional metadata. The test runner is a procedure stored in
the test-runner* parameter; it receives messages as
alists and is free to execute tests immediately, collect them for
later, or take any other action.
This separation is the central design principle: code that defines tests is portable across all conforming implementations, while code that runs tests may vary to suit different environments: CI pipelines, interactive REPLs and IDE, specific testing/reporting tools.
is macro. An
assertion captures an expression (the body), a thunk that
evaluates it, and its source location. When the body is a predicate
application (pred arg …), the assertion also
captures a separate thunk that evaluates the arguments, enabling
richer failure messages.test form. A test groups zero or more assertions
together under a human-readable description string. Tests are the
smallest unit that a test runner schedules and reports on. A test
body procedure is a one-argument procedure whose argument is
supplied by the runner when the test is executed.suite or suite-loader form. Suites
impose hierarchical structure and may carry metadata that influences
the test runner's behavior for the entire group.assertion/, test/,
suite/) indicate which kind of entity they belong
to.test-runner* parameter. Every message contains at
least a type key whose value is a symbol identifying
the kind of message (e.g. runner/run-assertion,
runner/load-test, runner/load-suite). The
remaining keys carry the entities and any additional context.test-runner* parameter. The test runner receives
every message produced by the definition primitives and decides how
to handle it: for example, by executing an assertion immediately, by
collecting a test for deferred execution, or by building a suite
hierarchy. This SRFI specifies the messages a test
runner must accept; it does not prescribe execution order,
concurrency, or reporting strategy.((tags . (integration)))), marking
them as slow, or specifying a timeout.filename, line,
and column.test-runner* parameter(test-runner*)(test-runner* runner)test-runner* is a
parameter object (as defined
by make-parameter) that holds the current test runner
procedure. All definition primitives—is,
test, and suite—deliver their messages
by calling (test-runner*) to obtain the runner and then
applying it to a message.
If test-runner* is never set, invoking any definition
primitive should produce a diagnostic indicating that no
runner has been configured. Libraries that provide a test runner
should set test-runner* upon loading so that
end users need not configure it manually.
Because test-runner* is a parameter, it can be
rebound with parameterize to install a different runner
for a dynamic extent, which is useful for testing the test framework
itself or for running tests with alternative reporters.
(is expression)(is expression description)(is (predicate argument …))(is (predicate argument …) description)The is macro is the sole assertion primitive. Each
invocation constructs an assertion entity (an alist) and
delivers it to the current test runner by sending
a runner/run-assertion message. The test runner decides
how to execute the assertion and what to do with the result.
An optional description expression may be supplied as the second argument. It must evaluate to a string. The description is stored in the assertion entity so reporters can present the domain meaning of an assertion in addition to, or instead of, the captured source form.
The assertion entity always contains the following keys:
| Key | Value |
|---|---|
assertion/body-thunk |
A thunk that, when called, evaluates the original expression and returns its value. |
assertion/body |
The source form of the expression as a datum (unevaluated), useful for reporting. |
assertion/description |
The human-readable description string, or #f if
no description was provided. |
assertion/location |
An assertion source code location |
When the body has the predicate-application
shape (predicate argument …),
the entity additionally contains:
| Key | Value |
|---|---|
assertion/args-thunk |
A thunk that, when called, evaluates the arguments and returns the values as a list. This enables a test runner to display the actual argument values in a failure report, separately from the predicate. |
The message sent to the test runner has the form:
`((type . runner/run-assertion)
(assertion . ,assertion-entity))The return value of is is determined by the test
runner. A conforming runner should return the value produced
by the body thunk when the assertion succeeds. This makes it possible
to reuse the value returned by is forms:
(is (= 7 (is (+ 3 4)))) ; inner is returns 7suite but outside of
a test, the test runner should signal an
error. Assertions are the province of tests; suites only group
tests.;; Atomic value — truthy means pass
(is #t)
(is 42)
;; Variable
(let ((x "hello"))
(is x))
;; Predicate form — enables rich failure messages
(is (= 4 (+ 2 2)))
(is (string=? "hello" (greet "world")))
(is (even? 14))
(is (lset= = '(1 2 3) '(3 2 1)))
;; Assertion description in a complete test
(test ("permission inheritance" ctx)
(define db (assoc-ref ctx 'db))
(define project (create-project! db "srfi-269"))
(define user (project-creator project))
(define permissions (user-permissions db user project))
(is (member 'project/admin permissions)
"project creator receives administration permission for the project"))
(test (description context) body …)(test (description context) 'metadata metadata-alist body …)The test macro defines a single, independently
executable unit of testing. It constructs a test entity (an
alist) that captures the test body as a one-argument procedure, its
description, metadata, and source location, then immediately delivers
it to the current test runner by sending a runner/load-test
message.
A test runner executes the body procedure by applying it to a context value:
((assoc-ref test 'test/body-procedure) context)The context value is supplied by the runner. This SRFI does not prescribe its representation or contents; it may be an alist, record, object, or any other Scheme value. Runners can use it to pass fixture values, callbacks, runner services, metadata, or other data to tests.
A test should be self-contained: its body should not depend on side effects produced by surrounding expressions or by other tests. Because a test runner may execute tests in any order, at any point in time, or skip them entirely, relying on external state makes test results unpredictable.
description is a string that serves as a human-readable
label for the test. context is an identifier used as the
formal parameter of the body procedure. Tests that do not need the
context can use _ as the identifier. The
body forms typically contain zero or more is
assertions, but may contain arbitrary Scheme expressions
(e.g. local definitions, setup code).
The test entity contains the following keys:
| Key | Value |
|---|---|
test/body-procedure |
A procedure of one argument. When called with a context value, it evaluates the body forms in order with context bound to that value. |
test/description |
The description string. |
test/metadata |
The metadata-alist, or '() if none
was provided. |
test/location |
A source location alist. |
The message sent to the test runner has the form:
`((type . runner/load-test)
(test . ,test-entity))test
inside the body of another test is an error; the test
runner should signal it. However, in some cases, it can be
useful to have a test or suite inside the test, it can be achieved,
if the test-runner* is parameterized (temporary redefined).suite. When inside a suite, the test runner
associates the test with the enclosing suite path.The return value of test is unspecified. Because
test is a loading form that registers a test
with the runner rather than executing it, no meaningful value is
produced. Code must not rely on the return value.
;; Minimal test
(test ("addition works" _)
(is (= 4 (+ 2 2))))
;; Multiple assertions in one test
(test ("string operations" _)
(is (string=? "HELLO" (string-upcase "hello")))
(is (= 5 (string-length "hello"))))
;; Test using context and metadata
(test ("database round-trip" ctx)
'metadata
'((tags . (integration))
(timeout . 30))
(define db (assoc-ref ctx 'db))
(is (equal? sample-record (db-read db (db-write db sample-record)))))
;; Test with setup code
(test ("list reversal" _)
(define xs '(1 2 3))
(is (equal? '(3 2 1) (reverse xs))))suite — test suite definition macro(suite description body …)(suite description 'metadata metadata-alist body …)The suite macro defines a grouping unit that organizes
tests and nested suites into a hierarchy. It constructs a suite
entity (an alist), then immediately delivers it to the current
test runner by sending a runner/load-suite message. The
test runner evaluates the suite body, during which any
enclosed test and suite forms are loaded and
possibly associated with the context of this suite.
description is a string that serves as a human-readable
label for the suite. The body forms typically contain
test forms, and nested suite forms. It
should not contain any other code, especially test setup code, as
tests should be self-contained as they can be executed in arbitrary
orders and multiple times.
The suite entity contains the following keys:
| Key | Value |
|---|---|
suite/body-thunk |
A thunk that, when called, evaluates
the body forms in order. During evaluation, enclosed
test and suite forms register
themselves with the current test runner under this suite’s
context. |
suite/description |
The description string. |
suite/metadata |
The metadata-alist, or '() if none
was provided. |
suite/location |
A source location alist. |
The message sent to the test runner has the form:
`((type . runner/load-suite)
(suite . ,suite-entity))test.
Invoking suite inside the body of a test
is an error; the test runner should signal it.is assertions inside a suite but outside of
any test are an error; the test runner should
signal it. Assertions belong inside tests.The return value of suite is unspecified. Because
suite is a loading form that registers a suite
with the runner rather than executing its tests, no meaningful value
is produced. Code must not rely on the return value.
;; Flat suite
(suite "arithmetic"
(test ("addition" _)
(is (= 4 (+ 2 2))))
(test ("multiplication" _)
(is (= 6 (* 2 3)))))
;; Nested suites
(suite "strings"
(suite "case conversion"
(test ("upcase" _)
(is (string=? "HELLO" (string-upcase "hello"))))
(test ("downcase" _)
(is (string=? "hello" (string-downcase "HELLO")))))
(suite "splitting"
(test ("split on comma" _)
(is (equal? '("a" "b" "c")
(string-split "a,b,c" #\,))))))
;; Suite with metadata
(suite "integration tests" 'metadata '((tags . (integration))
(slow? . #t))
(test ("end-to-end round trip" _)
(is (equal? expected (round-trip input)))))suite-loader — deferred suite definition(suite-loader description body …)(suite-loader description 'metadata metadata-alist body …)suite-loader is the deferred counterpart
of suite. It constructs the same suite entity
but does not immediately send it to the test runner.
Instead, it returns a suite loader, a procedure that, when invoked,
sends the runner/load-suite message. The returned loader
should carry the information or be registered in some registry to make
the suite-loader? predicate return #t on
it.
The relationship between the two forms is:
(suite desc body …)
≡
((suite-loader desc body …))Although is and test forms can appear at
the top level, test modules should wrap all tests in
a suite-loader (or define-suite) so that
test runners can discover and load them as a unit. A bare top-level
test form is loaded as soon as the module is evaluated,
with no way for a runner to discover it independently or defer its
execution.
Suite loaders are the primary building block for composable, reusable test suites. Because they are first-class procedures, they can be stored in variables, passed as arguments, and invoked inside other suites to include their tests:
(define my-unit-tests
(suite-loader "unit tests"
(test ("one" _) (is #t))
(test ("two" _) (is (= 2 (+ 1 1))))))
(define my-integration-tests
(suite-loader "integration tests"
(test ("round-trip" _)
(is (equal? x (decode (encode x)))))))
;; Compose into a top-level public suite
(define all-tests
(suite-loader "all tests"
(my-unit-tests)
(my-integration-tests)))
(export all-tests)
define-suite — named suite definition(define-suite (name) body …)(define-suite (name) 'metadata metadata-alist body …)define-suite is a convenience form that combines
suite-loader with an ordinary definition. The suite name
is written in a parenthesized header, similar to a procedure
definition with define. It is equivalent to:
(define name
(suite-loader (symbol->string 'name) body …))Or, with metadata:
(define name
(suite-loader (symbol->string 'name) 'metadata metadata-alist body …))The suite description string is derived from name by converting the symbol to a string. This form is intended for top-level suite definitions in test modules.
(define-suite (arithmetic-tests)
(test ("addition" _)
(is (= 4 (+ 2 2))))
(test ("subtraction" _)
(is (= 0 (- 2 2)))))
;; Now (arithmetic-tests) can be called from another suite
;; or from the REPL to load and run the tests.(test? obj) → boolean#t if obj is a test entity:
an alist that contains at least the
keys test/body-procedure
and test/description.(suite? obj) → boolean#t if obj is a suite entity:
an alist that contains at least the
keys suite/body-thunk
and suite/description.(suite-loader? obj) → boolean#t if obj is a suite loader
produced by suite-loader or define-suite.
Implementations can mark suite loaders with a procedure property or
somehow else; ordinary lambdas do not satisfy this predicate.A sample implementation is a part of the suitbl
testing library and provided as part of
the guile-ares-rs
project. The definitions API is contained in a single module:
is, test, suite,
suite-loader, define-suite,
test-runner*, and predicates).The implementation depends on syntax-case,
make-parameter, and association lists. Entities are
represented as plain alists rather than records or opaque types, which
keeps the representation transparent, inspectable, and maximally
portable across Scheme implementations.
The is macro distinguishes two shapes: the
predicate-application shape
(pred arg …), which
produces both a body thunk and a separate arguments thunk, and an
arbitrary expression, which produces only a body thunk. Both shapes
accept an optional description expression. When present, it is
evaluated and stored under assertion/description;
otherwise, #f is stored. Both shapes capture the
unevaluated source form as a datum.
The test macro parses a leading
(description context) pair and
uses context as the formal parameter of
the test/body-procedure lambda. The runner can later
call that procedure with any context value it chooses.
The immediate suite form is defined in terms of
suite-loader: suite expands
to ((suite-loader …)). The deferred form is
the primitive; the immediate form simply invokes it. This factoring
keeps the macro logic in one place and makes suite loaders the natural
unit of composition.
The suite-loader form returns a procedure that, when
called, sends a runner/load-suite message to the current
test runner. The returned procedure carries metadata that allows
the suite-loader? predicate to identify it (see
portability notes below).
A small number of Guile-specific features are used in the sample implementation. Each has a straightforward equivalent in other Scheme systems:
syntax-source, %search-load-path)syntax-source
at macro-expansion time to obtain a filename/line/column alist, and
resolves relative paths via %search-load-path. Other
implementations can use their own source-location API (e.g.
syntax-line/syntax-column in Racket, or
the source-info facility of the host system). When no
source-location API is available, the assertion/location,
test/location, and suite/location keys
may be set to #f.set-procedure-properties!,
procedure-property)suite-loader? can distinguish them from ordinary
lambdas. R7RS implementations can use
SRFI 229 or
SRFI 259, where
available, to implement equivalent procedure properties. Other
implementations can use alternative strategies: placing
suite-loader in a hash table that serves as a global registry, or
any other mechanism that allows a reliable predicate.A test suite for the definitions API is provided:
is, test,
and suite, and the behavior
of define-suite. The tests intercept messages via a
logging test runner to inspect the entities without executing
them.The design of this SRFI was influenced
by clojure.test's use first-class test entities and
its is macro, and by JUnit's test primitives. SRFI 64
provided the foundation and demonstrated the need for a portable and
runtime-friendly testing API; this SRFI builds on the lessons learned
from its adoption.
suitbl
testing library and this specification.
© 2025, 2026 Andrew Tropin, Ramin Honary.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice (including the next paragraph) shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.