SRFI 269: Portable Test DefinitionsSRFI 269: Portable Test Definitionsby Andrew Tropin, Ramin Honary
This SRFI is currently in draft status. Here is an explanation of each status that a SRFI can hold. To provide input on this SRFI, please send email to srfi-269@nospamsrfi.schemers.org. To subscribe to the list, follow these instructions. You can access previous messages via the mailing list archive.
This SRFI defines a portable API for test definitions that is
decoupled from test execution and reporting. It provides three
primitives: the universal is macro for
assertions, test for grouping assertions into
independently executable units, and suite for organizing
tests into hierarchies. Tests and suites can carry user-provided
metadata to adjust the behavior of a test runner, for example to
select tests by tags or to enforce timeout values. The API is tiny,
yet capable and flexible. By focusing on the definition and leaving
execution semantics to test runners, this SRFI offers a common ground
that can reduce fragmentation among testing libraries.
Unlike side-effect-driven testing frameworks (e.g. SRFI 64), this API produces first-class runtime entities, making it easy to filter, schedule, wrap them in exception guards and continuation barriers, run in arbitrary order, and re-run dynamically generated test subsets. In addition to the usual CLI test runners, it enables runtime-friendly test runners that integrate well with highly interactive development workflows inside REPLs and IDEs, significantly increasing control over test execution, and shortening the feedback loop.
To bridge the test definitions and test runners, the SRFI specifies a message-passing programming interface and test loading and execution semantics recommendations for test runner implementers.
Most of the Scheme libraries and applications benefit from tests,
and most of test suites benefit from portability.
SRFI 64
(2005) was a valuable first step toward a common testing API, and its
widespread adoption demonstrates the need. Yet the Scheme ecosystem
remains fragmented: implementations maintain their own incompatible
testing libraries
(RackUnit,
Chicken's test
egg,
Chibi's (chibi
test), numerous ad hoc solutions), each with its own
terminology and conventions. There is no shared vocabulary for what
"assertion," "test," and "test suite" mean, making it difficult to
write portable tests, and nearly impossible to build portable testing
tools.
SRFI 64's design couples three distinct concerns: defining tests,
executing them, and reporting results. Writing
a test-assert form is running it: the assertion
fires as a side effect at load time and that leads to multiple
consequences. Tests cannot be defined in one place and run in
another; the execution strategy cannot be changed without rewriting
test code; the test-begin/test-end model is
fragile and provides no first-class grouping. SRFI 64's own rationale
acknowledges this trade-off, noting that the API "may be a little less
elegant and 'compositional' than using explicit test objects."
Because SRFI 64 tests are imperative side effects rather than first-class values, they cannot be filtered by tag, reordered, re-run as a subset, or inspected programmatically. The test runner relies on global mutable state, making tests non-reentrant and difficult to compose. These limitations are especially painful in interactive workflows: at a REPL or inside an IDE. A programmer wants to pick one failing test, re-run it, examine the failure/use a debugger, fix the code, and iterate: a tight feedback loop that is impossible when tests are ephemeral side effects that vanish after execution.
This SRFI addresses these problems through three design decisions:
is, test,
suite. They construct first-class entities (runtime
objects) and deliver them to a pluggable test runner via a
message-passing protocol. Definition code is fully portable;
execution and reporting strategy varies by environment.is macro
captures both the unevaluated source form and a thunk that evaluates
it, allowing runners to inspect and execute assertions separately.
The result is an API that is equally at home in CI pipelines and in
live REPL sessions.The API surface is deliberately minimal: three definition forms, one parameter, one setter procedure, three predicates, and one deferred variant. Yet it covers assertions with rich diagnostics, named tests, nested suites, user-provided metadata, and deferred composable suite loaders. This SRFI is intended to supersede SRFI 64 for the purpose of test definition. By standardizing what a test is and leaving how it is run to test runners, this SRFI provides a stable foundation on which future standards for runners, reporters, and discovery mechanisms can be built.
The API specified by this SRFI is organized into two layers:
is, test,
and suite, a current-test-runner parameter,
the set-current-test-runner! procedure, a small set of
predicates, and deferred variant suite-loader.The sample implementation section provides informative guidance and materials for test runner implementers on loading, scheduling, and reporting.
The definition primitives do not execute tests themselves. Instead,
each form constructs a first-class entity, an association
list (alist), and delivers it to the current test runner via a
message. A test entity captures the test body as a
one-argument procedure, its source location, description string, and
optional metadata. The test runner is a procedure stored in
the current-test-runner parameter; it receives messages as
alists and is free to execute tests immediately, collect them for
later, or take any other action.
This separation is the central design principle: code that defines tests is portable across all conforming implementations, while code that runs tests may vary to suit different environments: CI pipelines, interactive REPLs and IDE, specific testing/reporting tools.
is macro. An
assertion captures an expression (the body), a thunk that
evaluates it, and its source location.test form. A test groups zero or more assertions
together under a human-readable description string. Tests are the
smallest unit that a test runner schedules and reports on. A test
body procedure is a one-argument procedure whose argument is
supplied by the runner when the test is executed.suite or suite-loader form. Suites
impose hierarchical structure and may carry metadata that influences
the test runner's behavior for the entire group.assertion/, test/,
suite/) indicate which kind of entity they belong
to.current-test-runner parameter. Every message
contains at least a type key whose value is a symbol
identifying the kind of message
(e.g. runner/run-assertion,
runner/load-test, runner/load-suite). The
remaining keys carry the entities and any additional context.current-test-runner parameter. The test runner
receives every message produced by the definition primitives and
decides how to handle it: for example, by executing an assertion
immediately, by
collecting a test for deferred execution, or by building a suite
hierarchy. This SRFI specifies the messages a test
runner must accept; it does not prescribe execution order,
concurrency, or reporting strategy.((tags . (integration)))), marking
them as slow, or specifying a timeout.filename, line,
and column.current-test-runner parameter(current-test-runner)(set-current-test-runner! runner)parameterize installs a different runner for a dynamic
extent.current-test-runner is a
parameter object (as defined
by make-parameter) that holds the current test runner
procedure. All definition primitives—is,
test, and suite—deliver their messages
by calling (current-test-runner) to obtain the runner and
then applying it to a message.
If current-test-runner is never set, invoking any
definition primitive should produce a diagnostic indicating
that no runner has been configured. Libraries that provide a test
runner should call set-current-test-runner! upon
loading so that end users need not configure it manually.
Because current-test-runner is a parameter, it can be
rebound with parameterize to install a different runner
for a dynamic extent, which is useful for testing the test framework
itself or for running tests with alternative reporters.
(is expression)(is expression description)The is macro is the sole assertion primitive. Each
invocation constructs an assertion entity (an alist) and
delivers it to the current test runner by sending
a runner/run-assertion message. The test runner decides
how to execute the assertion and what to do with the result.
Programs may rename is when importing it, or wrapper
libraries may re-export it under a name such as check, using
the implementation's library renaming facilities.
An optional description expression may be supplied as the second argument. It must evaluate to a string. The description is stored in the assertion entity so reporters can present the domain meaning of an assertion in addition to, or instead of, the captured source form.
The assertion entity always contains the following keys:
| Key | Value |
|---|---|
assertion/body-thunk |
A thunk that, when called, evaluates the original expression and returns its value. |
assertion/body |
The source form of the expression as a datum (unevaluated), useful for reporting. |
assertion/description |
The human-readable description string, or #f if
no description was provided. |
assertion/location |
An assertion source code location |
The message sent to the test runner has the form:
`((type . runner/run-assertion)
(assertion . ,assertion-entity))The return value of is is determined by the test
runner. A conforming runner should return the value produced
by the body thunk when the assertion succeeds. This makes it possible
to reuse the value returned by is forms:
(is (= 7 (is (+ 3 4)))) ; inner is returns 7suite but outside of
a test, the test runner should signal an
error. Assertions are the province of tests; suites only group
tests.;; Atomic value — truthy means pass
(is #t)
(is 42)
;; Variable
(let ((x "hello"))
(is x))
;; Procedure applications
(is (= 4 (+ 2 2)))
(is (string=? "hello" (greet "world")))
(is (even? 14))
(is (lset= = '(1 2 3) '(3 2 1)))
;; Assertion description in a complete test
(test "permission inheritance" (ctx)
(define db (cdr (assq 'db ctx)))
(define project (create-project! db "srfi-269"))
(define user (project-creator project))
(define permissions (user-permissions db user project))
(is (member 'project/admin permissions)
"project creator receives administration permission for the project"))
(test description () body …)(test description (context) body …)(test description () 'metadata metadata-alist body …)(test description (context) 'metadata metadata-alist body …)The test macro defines a single, independently
executable unit of testing. It constructs a test entity (an
alist) that captures the test body as a one-argument procedure, its
description, metadata, and source location, then immediately delivers
it to the current test runner by sending a runner/load-test
message.
A test runner executes the body procedure by applying it to a context value:
((cdr (assq 'test/body-procedure test)) context)The context value is supplied by the runner and is an association list. A runner should construct it anew for every invocation of a test body procedure. Context producers, such as fixtures or runner middleware, extend a context by constructing a new alist and passing it to the next producer or to the test. They should not modify the pairs that form an existing context. Prepending a new association lets a producer add a key or shadow an earlier association without modifying the original context.
Treating the context structure as immutable does not require its values to be immutable. A context may contain stateful resources such as database connections, sockets, ports, or boxes. Tests and context producers may perform the side effects supported by those values. The context alist itself remains unchanged; replacing or adding an association requires constructing and passing a new context.
A test should be self-contained: its body should not depend on side effects produced by surrounding expressions or by other tests. Side effects directed at values in the context are permitted, but a runner should arrange setup and cleanup so that each execution starts with the intended state. Because a test runner may execute tests in any order, at any point in time, or skip them entirely, relying on unrelated external state makes test results unpredictable.
A test entity is not single-use. A test runner may call its body
procedure multiple times, possibly with a different context value each
time. Every call evaluates the test body anew, so each is
form constructs and delivers a new assertion entity, and any other
side effects occur again.
description is a string that serves as a human-readable
label for the test. The context list is either empty or contains a
single context identifier. When present, the identifier is
used as the formal parameter of the body procedure. When the list is
empty, the runner-supplied context value is ignored and no context
identifier is bound in the test body. The body forms
typically contain zero or more is assertions, but may
contain arbitrary Scheme expressions (e.g. local definitions,
setup code).
The test entity contains the following keys:
| Key | Value |
|---|---|
test/body-procedure |
A procedure of one argument. When called with a context value, it evaluates the body forms in order. If a context identifier was provided, it is bound to that value. |
test/description |
The description string. |
test/metadata |
The metadata-alist, or '() if none
was provided. |
test/location |
A source location alist. |
The message sent to the test runner has the form:
`((type . runner/load-test)
(test . ,test-entity))test
inside the body of another test is an error; the test
runner should signal it. However, in some cases, it can be
useful to have a test or suite inside the test, it can be achieved,
if current-test-runner is parameterized (temporarily
redefined).suite. When inside a suite, the test runner
associates the test with the enclosing suite path.The return value of test is unspecified. Because
test is a loading form that registers a test
with the runner rather than executing it, no meaningful value is
produced. Code must not rely on the return value.
;; Minimal test
(test "addition works" ()
(is (= 4 (+ 2 2))))
;; Multiple assertions in one test
(test "string operations" ()
(is (string=? "HELLO" (string-upcase "hello")))
(is (= 5 (string-length "hello"))))
;; Test using context and metadata
(test "database round-trip" (ctx)
'metadata
'((tags . (integration))
(timeout . 30))
(define db (cdr (assq 'db ctx)))
(is (equal? sample-record (db-read db (db-write db sample-record)))))
;; Test with setup code
(test "list reversal" ()
(define xs '(1 2 3))
(is (equal? '(3 2 1) (reverse xs))))suite — test suite definition macro(suite description body …)(suite description 'metadata metadata-alist body …)The suite macro defines a grouping unit that organizes
tests and nested suites into a hierarchy. It constructs a suite
entity (an alist), then immediately delivers it to the current
test runner by sending a runner/load-suite message. The
test runner evaluates the suite body, during which any
enclosed test and suite forms are loaded and
possibly associated with the context of this suite.
description is a string that serves as a human-readable
label for the suite. The body forms typically contain
test forms, and nested suite forms. It
should not contain any other code, especially test setup code, as
tests should be self-contained as they can be executed in arbitrary
orders and multiple times.
The suite entity contains the following keys:
| Key | Value |
|---|---|
suite/body-thunk |
A thunk that, when called, evaluates
the body forms in order. During evaluation, enclosed
test and suite forms register
themselves with the current test runner under this suite’s
context. |
suite/description |
The description string. |
suite/metadata |
The metadata-alist, or '() if none
was provided. |
suite/location |
A source location alist. |
The message sent to the test runner has the form:
`((type . runner/load-suite)
(suite . ,suite-entity))test.
Invoking suite inside the body of a test
is an error; the test runner should signal it.is assertions inside a suite but outside of
any test are an error; the test runner should
signal it. Assertions belong inside tests.The return value of suite is unspecified. Because
suite is a loading form that registers a suite
with the runner rather than executing its tests, no meaningful value
is produced. Code must not rely on the return value.
;; Flat suite
(suite "arithmetic"
(test "addition" ()
(is (= 4 (+ 2 2))))
(test "multiplication" ()
(is (= 6 (* 2 3)))))
;; Nested suites
(suite "strings"
(suite "case conversion"
(test "upcase" ()
(is (string=? "HELLO" (string-upcase "hello"))))
(test "downcase" ()
(is (string=? "hello" (string-downcase "HELLO")))))
(suite "splitting"
(test "split on comma" ()
(is (equal? '("a" "b" "c")
(string-split "a,b,c" #\,))))))
;; Suite with metadata
(suite "integration tests" 'metadata '((tags . (integration))
(slow? . #t))
(test "end-to-end round trip" ()
(is (equal? expected (round-trip input)))))suite-loader — deferred suite definition(suite-loader description body …)(suite-loader description 'metadata metadata-alist body …)(loader)(loader metadata-alist)suite-loader is the deferred counterpart
of suite. It constructs the same suite entity
but does not immediately send it to the test runner.
Instead, it returns a suite loader, a procedure that accepts zero or
one argument and, when invoked, sends
the runner/load-suite message. The optional argument must
be a metadata alist. Its entries are placed before the metadata
provided when the suite loader was defined. Consequently, alist
lookup procedures that return the first matching entry observe
call-time values as taking precedence over definition-time values.
Invoking a suite loader must not modify its definition-time metadata,
so the same loader can be invoked repeatedly with different metadata.
The returned loader should carry the information or be registered in
some registry to make the suite-loader? predicate
return #t on it.
The relationship between the two forms is:
(suite desc body …)
≡
((suite-loader desc body …))Although is and test forms can appear at
the top level, test modules should wrap all tests in
a suite-loader (or define-suite) so that
test runners can discover and load them as a unit. A bare top-level
test form is loaded as soon as the module is evaluated,
with no way for a runner to discover it independently or defer its
execution.
Suite loaders are the primary building block for composable, reusable test suites. Because they are first-class procedures, they can be stored in variables, passed as arguments, and invoked inside other suites to include their tests:
(define my-unit-tests
(suite-loader "unit tests"
(test "one" () (is #t))
(test "two" () (is (= 2 (+ 1 1))))))
(define my-integration-tests
(suite-loader "integration tests"
(test "round-trip" ()
(is (equal? x (decode (encode x)))))))
;; Compose into a top-level public suite. The same loader can be
;; called more than once with different metadata.
(define all-tests
(suite-loader "all tests"
(my-unit-tests '((configuration . default)))
(my-unit-tests '((configuration . strict)))
(my-integration-tests)))
(export all-tests)
define-suite — named suite definition(define-suite (name) body …)(define-suite (name) 'metadata metadata-alist body …)define-suite is a convenience form that combines
suite-loader with an ordinary definition. The suite name
is written in a parenthesized header, similar to a procedure
definition with define. It is equivalent to:
(define name
(suite-loader (symbol->string 'name) body …))Or, with metadata:
(define name
(suite-loader (symbol->string 'name) 'metadata metadata-alist body …))The suite description string is derived from name by converting the symbol to a string. This form is intended for top-level suite definitions in test modules.
(define-suite (arithmetic-tests)
(test "addition" ()
(is (= 4 (+ 2 2))))
(test "subtraction" ()
(is (= 0 (- 2 2)))))
;; Now (arithmetic-tests) can be called from another suite
;; or from the REPL to load and run the tests.(test? obj) → boolean#t if obj is a test entity:
an alist that contains at least the
keys test/body-procedure
and test/description.(suite? obj) → boolean#t if obj is a suite entity:
an alist that contains at least the
keys suite/body-thunk
and suite/description.(suite-loader? obj) → boolean#t if obj is a suite loader
produced by suite-loader or define-suite.
Implementations can mark suite loaders with a procedure property or
somehow else; ordinary lambdas do not satisfy this predicate.A sample implementation is a part of the suitbl
testing library and provided as part of
the guile-ares-rs
project. The definitions API is contained in a single module:
is, test, suite,
suite-loader, define-suite,
current-test-runner,
set-current-test-runner!, and predicates).The implementation depends on syntax-case,
make-parameter, and association lists. Entities are
represented as plain alists rather than records or opaque types, which
keeps the representation transparent, inspectable, and maximally
portable across Scheme implementations.
The is macro accepts an arbitrary expression and
produces a body thunk that evaluates it. It also accepts an optional
description expression. When present, it is evaluated and stored
under assertion/description; otherwise, #f is
stored. The unevaluated source form is captured as a datum.
The test macro parses a description expression
followed by a argument list. The list is either empty or contains one
context identifier. The generated
test/body-procedure always accepts one context value. If
an identifier is present, it is used as the formal parameter;
otherwise, the value is ignored. The runner can later call that
procedure with any context value it chooses.
The immediate suite form is defined in terms of
suite-loader: suite expands
to ((suite-loader …)). The deferred form is
the primitive; the immediate form simply invokes it. This factoring
keeps the macro logic in one place and makes suite loaders the natural
unit of composition.
The suite-loader form returns a procedure that accepts
an optional metadata alist and, when called, sends
a runner/load-suite message to the current test runner.
The implementation constructs the effective suite metadata by
appending the definition-time metadata to the call-time metadata,
without modifying either alist. The returned procedure carries an
identifying marker that allows the suite-loader? predicate
to identify it (see portability notes below).
A small number of Guile-specific features are used in the sample implementation. Each has a straightforward equivalent in other Scheme systems:
syntax-source, %search-load-path)syntax-source
at macro-expansion time to obtain a filename/line/column alist, and
resolves relative paths via %search-load-path. Other
implementations can use their own source-location API (e.g.
syntax-line/syntax-column in Racket, or
the source-info facility of the host system). When no
source-location API is available, the assertion/location,
test/location, and suite/location keys
may be set to #f.set-procedure-properties!,
procedure-property)suite-loader? can distinguish them from ordinary
lambdas. R7RS implementations can use
SRFI 229 or
SRFI 259, where
available, to implement equivalent procedure properties. Other
implementations can use alternative strategies: placing
suite-loader in a hash table that serves as a global registry, or
any other mechanism that allows a reliable predicate.A test suite for the definitions API is provided:
is, test,
and suite, and the behavior
of define-suite. The tests intercept messages via a
logging test runner to inspect the entities without executing
them.The design of this SRFI was influenced
by clojure.test's use first-class test entities and
its is macro, and by JUnit's test primitives. SRFI 64
provided the foundation and demonstrated the need for a portable and
runtime-friendly testing API; this SRFI builds on the lessons learned
from its adoption.
suitbl
testing library and this specification.
© 2025, 2026 Andrew Tropin, Ramin Honary.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice (including the next paragraph) shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.