SRFI 269: Portable Test DefinitionsSRFI 269: Portable Test Definitionsby Andrew Tropin, Ramin Honary
This SRFI is currently in draft status. Here is an explanation of each status that a SRFI can hold. To provide input on this SRFI, please send email to srfi-269@nospamsrfi.schemers.org. To subscribe to the list, follow these instructions. You can access previous messages via the mailing list archive.
This SRFI defines a portable API for test definitions that is
decoupled from test execution and reporting. It provides three
primitives: the universal is macro for
assertions, test for grouping assertions into
independently executable units, and suite for organizing
tests into hierarchies. Tests and suites can carry user-provided
metadata to adjust the behavior of a test runner, for example to
select tests by tags or to enforce timeout values. The API is tiny,
yet capable and flexible. By focusing on the definition and leaving
execution semantics to test runners, this SRFI offers a common ground
that can reduce fragmentation among testing libraries.
Unlike side-effect-driven testing frameworks (e.g. SRFI 64), this API produces first-class runtime entities, making it easy to filter, schedule, wrap them in exception guards and continuation barriers, run in arbitrary order, and re-run dynamically generated test subsets. In addition to the usual CLI test runners, it enables runtime-friendly test runners that integrate well with highly interactive development workflows inside REPLs and IDEs, significantly increasing control over test execution, and shortening the feedback loop.
To bridge the test definitions and test runners, the SRFI specifies a message-passing programming interface and test loading and execution semantics recommendations for test runner implementers.
Most of the Scheme libraries and applications benefit from tests,
and most of test suites benefit from portability.
SRFI 64
(2005) was a valuable first step toward a common testing API, and its
widespread adoption demonstrates the need. Yet the Scheme ecosystem
remains fragmented: implementations maintain their own incompatible
testing libraries
(RackUnit,
Chicken's test
egg,
Chibi's (chibi
test), numerous ad hoc solutions), each with its own
terminology and conventions. There is no shared vocabulary for what
"assertion," "test," and "test suite" mean, making it difficult to
write portable tests, and nearly impossible to build portable testing
tools.
SRFI 64's design couples three distinct concerns: defining tests,
executing them, and reporting results. Writing
a test-assert form is running it: the assertion
fires as a side effect at load time and that leads to multiple
consequences. Tests cannot be defined in one place and run in
another; the execution strategy cannot be changed without rewriting
test code; the test-begin/test-end model is
fragile and provides no first-class grouping. SRFI 64's own rationale
acknowledges this trade-off, noting that the API "may be a little less
elegant and 'compositional' than using explicit test objects."
Because SRFI 64 tests are imperative side effects rather than first-class values, they cannot be filtered by tag, reordered, re-run as a subset, or inspected programmatically. The test runner relies on global mutable state, making tests non-reentrant and difficult to compose. These limitations are especially painful in interactive workflows: at a REPL or inside an IDE. A programmer wants to pick one failing test, re-run it, examine the failure/use a debugger, fix the code, and iterate: a tight feedback loop that is impossible when tests are ephemeral side effects that vanish after execution.
This SRFI addresses these problems through three design decisions:
is, test,
suite. They construct first-class entities (runtime
objects) and deliver them to a pluggable test runner via a
message-passing protocol. Definition code is fully portable;
execution and reporting strategy varies by environment.is macro
captures both the unevaluated source form and a thunk that evaluates
it, allowing runners to inspect and execute assertions separately.
The result is an API that is equally at home in CI pipelines and in
live REPL sessions.The API surface is deliberately minimal: three definition forms, one parameter, one setter procedure, three predicates, and two deferred variants. Yet it covers assertions with rich diagnostics, named tests, nested suites, user-provided metadata, and deferred composable suite loaders. This SRFI is intended to supersede SRFI 64 for the purpose of test definition. By standardizing what a test is and leaving how it is run to test runners, this SRFI provides a stable foundation on which future standards for runners, reporters, and discovery mechanisms can be built.
The API specified by this SRFI is organized into two layers:
is, test,
and suite, a current-test-runner parameter,
the set-default-test-runner! procedure, a small set of
predicates, and deferred variants test-loader
and suite-loader.The sample implementation section provides informative guidance and materials for test runner implementers on loading, scheduling, and reporting.
The definition primitives do not execute tests themselves. Instead,
each form constructs a first-class entity, an association
list (alist), and delivers it to the current test runner via a
message. A test entity captures the test body as a
one-argument procedure, its description string, optional metadata, and
its source location when available. The test runner is a procedure
stored in the current-test-runner parameter; it receives
messages as alists and is free to execute tests immediately, collect
them for later, or take any other action.
This separation is the central design principle: code that defines tests is portable across all conforming implementations, while code that runs tests may vary to suit different environments: CI pipelines, interactive REPLs and IDE, specific testing/reporting tools.
is macro.
An assertion captures an expression (the body), a thunk
that evaluates it, and its source location when available. It also
has an optional description to clarify the intent of the assertion
when it is not clear from the asserting expression.test form. A test groups zero or more
assertions together under a human-readable description string.
Tests are the smallest unit that a test runner schedules and reports
on. A test body procedure is a one-argument procedure whose
argument is supplied by the runner when the test is executed.suite or suite-loader
form. Suites impose hierarchical structure and may carry metadata
that potentially influences the test running behavior for the entire
group.assertion/, test/,
suite/) indicate which kind of entity they belong
to.current-test-runner parameter. Every message
contains at least a type key whose value is a symbol
identifying the kind of message
(e.g. runner/run-assertion,
runner/load-test, runner/load-suite). The
remaining keys carry the entities and any additional context.current-test-runner parameter. The test runner
receives every message produced by the definition primitives and
decides how to handle it: for example, by executing an assertion
immediately, by collecting a test for deferred execution, or by
building a suite hierarchy. This SRFI specifies the messages a test
runner must accept; it does not prescribe execution order,
concurrency, or reporting strategy.((tags . (integration)))),
specifying a timeout, or adding fixtures.filename, line,
and column or any other object useful for the specific
Scheme implementation.current-test-runner parameter(current-test-runner)(set-default-test-runner! runner)parameterize installs a different runner for a dynamic
extent.current-test-runner is a
parameter object (as defined
by make-parameter) that holds the current test runner
procedure. All definition primitives—is,
test, and suite—deliver their messages
by calling (current-test-runner) to obtain the runner and
then applying it to a message.
If current-test-runner is never set, invoking any
definition primitive should produce a diagnostic indicating
that no runner has been configured. Libraries that provide a test
runner may call set-default-test-runner! upon
loading so that end users need not configure it manually.
Because current-test-runner is a parameter, it can be
rebound with parameterize to install a different runner
for a dynamic extent, which is useful for testing the test framework
itself or for running tests with alternative test runner.
(is expression)(is expression description)The is macro is the sole assertion primitive. Each
invocation constructs an assertion entity (an alist) and
delivers it to the current test runner by sending
a runner/run-assertion message. The test runner decides
how to execute the assertion and what to do with the result.
Programs may rename is when importing it, or wrapper
libraries may re-export it under a name such as check, using
the implementation's library renaming facilities.
An optional description expression may be supplied as the second argument. It must evaluate to a string. When supplied, the description is stored in the assertion entity so reporters can present the domain meaning of an assertion in addition to, or instead of, the captured source form.
The assertion entity contains the following keys.
The assertion/description and
assertion/location keys are optional:
| Key | Value |
|---|---|
assertion/body-thunk |
A thunk that, when called, evaluates the original expression and returns its value. |
assertion/body |
The source form of the expression as a datum (unevaluated), useful for reporting. |
assertion/description |
The human-readable description. This key is omitted if no description was provided. |
assertion/location |
An assertion source code location. This key may be omitted if source-location information is unavailable. |
The message sent to the test runner has the form:
`((type . runner/run-assertion)
(assertion . ,assertion-entity))The return value of is is determined by the test
runner. A conforming runner should return the value produced
by the body thunk when the assertion succeeds. This makes it possible
to reuse the value returned by is forms:
(is (= 7 (is (+ 3 4)))) ; inner is returns 7suite but outside of
a test, the test runner should signal an
error. Assertions are the province of tests; suites only group
tests.;; Atomic value — truthy means pass
(is #t)
(is 42)
;; Variable
(let ((x "hello"))
(is x))
;; Procedure applications
(is (= 4 (+ 2 2)))
(is (string=? "hello" (greet "world")))
(is (even? 14))
(is (lset= = '(1 2 3) '(3 2 1)))
;; Assertion description in a complete test
(test "permission inheritance" (ctx)
(define db (cdr (assq 'db ctx)))
(define project (create-project! db "srfi-269"))
(define user (project-creator project))
(define permissions (user-permissions db user project))
(is (member 'project/admin permissions)
"project creator receives administration permission for the project"))
(test description () body …)(test description (context) body …)(test description () 'metadata metadata-alist body …)(test description (context) 'metadata metadata-alist body …)The test macro defines a single, independently
executable unit of testing. It constructs a test entity (an
alist) that captures the test body as a one-argument procedure, its
description and metadata, and its source location when available. It
then immediately delivers the entity to the current test runner by
sending a runner/load-test message.
A test runner executes the body procedure by applying it to a context value:
((cdr (assq 'test/body-procedure test)) context)The context value is supplied by the runner and is an association list. A runner should construct it anew for every invocation of a test body procedure. Context producers, such as test fixtures or runner middleware, extend a context by constructing a new alist and passing it to the next producer or to the test. They should not modify the pairs that form an existing context.
Treating the context structure as immutable does not require its values to be immutable. A context may contain stateful resources such as database connections, sockets, ports, or boxes. Tests and context producers may perform the side effects supported by those values. The context alist itself remains unchanged; replacing or adding an association requires constructing and passing a new context.
A test should be self-contained: its body should not depend on side effects produced by surrounding expressions or by other tests. Side effects directed at values in the context are permitted, but a runner should arrange setup and cleanup so that each execution starts with the intended state. Because a test runner may execute tests in any order, at any point in time, or skip them entirely, relying on unrelated external state makes test results unpredictable.
A test entity is not single-use. A test runner may call test body
procedure multiple times, possibly with a different context value each
time. Every call evaluates the test body anew, so
each is form constructs and delivers a new assertion
entity, and any other side effects occur again.
description is a string that serves as a human-readable
label for the test. The context list is either empty or contains a
single context identifier. When present, the identifier is
used as the formal parameter of the body procedure. When the list is
empty, the runner-supplied context value is ignored and no context
identifier is bound in the test body. The body forms
typically contain zero or more is assertions, but may
contain arbitrary Scheme expressions (e.g. local definitions,
setup code).
The test entity contains the following keys.
The test/location key is optional:
| Key | Value |
|---|---|
test/body-procedure |
A procedure of one argument. When called with a context value, it evaluates the body forms in order. If a context identifier was provided, it is bound to that value. |
test/description |
The description string. |
test/metadata |
The metadata-alist, or '() if none
was provided. |
test/location |
A source location. This key may be omitted if source-location information is unavailable. |
The message sent to the test runner has the form:
`((type . runner/load-test)
(test . ,test-entity))test
inside the body of another test is an error; the test
runner should signal it. However, in some cases, it can be
useful to have a test or suite inside the test, it can be achieved,
if current-test-runner is parameterized (temporarily
redefined).suite. When inside a suite, the test runner
associates the test with the enclosing suite path.The return value of test is unspecified. Because
test is a loading form that registers a test
with the runner rather than executing it, no meaningful value is
produced. Code must not rely on the return value.
;; Minimal test
(test "addition works" ()
(is (= 4 (+ 2 2))))
;; Multiple assertions in one test
(test "string operations" ()
(is (string=? "HELLO" (string-upcase "hello")))
(is (= 5 (string-length "hello"))))
;; Test using context and metadata
(test "database round-trip" (ctx)
'metadata
'((tags . (integration))
(timeout . 30))
(define db (cdr (assq 'db ctx)))
(is (equal? sample-record (db-read db (db-write db sample-record)))))
;; Test with setup code
(test "list reversal" ()
(define xs '(1 2 3))
(is (equal? '(3 2 1) (reverse xs))))test-loader — deferred test definition(test-loader description context-list body …)(test-loader description context-list 'metadata metadata-alist body …)() or
(context), as for test.test-loader accepts the same arguments and constructs
the same test entity as test, but returns a loader instead of
loading it immediately. The loader accepts zero or one metadata alist
and sends a runner/load-test message when invoked.
Load-time metadata is sent separately under load/metadata.
`((type . runner/load-test)
(test . ,test-entity)
(load/metadata . ,metadata-alist))test can be implemented via test-loader:
(test arguments …)
≡
((test-loader arguments …))suite — test suite definition macro(suite description body …)(suite description 'metadata metadata-alist body …)The suite macro defines a grouping unit that organizes
tests and nested suites into a hierarchy. It constructs a suite
entity (an alist), then immediately delivers it to the current
test runner by sending a runner/load-suite message. The
test runner evaluates the suite body, during which any
enclosed test and suite forms are loaded and
possibly associated with the context of this suite.
description is a string-valued expression that serves as
a human-readable label for the suite. The body forms
typically contain test forms, and
nested suite forms. It should not contain any other
code, especially test setup code, as tests should be self-contained as
they can be executed in arbitrary orders and multiple times.
The suite entity contains the following keys.
The suite/location key is optional:
| Key | Value |
|---|---|
suite/body-thunk |
A thunk that, when called, evaluates
the body forms in order. During evaluation, enclosed
test and suite forms register
themselves with the current test runner under this suite’s
context. |
suite/description |
The description string. |
suite/metadata |
The metadata-alist, or '() if none
was provided. |
suite/location |
A source location. This key may be omitted if source-location information is unavailable. |
The message sent to the test runner has the form:
`((type . runner/load-suite)
(suite . ,suite-entity))test.
Invoking suite inside the body of a test
is an error; the test runner should signal it. The only
exception is if the test runner is overriden inside the test, which
means that current hierarchy is reset to the empty one and suite is
not "nested" inside the test anymore.is assertions inside a suite but outside of
any test are an error; the test runner should
signal it. Assertions belong inside tests.The return value of suite is unspecified. Because
suite is a loading form that registers a suite
with the runner rather than executing its tests, no meaningful value
is produced. Code must not rely on the return value.
;; Flat suite
(suite "arithmetic"
(test "addition" ()
(is (= 4 (+ 2 2))))
(test "multiplication" ()
(is (= 6 (* 2 3)))))
;; Nested suites
(suite "strings"
(suite "case conversion"
(test "upcase" ()
(is (string=? "HELLO" (string-upcase "hello"))))
(test "downcase" ()
(is (string=? "hello" (string-downcase "HELLO")))))
(suite "splitting"
(test "split on comma" ()
(is (equal? '("a" "b" "c")
(string-split "a,b,c" #\,))))))
;; Suite with metadata
(suite "integration tests"
'metadata
'((tags . (integration))
(slow? . #t))
(test "end-to-end round trip" ()
(is (equal? expected (round-trip input)))))suite-loader — deferred suite definition(suite-loader description body …)(suite-loader description 'metadata metadata-alist body …)(loader)(loader metadata-alist)suite-loader is the deferred counterpart
of suite. It constructs the same suite entity
but does not immediately send it to the test runner.
Instead, it returns a suite loader, a procedure that accepts zero or
one argument that, when invoked, sends
the runner/load-suite message. The optional argument
must be a metadata alist. This load-time metadata is sent separately
under load/metadata key. The test runner decides on a merging
strategy. The default option is just to prepend the suite's metadata with load
metadata. This, way alist lookup procedures that return the first
matching entry observe load-time values as taking precedence over
definition-time values. Invoking a suite loader must not modify its
definition-time metadata, so the same loader can be invoked repeatedly
with different metadata. The returned loader should carry the
information, or be registered in some registry to make
the suite-loader? predicate return #t on it.
This can be later used for discovery of test suites loaders (deferred
test suites).
When a suite loader is invoked with load-time metadata, the message sent to the test runner has the form:
`((type . runner/load-suite)
(suite . ,suite-entity)
(load/metadata . ,metadata-alist))suite can be implemented via suite-loader:
(suite desc body …)
≡
((suite-loader desc body …))Although while is, suite
and test forms can be used at the top level, test
modules should wrap all tests, suites and assertions in
a suite-loader (or define-suite) so that
test runners can discover and load them as a unit. A bare top-level
test and similiar forms are loaded as soon as the module
is evaluated, with no way for a runner to discover it independently or
defer its execution.
Suite loaders are the primary building block for composable, reusable test suites. Because they are first-class procedures, they can be stored in variables, passed as arguments, and invoked inside other suites to include their tests:
(define my-unit-tests
(suite-loader "unit tests"
(test "one" () (is #t))
(test "two" () (is (= 2 (+ 1 1))))))
(define my-integration-tests
(suite-loader "integration tests"
(test "round-trip" ()
(is (equal? x (decode (encode x)))))))
;; Compose into a top-level public suite. The same loader can be
;; called more than once with different metadata.
(define all-tests
(suite-loader "all tests"
(my-unit-tests '((configuration . default)))
(my-unit-tests '((configuration . strict)))
(my-integration-tests)))
(export all-tests)
define-suite — named suite definition(define-suite (name) body …)(define-suite (name) 'metadata metadata-alist body …)define-suite is a convenience form that combines
suite-loader with an ordinary definition. The suite name
is written in a parenthesized header, similar to a procedure
definition with define. It is equivalent to:
(define name
(suite-loader (symbol->string 'name) body …))Or, with metadata:
(define name
(suite-loader (symbol->string 'name) 'metadata metadata-alist body …))The suite description string is derived from name by converting the symbol to a string. This form is intended for top-level suite definitions in test modules.
(define-suite (arithmetic-tests)
(test "addition" ()
(is (= 4 (+ 2 2))))
(test "subtraction" ()
(is (= 0 (- 2 2)))))
;; Now (arithmetic-tests) can be called from another suite
;; or from the REPL to load and run the tests.(test? obj) → boolean#t if obj is a test entity:
an alist that contains at least the
keys test/body-procedure
and test/description.(suite? obj) → boolean#t if obj is a suite entity:
an alist that contains at least the
keys suite/body-thunk
and suite/description.(suite-loader? obj) → boolean#t if obj is a suite loader
(usually produced by suite-loader
or define-suite). Implementations can mark suite
loaders with a procedure property or somehow else; ordinary lambdas
do not satisfy this predicate.A compact sample implementation is provided in the (srfi
srfi-269) library:
define-library implementation using
syntax-rules, including a simple sequential test runner.The sample implementation uses
SRFI 229 tagged
procedures to identify suite loaders and records source locations
as #f. It is intentionally small and includes a basic
runner suitable for examples and experimentation, it doesn't set
context, merge load-time, inherit suite metadatas or any other
advanced functionality.
suitbl — a full-fledged testing library based
on SRFI-269suitbl is provided as part of
the guile-ares-rs
project. It has pretty advanced test runner, with customizable
schedulers, reporters and many other features. The definitions API is
SRFI-269 compatible and contained in a single module:
is, test, suite,
test-loader, suite-loader,
define-suite, current-test-runner,
set-default-test-runner!, and predicates).The implementation depends on syntax-case, and guile's
procedure properties.
The is macro accepts an arbitrary expression and
produces a body thunk that evaluates it. It also accepts an optional
description expression. When present, it is evaluated and stored
under assertion/description; otherwise, the key is omitted.
The unevaluated source form is captured as a datum.
Warning: Do not recognize the
'metadata marker by treating quote or
metadata as hygienic pattern literals. Lexical bindings
with either name at the macro use site can prevent the marker from
matching. Instead, compare the marker's source datum structurally with
(quote metadata). A syntax-case
implementation can compare (syntax->datum marker) with
that datum. A syntax-rules implementation can capture the
two marker components as pattern variables, compare their quoted datum
with an auxiliary predicate, and retain the candidate as the first body
form when it is not the marker.
The test macro parses a description expression
followed by an argument list. The list is either empty or contains one
context identifier. The generated
test/body-procedure always accepts one context value. If
an identifier is present, it is used as the formal parameter;
otherwise, the value is ignored. The runner can later call that
procedure with any context value it chooses. It can be a subject
for future specification of test runner.
The immediate suite form can be defined in terms of
suite-loader: suite expands
to ((suite-loader …)). The deferred form is
the primitive; the immediate form simply invokes it. This factoring
keeps the macro logic in one place and makes suite loaders the natural
unit of composition.
The suite-loader form returns a procedure that accepts
an optional metadata alist and, when called, sends
a runner/load-suite message to the current test runner.
The test runner constructs the effective suite metadata by appending
the definition-time metadata to the call-time metadata, without
modifying either alist. The returned procedure carries an identifying
marker that allows the suite-loader? predicate to
identify it (see portability notes below).
A small number of Guile-specific features are used in the
suitbl definitions implementation. Each has a
straightforward equivalent in other Scheme systems:
syntax-source, %search-load-path)suitbl implementation calls Guile's
syntax-source
at macro-expansion time to obtain a filename/line/column alist, and
resolves relative paths via %search-load-path. Other
implementations can use their own source-location API (e.g.
syntax-line/syntax-column in Racket, or
the source-info facility of the host system). When no
source-location API is available, the assertion/location,
test/location, and suite/location keys
may be omitted or set to #f.set-procedure-properties!,
procedure-property)suite-loader? can distinguish them from ordinary
lambdas. R7RS implementations can use
SRFI 229 or
SRFI 259, where
available, to implement equivalent procedure properties. Other
implementations can use alternative strategies: placing
suite-loader in a hash table that serves as a global registry, or
any other mechanism that allows a reliable predicate.Test suites for both implementations are provided:
is, test,
and suite, and the behavior
of define-suite. The tests intercept messages via a
logging test runner to inspect the entities without executing
them.The design of this SRFI was influenced
by clojure.test's use first-class test entities and
its is macro, and by JUnit's test primitives for building
hierarchies. SRFI 64 provided the foundation and demonstrated the
need for a portable and runtime-friendly testing API; this SRFI builds
on the lessons learned from its adoption.
Thanks to Andrew Kravchuk for showcasing Common Lisp testing ecosystem and sharing the experience.
Thanks to Daphne Preston-Kendal for providing valuable feedback and identifying several significant bugs and misimplementations.
Thanks to NLnet for funding the work
on the suitbl testing library and this specification.
© 2025, 2026 Andrew Tropin, Ramin Honary.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice (including the next paragraph) shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.