177: Portable Keyword Arguments

by Lassi Kortela

Status

This SRFI is currently in withdrawn status. Here is an explanation of each status that a SRFI can hold. To provide input on this SRFI, please send email to srfi-177@nospamsrfi.schemers.org. To subscribe to the list, follow these instructions. You can access previous messages via the mailing list archive.

Abstract

Many Scheme implementations have keyword arguments, but they have not been widely standardized. This SRFI defines the macros lambda/kw and call/kw. They can be used identically in every major implementation currently in use, making it safe to use keyword arguments in portable code. The macros expand to native keyword arguments in Schemes that have them, letting programmers mix portable code and implementation-specific code.

Withdrawal notice

After long and careful exploration of keyword argument systems on this SRFI's mailing list, the SRFI itself is being withdrawn.

While the specification presented in this SRFI accomplishes its goal of portable keyword arguments, and could be used as-is, we were ultimately not satisfied with how it meshes with the rest of Scheme and with Lisp tradition. The keyword call syntax presented here is very unorthodox, wrapping keyword arguments in a sublist at the end of each procedure call. This is unlike every other Lisp dialect out there, all of which splice keyword arguments into the same argument list as ordinary positional arguments.

We also could not find a reasonable way to get rid of the call/kw prefix required in portable code at the start of each keyword procedure call. Several Schemers on the mailing list perceived this prefix as too heavy a requirement for all keyword calls. This would also most likely preclude SRFI 177 from being included in the Large Edition of the R7RS language. Prefixes shorter than call/kw were tried, but were found to be a bit cryptic while not solving the fundamental problem.

Table of contents

Rationale

Why keywords?

Keyword arguments are a very useful tool for managing complexity as programs grow. They are a natural solution to the "no, wait, this procedure still needs another argument" problem which is almost guaranteed to pop up many times over the lifetime of any non-trivial program. Humans simply cannot plan years ahead at this level of detail, and adding keyword arguments as an afterthought is less objectionable than accumulating long lists of optional positional arguments or refactoring central APIs every few years when third-party code depends on them.

While the same information can be passed using records, keyword arguments are more convenient for simple jobs since a separate record type does not have to be defined for each procedure taking them. They are also less verbose at the call site since an extra make-foo-record call is not needed.

Standardization

Scheme implementations with native keywords

These syntaxes work with the default settings of each Scheme at the time of writing. A couple of them optionally support an alternative read syntax for keywords.

Implementation Defining keyword arguments Supplying them
Chicken (lambda (foo #!key bar) ...) bar:
Kawa (lambda (foo #!key bar) ...) bar:
Gambit (lambda (foo #!key bar) ...) bar: (and bar:)
Bigloo (lambda (foo #!key bar) ...) :bar (and bar:)
Gauche (lambda (foo :key (bar #f)) ...) :bar
Sagittarius (lambda (foo :key (bar #f)) ...) :bar
STklos (lambda (foo :key (bar #f)) ...) :bar
S7 (lambda* (foo (bar #f)) ...) :bar
Guile (lambda* (foo #:key bar) ...) #:bar
Racket (lambda (foo #:bar (bar #f)) ...) #:bar

The let-keywords emulation library

Chibi-Scheme comes with a let-keywords macro in the (chibi optional) library. Chibi does not have special read syntax for keywords: quoted symbols are used to the same effect. Chibi does not offer a keyword-aware version of lambda or define. Instead, an ordinary procedure is defined with a rest argument to stand in for the keywords part. The rest argument is then destructured using let-keywords within the procedure body, effectively unpacking it like a traditional Lisp property list (plist).

A portable approach

Procedure calls

From the above table of native keywords it is clear that there is no portable syntax to supply keyword arguments in a procedure call. While all of the implementations support the syntax (proc arg1 arg2 #:key3 arg3 #:key4 arg4) with the keys in any order, the keyword prefix or suffix #: varies. On the other hand the implementations all have a low-level macro system, as well as procedures to turn symbols into keywords. Hence the easiest portable approach is to provide a macro that lets the caller use ordinary symbols instead of keywords for the read syntax. The symbols are then converted into the implementation’s native keyword syntax by the macro. The call/kw procedure in this SRFI implements this pattern.

Procedure definitions

The next problem is how to define procedures that take keyword arguments. Portable code cannot use any syntax that requires keyword objects in the lambda list, since as noted above there is no portable representation. Here too we need to use symbols in place of keywords. For most of the implementations with native keywords, we need to insert the constant #!key or #:key into the lambda list. This turns out to be trivially easy in all of them. For a few implementations, we need to turn each keyword symbol bar into (bar #f) to ensure that the default value is #f instead of undefined. Finally, Racket requires an exotic syntax #:bar (bar #f) where the keyword’s name appears twice (once as a keyword and again as a symbol). This is easy to code as well.

Keywords as objects vs syntactic markers

Most Scheme implementations with keywords follow Common Lisp's approach and treat them as symbol-like objects. Kawa and Racket take another approach, and treat (unquoted) keywords as syntactic markers signifying a keyword argument inside a procedure call. In these dialects of Scheme, it is a syntax error to have an unquoted keyword in source code apart from valid locations inside a procedure call. If we write our macros to expand into an ordinary procedure call in these Schemes, we can use keywords as markers without a problem.

Emulating keywords in Schemes that don’t have them

For Schemes that don’t have keyword arguments, the best approach is the one employed by Chibi's let-keywords: use a rest argument as in (lambda (arg1 arg2 . kvs) ...​). This argument takes a property list of keywords and their values. Once again the keywords can be represented by standard Scheme symbols.

Another approach would be to pass the keywords first: (lambda (kvs arg1 arg2) ...​). This is not ideal since it’s really nice to be able to call a keyword procedure just like an ordinary procedure when you don’t supply any keyword arguments (i.e. they get default values). Using a rest argument preserves this property; using one or more preceding arguments does not.

Using the rest argument for keywords implies that users cannot have their own rest argument or optional positional arguments. That's fine; the interplay of keywords with optional and rest arguments is somewhat confusing to most people anyway. And our keyword arguments are all optional, so a keyword argument can be used for any purpose an optional or rest argument might.

The keyword arguments could be passed in a vector. We cannot do this because it’s not portable to use a vector as the rest argument to a procedure; it has to be a list.

R5RS has the venerable syntax-rules pattern-matching hygienic macro system. It turns out to be powerful enough to implement keyword arguments without too much effort or duct tape. Since syntax-rules pattern matching is based mainly on list structure and we don’t have any portable keyword syntax, we have to rely solely on list structure to separate positional arguments from keyword arguments. With the ubiquitous tail patterns from SRFI 46 (Basic syntax-rules extensions) we can check whether the last element of a list is another list. Consequently we can achieve the syntax (lambda/kw (pos1 pos2 (key3 key4)) ...​) and (call/kw pos1 pos2 (key3 key4)). This is reasonably simple and unambiguous.

Specification

Summary

This SRFI provides lambda/kw and call/kw which are used as follows:

(define foo
  (lambda/kw (a b (c d e))
    (list a b c d e)))

(foo 1 2)                        ; => (1 2 #f #f #f)
(apply foo 1 2 '())              ; => (1 2 #f #f #f)
(call/kw foo 1 2 ())             ; => (1 2 #f #f #f)
(call/kw foo 1 2 (d 4))          ; => (1 2 #f 4 #f)
(call/kw foo 1 2 (d 4 e 5))      ; => (1 2 #f 4 5)
(call/kw foo 1 2 (e 5 c 3 d 4))  ; => (1 2 3 4 5)

Details

Syntax (lambda/kw (formals...​ (keywords...​)) body...​)

Like lambda, but makes a procedure that can take keyword arguments in addition to positional arguments.

formals are zero or more identifiers naming positional arguments. keywords are zero or more identifiers naming keyword arguments. All positional arguments need to be supplied by the caller on every call. There is no way to define optional positional arguments or a rest argument. By contrast, all keyword arguments are optional. Every keyword argument takes on the default value #f when no value is supplied by the caller. There is no support for user-defined default values; this simplifies the syntax of keyword lambdas and makes it easier to write wrapper procedures for them since the wrapper can always pass #f to any keyword argument it does not use.

body is evaluated as if in the context (lambda (...​) body...​) so anything that can go at the beginning of a lambda can go at the beginning of body. But the lambda around body may be wrapped inside another lambda depending on the implementation. Within body, all of the arguments in formals and keywords can be accessed as variables using the identifiers given by the user.

The returned procedure p can be called in any way that an ordinary procedure can, e.g. (p formals...​) or (apply p formals). If the number of arguments given to the procedure is equal to the number of formals, all keyword arguments take on the default value #f. Giving more arguments than that results in undefined behavior. To supply a value for one or more keyword arguments in a well-defined and portable way, call p via (call/kw p formals ...​ (keyword1 value1 ...​)). There may be additional implementation-defined ways to call p.

Syntax (define/kw (name formals...​ (keywords...​)) body...​)

Like lambda/kw, but binds the keyword lambda to name like define does.

Syntax (call/kw kw-lambda args...​ (kw-args...​))

This macro expands to a call to kw-lambda with zero or more positional arguments args and zero or more keyword arguments kw-args. It is an error to pass a different number of args than the number of positional arguments expected by kw-lambda.

Keyword arguments may be passed in any combination, and in any order. It is an error to have duplicate keywords in kw-args. As with ordinary procedure calls in Scheme, nothing is guaranteed about the evaluation order of args and kw-args. For a guaranteed evaluation order, bind arguments with let* before passing them in.

Each keyword argument in kw-args is written as two forms: keyword value. Each keyword is written as a standard Scheme symbol and is implicitly quoted (i.e. it has to be an identifier at read time; it cannot be some other expression that evaluates to a symbol). The symbols are internally converted to native keywords in implementations that have them.

Implementation

The sample implementation covers the following Scheme implementations that each have their own kind of native keywords:

There is also a generic R5RS implementation (relying on SRFI 46 tail patterns). This is used as the basis of generic R6RS and R7RS libraries for Schemes that don’t have native keywords.

The generic R6RS library works on the following Schemes:

The generic R7RS library works on the following Schemes:

Acknowledgements

Special thanks to Marc Nieper-Wißkirchen for his tireless championing of efficiency, hygiene and clarity on the SRFI's mailing list. In all Lisp systems we are aware of, keywords are global names: any two keywords with the same name are equal. Marc had the novel idea of exploring hygienic keywords so that keywords from different libraries would not clash, which would make it easier to write reliable utility libraries that add their own library-specific keyword arguments to procedures. Marc also had the idea of making define/kw expand to a macro instead of a procedure for efficiency reasons, and of using identifier syntax (familiar from R6RS syntax-case macros) to overload the macro definition so that using a bare identifier instead of procedure call syntax can still return a lambda for passing around as an object.

Thanks to John Cowan for encouragement, and for working hard to find a way to prepare SRFI 177 for inclusion in R7RS Large Edition. This ultimately didn't work out, but very useful discussions were had and future keyword SRFIs are still in the cards.

Thanks to Shiro Kawai for writing an optimized sample implementation for Gauche using er-macro-transformer.

Thanks to Amirouche Boubekki for balancing the discussion with skepticism of adding keyword arguments to Scheme at all.

Thanks to John and Shiro for surveys of the native keyword syntax and semantics of different Scheme implementations.

Thanks to many Scheme implementors and standardization people on the srfi-discuss mailing list for partaking in a big discussion about keyword unification in July 2019. Although the problem of unifying keyword syntax and semantics proved too complex to solve in the near future, the discussion spurred this SRFI to solve the most urgent problem of letting people write portable code.

Special thanks to Robert Strandh, author of the SICL Common Lisp implementation, for taking the time to explain compiler optimization of keyword argument passing to us Schemers.

Thanks to Göran Weinholt for collaborating on Docker containers that make it easy to work with exotic Scheme implementations.

Copyright

Copyright © Lassi Kortela (2019)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice (including the next paragraph) shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Editor: Arthur A. Gleckler