220: Line directives220: Line directivesby Lassi Kortela
This SRFI is currently in withdrawn status. Here is an explanation of each status that a SRFI can hold. To provide input on this SRFI, please send email to srfi-220@nospamsrfi.schemers.org. To subscribe to the list, follow these instructions. You can access previous messages via the mailing list archive.
See the Withdrawal notice below for the author's explanation of his decision to withdraw this SRFI.
Many language-agnostic programming tools rely on specially formatted source code comments to annotate the code with metadata. Such "magic comments" are hard for both humans and computers to parse reliably, as the purpose of a comment is to be free-form text that is not interpreted by machine.
This SRFI extends the standard Scheme directive syntax
(#!) to support line directives. They look
like magic comments to language-agnostic tools but read as
S-expressions in Scheme, combining the portability of magic
comments with the well-defined syntax and easy parsing of
ordinary Scheme code.
While this SRFI solves the problem it set out to solve, and could be used as-is, the mailing list disagreed with it on aesthetic grounds. No maintainers of established Scheme implementations commented on the proposal.
Hence the SRFI is withdrawn. The document and the mailing list archive stay as a reference for future discussions on the topic.
Particular points of contention:
#! read syntax should be kept
strictly for Scheme datums or overloaded to also handle
external syntax.#!/usr/bin/scheme shebang
lines should be incorporated into Scheme syntax in some way, or
removed in a pre-processing step so the Scheme reader never
sees them. (Both approaches are currently in use by Scheme
implementations.)Like other Lisp dialects, Scheme uses S-expressions for its surface syntax. S-expressions are naturally extensible: just use a particular symbol as the head of a list to indicate what's coming up in the rest of the list. Ideally we would use S-expressions not only for data, but also to write metadata such as license information, character encoding, and editor settings.
However, most programming languages have a much less regular and more ad-hoc syntax and do not have easily extensible syntactic constructs. To the extent that extensible constructs exist, the culture to use them is lacking. As a result, most languages eventually paint themselves into a corner when the time comes to augment source code with metadata. As a last resort, the metadata is snuck into comments which are ignored by the compiler. This is unfortunate as comments are meant for natural language text and deliberately have no syntactic structure to be parsed, checked, and enforced by the compiler. That makes tools dealing with magic comments very brittle, and their precise behavior difficult to understand.
For interoperability with language-agnostic tools such as text editors and license checkers, we need to play along with the mess caused by the lack of foresight in other languages. However, we can salvage a bit of rigor by parsing the directives that other languages put in a comment as a sequence of S-expressions in Scheme. Happily, the character set permitted in Scheme and Lisp symbols is so wide that most magic comments (which tend to contain alphanumeric characters and some punctuation delimited by whitespace) will naturally split into a list of symbols.
If we can turn each directive into a list of symbols, it becomes easy to programmatically find all the directives in a Scheme source file. A Scheme implementation can process the directives it understands, ignore the rest, and optionally let users install custom hooks to process directives.
Some Scheme implementations have read syntax for keywords. The following syntaxes are known to exist:
foo::foo#:fooIt also varies whether or not keywords are disjoint from symbols.
The presence of keyword syntax may cause some tokens that are read as symbols in most Scheme implementations to be read as keywords in others. Language-agnostic directives tend to use colons a lot, which may cause them to be parsed differently in Scheme implementations with keywords.
In Scheme, a semicolon (;) starts a comment that
runs until the end of the line. In RnRS parlance, this is called
a line comment. An identifier following #!
is called a directive. Combining these two terms, we
arrive at line directive: a directive that runs until
the end of the line.
#! is the best choice for a directive marker
because most Scheme script readers already need special handling
for #! appearing at the very beginning of a source
file since #! is the Unix indicator that says which
interpreter runs the script. (When the first line of a Unix
script starts with #! it is often called the
"shebang line", or the "hashbang line".)
#! is also the only syntax used for reader
directives in standard Scheme, and a declaration of something
like character encoding is conceptually close to such
directives.
The ANSI Common Lisp standard, section 2.4.8 Syntax -> Standard
Macro Characters -> Sharpsign, says the dispatch macro
character #! is explicitly reserved to the user. No
conforming implementation defines it.
The
ISLisp standard (2007) uses # as a dispatch
macro character like Common Lisp and Scheme. It does not appear
to use #! for anything.
Scheme standards treat #! as a magic marker
generally followed by a directive that looks like an identifier
but serves a special purpose distinct from an ordinary identifier
with the same spelling.
#!true and
#!false for booleans and #!null for
the empty list.#!.#!.#!.#!r6rs and permits any
other identifier.#!fold-case and
#!no-fold-case.DSSSL and SRFI 89 use #!key and
#!optional and #!rest for lambda list
markers.
Various Scheme implementations use other identifiers following
#! for other
purposes.
SRFI 22 (Running Scheme Scripts on Unix) recognizes
#! at the very beginning of the file as starting a
comment that runs until the end of the line.
Guile uses #!...!# anywhere in the file for
multi-line comments.
# is a constituent character in symbols.
#! does not appear to be used for anything.
#! unofficially begins a comment
that runs until the end of the line.
Source: In the Clojure codebase, open
src/jvm/clojure/lang/LispReader.java and look for
dispatchMacros['!'] = new CommentReader();.
#! unofficially begins a comment
that runs until the end of the line.
Source: In the GNU Emacs codebase, open
src/lread.c and look inside read1().
The comment in the source code says #! appears at the
beginning of an executable file. Skip the first line. but
the code actually skips #! anywhere in the file.
#! at the very beginning of the
file starts a comment that runs until the end of the
line.
Source: In the Fennel codebase, open
src/fennel/parser.fnl and look inside
string-stream. It replaces a #! at the
start of the file with a traditional ;; comment
prefix.
# begins a comment that runs until the end of the
line. By extension, #! does the same.
# officially begins a comment
that runs until the end of the line. By extension,
#! does the same.
# is a constituent character in symbols.
Otherwise # does not appear to be used for anything,
and causes an illegal character syntax error.
# officially begins a comment
that runs until the end of the line. By extension,
#! does the same.
# officially begins a comment
that runs until the end of the line. By extension,
#! does the same.
As a minor exception, # does not begin a comment
when it is part of a symbol.
#!...!# at the very beginning of the
file is treated like a comment.
Elsewhere in the file #!key and
#!optional and #!rest are read in as
symbols, and #! followed by anything else is a
syntax error.
Source: In the librep codebase, open
src/lisp.c and look inside readl().
Both major Lisp standards, Common Lisp and Scheme, are easily
amenable to adding a #! syntax. While Common Lisp
reserves #! for users, customizing the readtable is
quite rare in practice, and the pervasive use of #!
in Unix scripts makes it unlikely that Common Lisp users would
configure #! to do anything other than read a
comment.
Of the surveyed non-standard Lisp dialects, all except
AutoLisp, Fennel, Lisp Flavored Erlang, and Rep handle
#! anywhere in the file akin to a comment. Both
Fennel and Rep handle #! at the beginning of the
file as a comment, which means their syntax could easily be
amended to handle #! elsewhere in the file the same
way. AutoLisp and LFE permit # in symbols. LFE does
not permit # to start a symbol; perhaps AutoLisp
doesn't either. If so, adding special meaning to #!
would not present a conflict in these dialects either.
Guile and Rep use #!...!# comments, i.e. they
require a !# at the end. We can expect
!# to cause a read error in most Scheme
implementations. However, using ;!# causes it to be
treated as an ordinary comment that runs until the end of the
line in those Schemes.
Scheme's read syntax is extended such that the two-character
sequence #! followed immediately by either a
newline, the end of the file, or one or more horizontal
whitespace characters causes a line directive to be read as
follows:
The accumulated list is returned as the Scheme representation of the line directive. The list is not evaluated as Scheme code, but may be either ignored or passed to an implementation-defined facility for processing.
It is an error to write things like #! #! foo or
#! outer (#! inner).
#! r6rs with a space and
#!r6rs withoutIt is implementation-defined whether #! r6rs
(with whitespace), once read, is processed the same way or in a
different way compared to #!r6rs (without
whitespace).
In R7RS section 7.1. Formal syntax, add a new case to
the definition of <directive>:
#! <intraline whitespace>+ <directive datums>
where
<directive datums> = | <datum> <intraline whitespace>* <directive datums>
and <datum> in this case contains no
newlines in the datum or surrounding whitespace.
For clarity, all Scheme symbols in the examples are written
using the |vertical-bar| syntax from R7RS.
Line directive:
#! /usr/bin/env fantastic-scheme
Read as S-expression:
(|/usr/bin/env| |fantastic-scheme|)
Line directives:
#! Copyright © 2019, 2020 Just A. Schemer <schemer@example.org> #! SPDX-License-Identifier: GPL-3.0-or-later
Read as S-expressions:
(|Copyright| |©| 2019 (|unquote| 2020) |Just| |A.| |Schemer| |<schemer@example.org>|) (|SPDX-License-Identifier:| |GPL-3.0-or-later|)
Line directive:
#! -*- mode: scheme -*-
Read as S-expression:
(|-*-| |mode:| |scheme| |-*-|)
Line directives:
#! Local Variables: #! mode: scheme #! coding: utf-8 #! comment-column: 0 #! End:
Read as S-expressions:
(|Local| |Variables:|) (|mode:| |scheme|) (|coding:| |utf-8|) (|comment-column:| 0) (|End:|)
Line directive:
#! vim: ft=lisp tw=60 ts=2 expandtab fileencoding=euc-jp :
Read as S-expression:
(|vim:| |ft=lisp| |tw=60| |ts=2| |expandtab| |fileencoding=euc-jp| |:|)
Line directive:
#! -*- mode: scheme -*- vim: set ft=scheme :
Read as S-expression:
(|-*-| |mode:| |scheme| |-*-| |vim:| |set| |ft=scheme| |:|)
A patch to the Gauche reader is attached. The patch makes it write all line directives read from source files to the current error port.
© 2021 Lassi Kortela.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice (including the next paragraph) shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.