Re: datum comments of sweet-expressions

["David A. Wheeler" <dwheeler@xxxxxxxxxxxx>]

I've modified the ANTLR grammar to support datum comments
of the form "#;"+space|tab, at the beginning of the line.  To do this:
* Productions can generate "empty", which means instead of just list/cons/append
   we need variants in actions that can handle "empty".  That causes many changes, but
   they're all pretty obvious, e.g., "cons" becomes "conse" (cons handling empty).
* A new terminal DATUM_COMMENTW is generated by "#;" followed by whitespace
   (and like the others doesn't consume the EOL) when we're in indent_processing mode.
* Several rules need to handle the new terminal.  The t_expr recurses to the "real" new
   rule t_expr_real if it get an empty result, and other minor changes handle it.
   Below is a summary.

Does this seem like the right direction?  If so, the next step is to implement
this in Scheme (and in Common Lisp).  This creates a requirement to "take over"
parts of the n-expression reader, unfortunately, but it does create useful functionality
(commenting datum blocks).

--- David A. Wheeler



================================================


@@ -1017,7 +1084,9 @@ collecting_tail returns [Object v]
 
 // Process line after ". hspace+" sequence.  Does not go past current line.
 post_period returns [Object v]
-  : scomment hspace* rpt=post_period {$v = $rpt.v;} // (scomment hspace*)*
+  : DATUM_COMMENTW hspace*
+      (ignored=n_expr hspace* sp2=post_period {$v = $sp2.v;} | /*empty*/ error )
+    | scomment hspace* rpt=post_period {$v = $rpt.v;} // (scomment hspace*)*
     | pn=n_expr hspace* (scomment hspace*)* (n_expr error)? {$v = $pn.v;}
     | COLLECTING hspace* pc=collecting_tail hspace*
       (scomment hspace*)* (n_expr error)? {$v = $pc.v;}

@@ -1066,6 +1135,8 @@ rest returns [Object v]
   : PERIOD /* Improper list */
       (hspace+  pp=post_period {$v = $pp.v;}
        | /*empty*/   {$v = list(".");})
+  | DATUM_COMMENTW hspace*
+    (ignored=n_expr hspace* sr2=rest {$v = $sr2.v;} | /*empty*/ error )
   | scomment hspace* (sr=rest {$v = $sr.v;} | /*empty*/ {$v = null;} )
   | COLLECTING hspace* collecting_tail hspace*
     (rr=rest             {$v = cons($collecting_tail.v, $rr.v);}

@@ -1088,9 +1159,11 @@ body returns [Object v]
      (same
        ( {isperiodp($i.v)}? => f=it_expr DEDENT
            {$v = $f.v;} // Improper list final value
-       | {! isperiodp($i.v)}? => nxt=body
-           {$v = cons($i.v, $nxt.v);} )
-     | DEDENT {$v = list($i.v);} ) ;
+       | {$i.v == empty}? => retry=body
+           {$v = $retry.v;}
+       | {!isperiodp($i.v) && ($i.v != empty)}? => nxt=body
+           {$v = conse($i.v, $nxt.v);} )
+     | DEDENT {$v = list1e($i.v);} ) ;
 
@@ -1128,22 +1201,21 @@ normal_it_expr returns [Object v]
 // An it_expr with a special prefix like \\ or $:
 
 special_it_expr returns [Object v]
-  : (GROUP_SPLIT | scomment) hspace* /* Initial; Interpet as group */
+  : DATUM_COMMENTW hspace*
+    (is_i=it_expr | comment_eol INDENT body ) {$v=empty;}
+  | (GROUP_SPLIT | scomment) hspace* /* Initial; Interpet as group */
       (group_i=it_expr {$v = $group_i.v;} /* Ignore initial GROUP/scomment */
        | comment_eol
          (INDENT g_body=body {$v = $g_body.v;} /* Normal GROUP use */
-          | same ( g_i=it_expr {$v = $g_i.v;} /* Plausible separator */
-                   /* Handle #!sweet EOL EOL t_expr */
-                   | comment_eol restart=t_expr {$v = $restart.v;} )
-          | DEDENT error ))
+          | same {$v = empty;} ))
 
 

--- David A. Wheeler