Overhauled Cene syntax

gistfile1.txt

Overhauled Cene syntax:


In an s-expression context, end of file accumulates zero s-expressions, then ends.


In an s-expression context, whitespace does nothing, then proceeds as usual.


In an s-expression context, a span of identifier text accumulates one s-expression consisting of the corresponding string, then proceeds as usual.

  abc


In an s-expression context, \ followed by optional whitespace and a non-identifier character invokes that character's freestanding s-expression reader macro, then proceeds as usual.

  \#...


In an s-expression context, \ followed by optional whitespace and a span of identifier text, followed by either a span of whitespace or a colon, invokes that identifier's freestanding s-expression reader macro, then proceeds as usual.

  \abc ...
  \abc:...


In an s-expression context, this runs the bounded s-expression reader macro `abc`, then assumes the next character is a closing bracket and consumes it, then proceeds as usual.

(The whitespace versions consume all the whitespace that occurs after the identifier before they invoke the reader macro.)

  (abc ...)
  (abc:...)
  [abc ...]
  [abc:...]


In an s-expression context, this runs the bounded s-expression reader macro `abc`. It does not consume the character after that, even though it's usually a closing bracket. Then it proceeds as usual.

(The whitespace version consumes all the whitespace that occurs after the identifier before it invokes the reader macro.)

  /abc ...
  /abc:...


In an s-expression context, this causes an unmatched bracket error.

  )
  ]


In a markup context, end of file accumulates no text, then ends.


In a markup context, a span of typical characters (whitespace and non-whitespace) accumulates corresponding text, then proceeds as usual.

  abc


In a markup context, \ followed by optional whitespace and a non-identifier character invokes that character's freestanding markup reader macro, then proceeds as usual.

  \#...


In a markup context, \ followed by optional whitespace and a span of identifier text, followed by either a span of whitespace or a colon, invokes that identifier's freestanding markup reader macro, then proceeds as usual.

  \abc ...
  \abc:...


In a markup context, this runs the bounded markup reader macro `abc`, then assumes the next character is a closing bracket and consumes it, then proceeds as usual.

(The whitespace versions consume all the whitespace that occurs after the identifier before they invoke the reader macro.)

  (.abc ...)
  (.abc:...)
  [.abc ...]
  [.abc:...]


In a markup context, this does several things:

  This determines the most recent opening bracket, which we'll call the surrounding opening bracket.
  
  This accumulates text corresponding to the surrounding opening bracket. If the surrounding opening bracket is what initiated the markup section, this accumulates "/" instead.
  
  This runs the bounded markup reader macro `abc`. It does not consume the character after what the reader macro consumes, even though it's usually a closing bracket.
  
  This accumulates text corresponding to the surrounding opening bracket's closing bracket. If the surrounding opening bracket is what initiated the markup section, this accumulates no text in this step instead.
  
  Then it proceeds as usual.

(The whitespace version consumes all the whitespace that occurs after the identifier before it invokes the reader macro.)

  /.abc ...
  /.abc:...


In a markup context, an opening bracket or / not followed by \ behaves similarly to if it was followed by \, but it uses a nameless bounded markup reader macro instead.

  (...)
  [...]
  /


In a markup context, this causes an unmatched bracket error.

  )
  ]


Typical macros:

Freestanding s-expression reader macro `=`: Line comment.

Freestanding markup reader macro `=`: Line comment.

Freestanding markup reader macros ``` (the grave accent character) `,` `<` `>` `{` `}`: The characters \ . [ ] ( ) respectively.

Freestanding markup reader macro `#`: A whitespace normalization command that recommends removing surrounding raw whitespace and inserting nothing.

Freestanding markup reader macros `s` `t` `r` `n`: A whitespace normalization command that recommends removing surrounding raw whitespace and inserting a space, a tab, a carriage return, or a newline, respectively. The inserted character is not considered "raw" whitespace.

Nameless bounded markup reader macro: Proceeds as usual until it reaches a corresponding closing bracket.

Bounded markup reader macros `qq` `uq`: Respectively increments or decrements the quoting depth temporarily while it accumulates the markup in its body. Most markup reader macros operate normally at a depth of 1, cause an error at any lower depth, and accumulate their own syntax's non-whitespace text and raw whitespace at any greater depth.

Bounded markup reader macro `ls` (standing for "lists and strings"): Operates normally at quoting depth 0, where it reads a single s-expression in its body and accumulates a markup interpolation containing that s-expression.

Bounded s-expression reader macro `str`: Reads its body as markup, reduces raw whitespace sequences to a single space each, follows whitespace normalization commands, and accumulates a single s-expression that's an interpolated string data structure (a nonempty sequence of strings, with an s-expression in between every pair of neighboring strings).


Examples:

Line comment: \= ...
String when there's an `\n` escape: [str/.qq:Hello, \n [.uq/.ls name]!]
String when the only escape is `[.ls ...]`: [str:Hello, [.ls name]!]


Reader macro behavior:

When a reader macro is defined, the definition runs an expression to determine its implementation function. This expression has a local variable which is bound to a first-class scope value representing the place where it was defined.

For an s-expression reader macro, the implementation function receives a scope representing the call site; a text input stream; and a sequence-of-s-expressions output stream; and a callback to call with the rest of the input stream and output stream once the macro's computation is finished. The implementation function and its function are monadic, each returning a value that represents a side effect. The side effects do things like reading and writing to the streams, orchestrating concurrent computations, and reading and writing from a global definition space.

For a markup reader macro, the implementation function is similar, but it receives a markup output stream instead of a sequence-of-s-expressions output stream. The markup output stream can be written to with text, with interpolated s-expressions, or with spans of raw whitespace (whitespace which is eligible for being normalized).

A scope consists of two values: A unique name, and a "qualify" function that converts names that have been constructed from surface syntax into names that should be referenced in the global definition space. The unique name should be replaced with a fresh one whenever passing the scope along to a subcomputation. Every effectful computation that receives a scope should first write a dummy value to a name derived in a certain way from the scope's unique name, so that it can be sure no similar computation has been given access to that unique name.