How to support other Lisp dialects

[shaunlebron]

Parinfer is a lazy, partial reader— allowing it the opportunity to ignore most irrelevant differences between dialects, but it needs to support more specific dialect features

Parinfer assumes Clojure-like syntax— blowing past characters presumed to have no effect on parens and indentation. This allows some flexibility with other dialects, but eventually causes problems with other language features below:

See issues:

• Racket: support here-strings and reading-comments #172
• support for comment characters other than semicolon #195
• Allow unbalanced brackets in symbols #197
• Mishandling semicolon inside vertical bar literal oakmac/parinfer#16

Some dialects we look at:

• Racket
• Hy
• Common Lisp
• Janet
• Pico Lisp
• Scheme
• Fennel (should be identical to clj syntax)

Here-strings

strings delimited by custom phrases

• Racket: #<<FOO\n ... \nFOO>>\n doc-racket-here
• Hy: #[FOO[ ... ]FOO] doc-hy-here

Multi-line strings

• Janet: ` or `` doc-janet-multi

Nestable block comments

may be nested, unlike in C /*.../*...*/...*/)

• Racket: #|...|# doc-racket-block
• Common Lisp: #|...|# doc-cl-block
• Pico Lisp: #{...}# doc-pico-block

Bracket types

• Common Lisp: only ( ) are significant— allowing unbalanced { [ in symbols doc-cl-bracket

Comment lines

• Janet: # ... doc-janet-comment

Comment forms

A preceding # can re-interpret the usual ; comment as commenting the next form instead of to the end of line:

• Racket/Scheme: #; ... doc-racket-comment-form and R6RS Scheme

Literal symbols

pipes |...| allow special characters inside symbols. Sometimes \| is allowed inside (CL, not Racket):

• Racket: |foo;bar(baz| or foo|;|bar|(|baz doc-racket-pipe, doc-racket-pipe-sym
• Common Lisp: |foo;\|bar(baz| doc-cl-pipe

[shaunlebron]

Rough plan:

1. Put some separation between Reader and Parinfer

For example, I built a minimal auto-formatting tool for Clojure as a set of callbacks to Parinfer’s reader here:

https://github.com/parinfer/parindent/blob/gh-pages/read.js

This reader takes a hooks object of callbacks— each triggered at certain times when reading:

• onInitState
• onInitLine
• onOpener
• onIndent
• onFinalState

This might help me think more generally about what Parinfer needs to do and when.

2. Allow featureFlags or dialect option

If it is tractable, all featureFlags may be built into one reader, with combination presets supported with a dialect option.

3. Consider more non-recoverable “inside-out” error states

These feature flags may create more possibilities for unintended changes to strings or comments. (e.g. ERROR_QUOTE_DANGER)

The crux of the issue I believe is how non-directional delimiters like " (and now |) can be reversed while the document is being edited— i.e. a character can suddenly switch from meaning “open” to “close” if one is typed behind it. And rather than it becoming imbalanced (thus suspending Parinfer), the newly opened delimiter can become closed by another character— accidentally entering into a valid but unintended “inside-out” state, erroneously allowing Parinfer to run and make changes to non-structural parts of the code.

[shaunlebron]

We can use onecompiler.com to test Racket and Common Lisp:

https://onecompiler.com/racket/3xjvghuay
https://onecompiler.com/commonlisp/3xjvgwzbj