Mishandling semicolon inside vertical bar literal

[ylixir]

Parinfer thinks that semicolons between vertical bars is a comment (it's not). At least in common lisp, racket, and lumen this is wrong

(define |;| 5)
(print |;|)

is changed to

(define |);| 5)
(print |);|)

which is wrong.

reproductions:
https://onecompiler.com/racket/3xjvghuay
https://onecompiler.com/commonlisp/3xjvgwzbj

this is burning me when | is being used to quote literal code in a transpiler:
https://github.com/sctb/lumen/blob/master/test.l#L202

[ylixir]

Upstream version of parinfer-lua

[shaunlebron]

@ylixir Thanks for reporting. Could you link to the docs for Racket, Common Lisp, and Lumen for how to parse these and what they mean?

[ylixir]

it's more or less used to quote literals in all three cases

I don't think it's documented for lumen, although it's common enough in the source. https://github.com/sctb/lumen/blob/master/test.l#L202

see the last section on this page for an example in common lisp: https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node27.html

2nd paragraph in this section for racket: https://docs.racket-lang.org/reference/reader.html#%28part._default-readtable-dispatch%29

[shaunlebron]

Dang, Common Lisp allows escaped pipes |\|| but Racket doesn’t.

[gilch]

Common Lisp (at least) also allows newlines between vertical bars. E.g.,

(|foo
bar|)

is a valid function call form in CL, but Parinfer would corrupt the symbol by inserting a right parenthesis like

(|foo)
bar|

And now we have an unclosed parenthesis (the other one is internal to the symbol). I think the logic for handling string literals is close to what we want for these symbols.

[gilch]

Dang, Common Lisp allows escaped pipes |\|| but Racket doesn’t.

The vertical lines are called "multiple escape characters" in Common Lisp.

They're just for escapes, so you don't have to wrap the whole symbol, just like you don't need to use a backslash on every character so e.g., f|oo|bar resolves as foobar in Racket and as |FooBAR| in Common Lisp. Also, FOO| |BAR resolves the same as |FOO BAR| and FOO\ BAR, in both.

The single escape character (\) does work on | in Racket, so \| is a valid symbol token.

It is true that you can write (e.g.) a symbol named "f oo |b a r" as |f oo \|b a r| in Common Lisp and can't in Racket, but
|f oo |\||b a r| works in either.