Update proof of concept with respect to PEP 701 changes #22
Description
PEP 701, as tracked in the CPython implementation and recently merged by @pablogsal , python/cpython#102856, required changes to the tokenizer and parser for the cleaned up implementation of f-strings. This impacts the initial proof of concept work in #1 written by @gvanrossum
First, we have discussed two possible syntaxes, as seen in the below examples:
html'<li>{name}</li>'(this is what we have generally done, with the tag not being dotted and no whitespace between it and the string part)html @ f'<li>{name}</li>', as with the transpiler work by @rmorshea in Interim Transpiler #20
In my reading of the PEP, the easiest/most straightforward change is to leverage PEP 701 f-strings directly, producing these tokens in either case. (So the scheme in #20 more explicitly maps to this.)
NAME - "html"
FSTRING_START - "'"
FSTRING_MIDDLE - '<li>'
LBRACE - '{'
NAME - 'name'
RBRACE - '}'
FSTRING_MIDDLE - '</li>'
FSTRING_END - "'"
In other words, this is the exact tokenization produced as now, other than the first token NAME. This tokenization should be straightforward to then parse, including any necessary updates to the POC (possibly including removing more of the changes that were done, one can hope!).
The advantage of this approach (at least at first glance) is that we avoid duplicating a lot of the new work that was done for PEP 701, such as working with brace nesting, and then any subsequent tokenization of the interpolations with potential nesting.