Parser: AssignmentWord inside rule cmd_suffix

[39555]

I noticed that you have an AssignmentWord inside the cmd_suffix rule, so echo hello FOO=1 is parsed as CommandSuffix(Word, AssignmentWord). Is there any reason for this? I far as I know you can't set variables in the command suffix in Bash.

[39555]

And a follow-up question: why are there multiple CompleteCommands inside the Program, if CompleteCommand is an alias for CompoundList and CompoundList is a sequence of commands? What separates these sequences of commands in Program?

[reubeno]

I noticed that you have an AssignmentWord inside the cmd_suffix rule, so echo hello FOO=1 is parsed as CommandSuffix(Word, AssignmentWord). Is there any reason for this? I far as I know you can't set variables in the command suffix in Bash.

There's a specific set of built-ins that can (and need to) take assignments as suffix arguments, although the builtins implement the assignment themselves. Notably, this includes declare, local, export, et al. It's really most important for handling array assignments.

From the bash reference manual:

Assignment statements may also appear as arguments to the alias, declare, typeset, export, readonly, and local builtin commands (declaration commands).

Consider these two cases:

bash$ echo a=(1 2 3)
bash: syntax error near unexpected token `('
bash$ declare a=(1 2 3)
bash$ declare -p a
declare -a a=([0]="1" [1]="2" [2]="3")

In the case of echo, it's a syntax error. In the case of declare, it's an assignment that needs to be handled by the builtin itself (the args to declare may influence how to assign the variable).

[reubeno]

And a follow-up question: why are there multiple CompleteCommands inside the Program, if CompleteCommand is an alias for CompoundList and CompoundList is a sequence of commands? What separates these sequences of commands in Program?

Where possible, the peg grammar was closely modeled on the POSIX spec, e.g.:

program          : linebreak complete_commands linebreak
                 | linebreak
                 ;
complete_commands: complete_commands newline_list complete_command
                 |                                complete_command
                 ;
complete_command : list separator_op
                 | list

Deviations were introduced to support bash-isms, or to handle cases where PEG semantics didn't quite do the right thing.

[39555]

Thanks for answers!

[39555]

@reubeno It seems like there should only be one complete_command at a time:
https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html

In situations where the shell parses its input as a program, once a complete_command has been recognized by the grammar (see 2.10 Shell Grammar ), the complete_command shall be executed before the next complete_command is tokenized and parsed.

[reubeno]

The complete_command shall be executed before the next complete_command is tokenized and parsed.

Okay, this is a known limitation. We tokenize/parse the entire input program in one go, whether it's input coming from reedline/interactive prompt or from a sourced/executed script.

The tokenizer is designed to be streaming, i.e., you can call it to recognize one token at a time. The parser, however, doesn't work in the same way. It takes a sequence of tokens and does its best, potentially yielding multiple complete commands. Very early on in the project we'd experimented with nom and other streaming-compatible parser combinator crates, but we had a hard time making enough progress. Being able to translate the POSIX grammar into something that looked much like it was an accelerator.

With that all said, I'd definitely be interested in any (ideally incremental) ideas for how we can adjust the behavior to be able to recognize one complete command at a time. (Among other things, this would fix cases where commands--like setting shell options--have a side effect on how we tokenize/parse. extglob is an example that comes to mind.)

[39555]

@reubeno Can you provide an example of how two complete commands look like? I'm having a hard time imagining that. Usually we tokenize+parse either a script from a file, or a user input from the reedine one at a time.

[39555]

I've created a new page with the answer if someone also would be interested #263

[reubeno]

@39555 -- I definitely want to continue encouraging good questions like these. I'm going to convert this issue to a discussion in the Q&A channel? Will be a good excuse to bootstrap more activity there.

If you have specific proposed changes or have identified defects/enhancements, you're of course welcome to continue filing specific tracking issues.