Failed to parse undelimited quote inside of here-doc

[idealseal]

The following works in bash:

var=$(
  sed 's/nice/cool/' - <<-EOF
  that's
  "nice"
EOF
)
echo $var

brush:

ERROR error: failed to parse word '$(
sed 's/is//' - <<-EOF
  that's
  is
  "nice"
EOF
)'

[reubeno]

I'm guessing this is a mix of #419 and #421.

[StudioLE]

I've been running into this issue as well.

The issue appears to be unpaired single quotes, double quotes, or backticks (" ' or `) in a heredoc in a command substitution.

It can be reproduced by adding these cases to
brush-shell/tests/cases/compat/here.yaml

cargo test --test brush-compat-tests

  - name: "Here doc with unpaired single quote in command substitution"
    stdin: |
      echo "$(cat <<'EOF'
      it's here
      EOF
      )"

  - name: "Here doc with unpaired double quote in command substitution"
    stdin: |
      echo "$(cat <<'EOF'
      she said "hello
      EOF
      )"

  - name: "Here doc with unpaired backtick in command substitution"
    stdin: |
      echo "$(cat <<'EOF'
      a ` b
      EOF
      )"

  - name: "Here doc with three single quotes in command substitution"
    stdin: |
      echo "$(cat <<'EOF'
      it's Bob's it's
      EOF
      )"

  - name: "Here doc with all three unmatched delimiters in command substitution"
    stdin: |
      echo "$(cat <<'EOF'
      `hello "world it's
      EOF
      )"

* Test case: [Here doc with unpaired single quote in command substitution]... 
    Oracle comparison:
      status matches (exit status: 0) ✔️
      stdout DIFFERS:
          ------ Oracle <> Test: stdout ---------------------------------
          + $(cat <<'EOF'
            it's here
          + EOF
          + )
            
          ---------------------------------------------------------------
      stderr matches ✔️
      temp dir matches ✔️
    FAILED.
* Test case: [Here doc with unpaired double quote in command substitution]... 
    Oracle comparison:
      status mismatch: exit status: 0 from oracle vs. exit status: 1 from test
      stdout DIFFERS:
          ------ Oracle <> Test: stdout ---------------------------------
          - she said "hello
          - 
          ---------------------------------------------------------------
      stderr DIFFERS:
          ------ Oracle <> Test: stderr ---------------------------------
          + error: failed to parse word '"$(cat <<'EOF'
          + she said "hello
          + EOF
          + )"'
          + 
          ---------------------------------------------------------------
      temp dir matches ✔️
    FAILED.
* Test case: [Here doc with unpaired backtick in command substitution]... 
    Oracle comparison:
      status matches (exit status: 0) ✔️
      stdout DIFFERS:
          ------ Oracle <> Test: stdout ---------------------------------
          + $(cat <<'EOF'
            a ` b
          + EOF
          + )
            
          ---------------------------------------------------------------
      stderr matches ✔️
      temp dir matches ✔️
    FAILED.
* Test case: [Here doc with three single quotes in command substitution]... 
    Oracle comparison:
      status matches (exit status: 0) ✔️
      stdout DIFFERS:
          ------ Oracle <> Test: stdout ---------------------------------
          + $(cat <<'EOF'
            it's Bob's it's
          + EOF
          + )
            
          ---------------------------------------------------------------
      stderr matches ✔️
      temp dir matches ✔️
    FAILED.
* Test case: [Here doc with all three unmatched delimiters in command substitution]... 
    Oracle comparison:
      status mismatch: exit status: 0 from oracle vs. exit status: 1 from test
      stdout DIFFERS:
          ------ Oracle <> Test: stdout ---------------------------------
          - `hello "world it's
          - 
          ---------------------------------------------------------------
      stderr DIFFERS:
          ------ Oracle <> Test: stderr ---------------------------------
          + error: failed to parse word '"$(cat <<'EOF'
          + `hello "world it's
          + EOF
          + )"'
          + 
          ---------------------------------------------------------------
      temp dir matches ✔️
    FAILED.

---

Analysis generated by Claude Code, but with considerable steering:

The root cause is in the word parser. The tokenizer correctly handles heredoc bodies inside $(...) - consume_nested_construct flattens the inner tokens (including heredoc body text) into the outer token's text via state.append_str. But the word parser's command_substitution rule uses "$(" command() ")" where command() = $(command_piece()*). Each command_piece delegates to word_piece, which tries double_quoted_sequence, single_quoted_literal_text, and backtick matching as its first alternatives.

These rules assume balanced quoting, but the tokenizer has already flattened heredoc body text (which can contain unbalanced quotes) into the token text with no way to distinguish it from regular code. When the unmatched delimiter finds a partner elsewhere in the token text (e.g. " matching the outer " from "$(...)"), PEG ordered choice commits to the match and consumes past the ) boundary, causing the parse to fail. When no partner is found, the quote rule fails but command_piece has no fallback for bare delimiter characters, so the content is parsed incorrectly.

PR #1014 solved the analogous problem at expansion time by introducing unexpanded_heredoc_word / heredoc_mode, but command() doesn't have access to that context since it's just finding the $(...) boundary.

flowchart TD
    A["Tokenizer: consume_nested_construct"] --> B["Flattens heredoc body into token text"]
    B --> C["Word parser receives full token"]
    C --> D["command_substitution rule"]
    D --> E["Matches opening dollar-paren"]
    E --> F["command - command_piece repeated"]
    F --> G["command_piece delegates to word_piece"]
    G --> H{"PEG ordered choice picks first match"}

    H --> I["1st: double_quoted_sequence"]
    H --> J["2nd: single_quoted_literal_text"]
    H --> K["3rd: unquoted_literal_text"]

    I --> I1{"Unmatched double-quote finds outer double-quote?"}
    I1 -->|Yes| I2["Consumes past closing paren"]
    I1 -->|No| I3["Fails, tries next alternative"]
    I2 --> FAIL["command_substitution cant find closing paren - PARSE FAILURE"]

    J --> J1{"Unmatched single-quote finds another single-quote later?"}
    J1 -->|Yes| J2["Consumes wrong span of text"]
    J1 -->|No| J3["Fails, tries next alternative"]
    J2 --> WRONG["Incorrect parse boundaries - WRONG OUTPUT"]

    K --> K1{"Char in exclusion set? single-quote double-quote backtick dollar"}
    K1 -->|Yes| K2["Fails with no fallback for bare delimiters"]
    K1 -->|No| K3["Matches literal text"]
    K2 --> WRONG

    style FAIL fill:#ff6b6b,color:#fff
    style WRONG fill:#ffa94d,color:#fff

Loading

The last point where we have heredoc context is inside consume_nested_construct. Tokens arrive with HereDocumentBodyStart, HereDocumentBodyEnd, and HereDocumentEndTag reasons and are buffered into pending_here_doc_tokens. They're then drained and flattened into the token text via state.append_str(cur_token_value.to_str()). After that line, the heredoc context is lost - it's just raw text in the token.