stepfunctions: DistributedMap's ResultWriter with JsonATA generate invalid CF template

[TomasChmelik]

Describe the bug

Regression of #33396

When specifying DistributedMap.result_writer_v2's (but same thing happends when using V1) parameters in JsonATA format, the template should contain "Arguments", but it contains "Parameters"

This causes CF stack update with These fields are required: [Arguments] at followed by path to the DistributedMap/ResultWriter error message

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Library Version

No response

Expected Behavior

DistributedMap/ResultWriter has Arguments property where target bucket and prefix is set.

      DefinitionString:
        Fn::Join:
          - ""
          - - '{"StartAt":"DistributedMap","States":{"DistributedMap":{"Type":"Map","QueryLanguage":"JSONata","End":true,"ItemProcessor":{"ProcessorConfig":{"Mode":"DISTRIBUTED","ExecutionType":"STANDARD"},"StartAt":"Pass","States":{"Pass":{"Type":"Pass","End":true}}},"ResultWriter":{"Resource":"arn:'
            - Ref: AWS::Partition
            - :states:::s3:putObject","Arguments":{"Bucket":"
            - Ref: Bucket83908E77
            - '","Prefix":"{% $states.input.prefix %}"}}}}}'

Current Behavior

DistributedMap/ResultWriter has Parameters property where target bucket and prefix is set, even though the DistributedMap has JsonATA query language selected

      DefinitionString:
        Fn::Join:
          - ""
          - - '{"StartAt":"DistributedMap","States":{"DistributedMap":{"Type":"Map","QueryLanguage":"JSONata","End":true,"ItemProcessor":{"ProcessorConfig":{"Mode":"DISTRIBUTED","ExecutionType":"STANDARD"},"StartAt":"Pass","States":{"Pass":{"Type":"Pass","End":true}}},"ResultWriter":{"Resource":"arn:'
            - Ref: AWS::Partition
            - :states:::s3:putObject","Parameters":{"Bucket":"
            - Ref: Bucket83908E77
            - '","Prefix":"{% $states.input.prefix %}"}}}}}'

Reproduction Steps

from aws_cdk import aws_stepfunctions, aws_s3

bucket = aws_s3.Bucket(self, 'Bucket')

distributed_map = aws_stepfunctions.DistributedMap(
    self,
    'DistributedMap',
    query_language=aws_stepfunctions.QueryLanguage.JSONATA,
    # doesn't work for either
    result_writer=aws_stepfunctions.ResultWriter(
        bucket=bucket,
        prefix='{% $states.input.prefix %}',
    ),
    # ...or v2
    result_writer_v2=aws_stepfunctions.ResultWriterV2(
        bucket=bucket,
        prefix='{% $states.input.prefix %}',
    ),
)
distributed_map.item_processor(aws_stepfunctions.Pass(self, 'Pass'))

aws_stepfunctions.StateMachine(
    self,
    'StateMachine',
    definition_body=aws_stepfunctions.DefinitionBody.from_chainable(distributed_map),
    state_machine_name='state-machine',
)

Possible Solution

No response

Additional Information/Context

No response

AWS CDK Library version (aws-cdk-lib)

2.214.0

AWS CDK CLI version

2.1027.0

Node.js Version

v22.9.0

OS

Ubuntu

Language

Python

Language Version

Python (3.13)

Other information

No response

[pahud]

Hi @TomasChmelik,

Thank you for reporting this issue. I've investigated and can confirm that this bug still exists in aws-cdk-lib@2.214.0, but only for ResultWriter without ItemReader.

Issue Verification

I reproduced your exact case and confirmed the bug:

Your reproduction code (ResultWriter only, no ItemReader):

distributed_map = sfn.DistributedMap(
    stack,
    'DistributedMap',
    query_language=sfn.QueryLanguage.JSONATA,
    result_writer_v2=sfn.ResultWriterV2(
        bucket=bucket,
        prefix='{% $states.input.prefix %}',
    ),
)

❌ Result with aws-cdk-lib@2.214.0:

{
  "ResultWriter": {
    "Resource": "arn:aws:states:::s3:putObject",
    "Parameters": {         // ❌ WRONG: Uses "Parameters" for JSONata
      "Bucket": "...",
      "Prefix": "{% $states.input.prefix %}"
    }
  },
  "QueryLanguage": "JSONata"
}

Root Cause

The fix was incomplete. In DistributedMap.toStateJson(), the fix was applied to ItemReader but not to ResultWriter:

// Line 318 - BUG: Still uses stateMachineQueryLanguage
const renderedResultWriter = this.renderResultWriter(stateMachineQueryLanguage);

// Line 322 - The correct query language is calculated here
const stateQueryLanguage = this.queryLanguage ?? stateMachineQueryLanguage;

// Line 325 - ItemReader correctly uses stateQueryLanguage  
...this.renderItemReader(stateQueryLanguage),

The fix should be:

const renderedResultWriter = this.renderResultWriter(stateQueryLanguage);

Workaround

Until this is fixed, you can work around it by adding an ItemReader (which works correctly):

distributed_map = sfn.DistributedMap(
    stack,
    'DistributedMap',
    query_language=sfn.QueryLanguage.JSONATA,
    item_reader=sfn.S3CsvItemReader(
        bucket=bucket,
        key='input.csv',
        csv_headers=sfn.CsvHeaders.use_first_row(),
    ),
    result_writer_v2=sfn.ResultWriterV2(
        bucket=bucket,
        prefix='{% $states.input.prefix %}',
    ),
)

This issue needs to be reopened as the fix was incomplete. The ResultWriter still uses the wrong query language parameter.

Thanks for the detailed report!

[TomasChmelik]

For me the Issue happens also when item_reader is specified. Even with your proposed workaround solution the output is still using Parameters for the ResultWriter (yes, it uses Arguments correctly for ItemReader):

      DefinitionString:
        Fn::Join:
          - ""
          - - '{"StartAt":"DistributedMap","States":{"DistributedMap":{"Type":"Map","QueryLanguage":"JSONata","End":true,"ItemProcessor":{"ProcessorConfig":{"Mode":"DISTRIBUTED","ExecutionType":"STANDARD"},"StartAt":"Pass","States":{"Pass":{"Type":"Pass","End":true}}},"ItemReader":{"Resource":"arn:'
            - Ref: AWS::Partition
            - :states:::s3:getObject","ReaderConfig":{"InputType":"CSV","CSVHeaderLocation":"FIRST_ROW"},"Arguments":{"Bucket":"
            - Ref: Bucket83908E77
            - '","Key":"input.csv"}},"ResultWriter":{"Resource":"arn:'
            - Ref: AWS::Partition
            - :states:::s3:putObject","Parameters":{"Bucket":"         <-- wrong Parameters
            - Ref: Bucket83908E77
            - '","Prefix":"{% $states.input.prefix %}"}}}}}'

But from your investigation it shouldn't work, since query language used for ItemReader is completely unrelated to one used for ResultWriter. The workaround should be to switch the whole state machine to use JsonATA instead of individual steps.

Is there a timeline when this could be fixed (I mean it looks like one line fix)?

[TomasChmelik]

Also more investigation: when DistributedMap is wrapped either in Parallel or Map step (might be same for more step types, didn't check), it loses connection to State machine's query language.

class NotWorking(Construct):
    def __init__(self, scope: Construct, construct_id: str):
        super().__init__(scope, construct_id)

        bucket = aws_s3.Bucket(self, 'Bucket')

        distributed_map = aws_stepfunctions.DistributedMap(
            self,
            'DistributedMap',
            query_language=aws_stepfunctions.QueryLanguage.JSONATA,
            map_execution_type=aws_stepfunctions.StateMachineType.EXPRESS,
            item_reader=aws_stepfunctions.S3JsonLItemReader(
                bucket=bucket,
                key='{% $states.input.filename %}',
            ),
            jsonata_item_selector='{% $states.input ~> | $ | {"data": $states.context.Map.Item.Value} | %}',
            result_writer_v2=aws_stepfunctions.ResultWriterV2(
                bucket=bucket,
                prefix='{% $states.input.prefix %}',
                writer_config=aws_stepfunctions.WriterConfig(
                    output_type=aws_stepfunctions.OutputType.JSONL,
                    transformation=aws_stepfunctions.Transformation.NONE,
                ),
            ),
        )
        distributed_map.item_processor(aws_stepfunctions.Pass(self, 'Pass'))

        map = aws_stepfunctions.Map(self, 'Map', query_language=aws_stepfunctions.QueryLanguage.JSONATA)
        map.item_processor(distributed_map)

        # parallel = aws_stepfunctions.Parallel(self, 'Parallel', query_language=aws_stepfunctions.QueryLanguage.JSONATA)
        # parallel.branch(distributed_map)

        aws_stepfunctions.StateMachine(
            self,
            'StateMachine',
            query_language=aws_stepfunctions.QueryLanguage.JSONATA,
            definition_body=aws_stepfunctions.DefinitionBody.from_chainable(
                map,
                # parallel,
            ),
            state_machine_name='state-machine',
        )

In both cases the ResultWriter of distributed map is using Parameters instead of Argument even thought query language is set on state machine and all its states

[pahud]

Yes you are right.

#!/usr/bin/env python3

from aws_cdk import App, Stack, aws_stepfunctions as sfn, aws_s3 as s3
from constructs import Construct

class ReproStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        bucket = s3.Bucket(self, 'Bucket')

        # BUG: ResultWriter uses "Parameters" instead of "Arguments" for JSONata
        distributed_map = sfn.DistributedMap(
            self, 'DistributedMap',
            query_language=sfn.QueryLanguage.JSONATA,
            result_writer_v2=sfn.ResultWriterV2(
                bucket=bucket,
                prefix='{% $states.input.prefix %}',
            ),
        )
        distributed_map.item_processor(sfn.Pass(self, 'Pass'))

        sfn.StateMachine(self, 'StateMachine', 
            definition_body=sfn.DefinitionBody.from_chainable(distributed_map))

app = App()
ReproStack(app, "ReproStack")
app.synth()

Expected (correct):

"ResultWriter": {
  "Resource": "arn:aws:states:::s3:putObject",
  "Arguments": { "Bucket": "...", "Prefix": "..." }
}

Actual (bug):

"ResultWriter": {
  "Resource": "arn:aws:states:::s3:putObject",
  "Parameters": { "Bucket": "...", "Prefix": "..." }
}

The bug occurs because DistributedMap.toStateJson() calls renderResultWriter(stateMachineQueryLanguage) before calculating the correct stateQueryLanguage.

Is there a timeline when this could be fixed (I mean it looks like one line fix)?

Yes it looks like we only need to move stateQueryLanguage calculation before renderResultWriter() call and use the correct parameter.

Fix/Solution

File: packages/aws-cdk-lib/aws-stepfunctions/lib/states/distributed-map.ts

Current code (lines ~315-325):

public toStateJson(stateMachineQueryLanguage?: QueryLanguage): object {
  let rendered: any = super.toStateJson(stateMachineQueryLanguage);
  // ... other code ...

  rendered.ItemProcessor.ProcessorConfig.ExecutionType = this.mapExecutionType;
  const renderedResultWriter = this.renderResultWriter(stateMachineQueryLanguage); // ❌ WRONG
  this.addWarningIfResultWriterIsEmpty(renderedResultWriter);
  // ItemReader and ResultWriter configuration will base on the Map's query language.
  // If Map's query language is not specified, then use state machine's query language.
  const stateQueryLanguage = this.queryLanguage ?? stateMachineQueryLanguage;
  return {
    ...rendered,
    ...this.renderItemReader(stateQueryLanguage), // ✅ CORRECT
    // ... rest
    ...renderedResultWriter,
  };
}

Fixed code:

public toStateJson(stateMachineQueryLanguage?: QueryLanguage): object {
  let rendered: any = super.toStateJson(stateMachineQueryLanguage);
  // ... other code ...

  rendered.ItemProcessor.ProcessorConfig.ExecutionType = this.mapExecutionType;
  // ItemReader and ResultWriter configuration will base on the Map's query language.
  // If Map's query language is not specified, then use state machine's query language.
  const stateQueryLanguage = this.queryLanguage ?? stateMachineQueryLanguage;
  const renderedResultWriter = this.renderResultWriter(stateQueryLanguage); // ✅ FIXED
  this.addWarningIfResultWriterIsEmpty(renderedResultWriter);
  return {
    ...rendered,
    ...this.renderItemReader(stateQueryLanguage),
    // ... rest
    ...renderedResultWriter,
  };
}

This appears to be a partial fix.

Timeline:

1. Original Issue stepfunctions: resultwriter on jsonata distributed maps fails deployments #33396 (Feb 2025): DistributedMap ResultWriter didn't work with JSONata at all
2. Partial Fix Applied: Only ItemReader was fixed, ResultWriter was missed
3. Issue Incorrectly Closed: Marked as "fixed" but only 50% complete
4. Issue stepfunctions: DistributedMap's ResultWriter with JsonATA generate invalid CF template #35403 discovers ResultWriter was never actually fixed

I am making it a p1 bug for the team. While we'll prioritize this bug, we welcome community PRs as well. Thank you for the report.