aws-s3-deployment: unexpected file tampering where a multiline JSON file gets manipulated to a single line after being uploaded to S3

[xiongbo-sjtu]

Describe the bug

We have a CDK package which leverages aws-s3-deployment to upload local files (each of which has multiline JSON representing an EMR cluster template) to a predefined S3 location. For each uploaded file, we expect that the content in S3 is identical to what's on the local disk.

What's the problem

We compute the md5 value for a local file, and then compare it against the Etag value of the corresponding remote file that's uploaded to S3 by aws-s3-deployment. We've never found a mismatch in the past 3 years until we consume a recent version (v2.185.0) of aws-cdk-lib.

Root cause

In 2025, there were non-backward compatible changes made to aws-s3-deployment/bucket-deployment-handler/index.py to address #22661.

However, this introduced a new bug: we got unexpected file tampering where a multiline JSON file gets manipulated to a single line after being uploaded to S3. As a result, the md5 value we compute for the corresponding local file doesn't match the Etag of the corresponding remote file in S3.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Library Version

v2.184.1

Expected Behavior

We have a CDK package which leverages aws-s3-deployment to upload local files (each of which has multiline JSON representing an EMR cluster template) to a predefined S3 location. For each uploaded file, we expect that the content in S3 is identical to what's on the local disk. Specifically, we compute the md5 value for a local file, and then compare it against the Etag value of the corresponding remote file that's uploaded to S3 by aws-s3-deployment. We expect the md5 value matches the Etag value, as observed in the past 3 years.

Current Behavior

We observe unexpected file tampering where a multiline JSON file gets manipulated to a single line after being uploaded to S3. As a result, the md5 value computed from the local file doesn't match the Etag of the corresponding remote file that's uploaded to S3 by aws-s3-deployment.

Reproduction Steps

• CDK code

const localPath = "lib/assets/cluster-templates"
const bucket = "my.s3.bucket"
const remotePath = "my/remote/path"

new s3Deployment.BucketDeployment(this, `UploadAssets-${index}`, {
    sources: [s3Deployment.Source.asset(localPath)],
    destinationBucket: bucket,
    destinationKeyPrefix: remotePath
});

• Local file

$md5 lib/assets/cluster-templates/RegularCluster.json
MD5 (lib/assets/cluster-templates/RegularCluster.json) = 1fe2c0c8b4e999eef7a5e4bcf341f8a5

• Remote file downloaded from S3: the following md5 matches the Etag provided by S3

$md5 ~/Downloads/RegularCluster.json
MD5 (~/Downloads/RegularCluster.json) = 77d7275f8d5aa6d089fb2e55c0ef5628

• Trimmed local file: I have to manipulate the content in order to match the above Etag

$cat lib/assets/cluster-templates/RegularCluster.json | jq -c | sed 's/,\([^ ]\)/, \1/g; s/:\([^/ ]\)/: \1/g; s/sys: /sys:/g' | tr -d '\n' | md5
77d7275f8d5aa6d089fb2e55c0ef5628

Possible Solution

Consider to introduce a flag in the BucketDeployment construct to support the old behavior, i.e., the file content should NOT be tampered when we leverage aws-s3-deployment to upload files from a local path to a remote S3 location.

Additional Information/Context

No response

AWS CDK Library version (aws-cdk-lib)

2.186.0

AWS CDK CLI version

2.1020.2

Node.js Version

23.1.0

OS

mac 15.5 (Sequoia)

Language

TypeScript

Language Version

No response

Other information

No response

[xiongbo-sjtu]

@gracelu0 @scorbiere FYI ^^^

[pahud]

@xiongbo-sjtu Thank you for the detailed report. Working on the investigation.

[pahud]

Hi @xiongbo-sjtu,

Thank you for reporting this regression issue. I can confirm this is a valid bug introduced in CDK v2.185.0 as a side effect of the fix for issue #22661.

Root Cause:
The issue is in the bucket deployment handler where JSON files are now being parsed and re-serialized using json.dumps(), which by default produces compact (single-line) output. This changes the file content and thus the MD5 hash, even when no marker replacement is needed.

Workaround:
For now, you can pin your CDK version to 2.184.1 to avoid this issue:

"aws-cdk-lib": "2.184.1"

Fix Needed:
The replace_markers_in_json function should preserve the original JSON formatting when possible, or at minimum use consistent indentation when re-serializing JSON.

Root Cause Analysis

The root cause is in the Python bucket deployment handler at packages/@aws-cdk/custom-resource-handlers/lib/aws-s3-deployment/bucket-deployment-handler/index.py.

Looking at the code, the issue stems from the replace_markers_in_json function and related JSON processing logic that was added to fix issue #22661:

def replace_markers_in_json(json_object, replace_tokens):
    """Replace markers in JSON content with proper escaping."""
    try:
        def replace_in_structure(obj):
            # ... processing logic ...

        processed = replace_in_structure(json_object)
        return json.dumps(processed)  # This compacts the JSON!
    except Exception as e:
        logger.error(f'Error processing JSON: {e}')
        return json_object

The problem is that json.dumps(processed) by default produces compact JSON output (single line), whereas the original file had multiline formatting. The fix for #22661 introduced JSON parsing and re-serialization, but didn't preserve the original formatting.

Reproducible or not

Yes, this issue is reproducible. I successfully created a minimal reproduction case that demonstrates the problem. The issue occurs when:

1. Using BucketDeployment with Source.asset() containing JSON files
2. The JSON files have multiline formatting
3. CDK version 2.185.0 or later is used

Here's the complete reproduction code I created to demonstrate the S3 deployment JSON formatting issue:

Directory Structure

/tmp/s3-deployment-issue/
├── assets/
│   └── test.json
├── lib/
│   └── s3-deployment-issue-stack.ts
├── bin/
│   └── s3-deployment-issue.ts
├── package.json
└── cdk.json

Test JSON File (assets/test.json)

{
  "name": "test-cluster",
  "version": "1.0.0",
  "configuration": {
    "master": {
      "instanceType": "m5.xlarge",
      "instanceCount": 1
    },
    "worker": {
      "instanceType": "m5.large",
      "instanceCount": 3
    }
  },
  "applications": [
    "Spark",
    "Hadoop"
  ]
}

CDK Stack (lib/s3-deployment-issue-stack.ts)

import * as cdk from 'aws-cdk-lib';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as s3deploy from 'aws-cdk-lib/aws-s3-deployment';
import { Construct } from 'constructs';

export class S3DeploymentIssueStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create S3 bucket
    const bucket = new s3.Bucket(this, 'TestBucket', {
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      autoDeleteObjects: true,
    });

    // Deploy assets to S3 - this should preserve the original JSON formatting
    new s3deploy.BucketDeployment(this, 'DeployAssets', {
      sources: [s3deploy.Source.asset('./assets')],
      destinationBucket: bucket,
      destinationKeyPrefix: 'cluster-templates/',
    });

    // Output the bucket name for testing
    new cdk.CfnOutput(this, 'BucketName', {
      value: bucket.bucketName,
      description: 'Name of the S3 bucket containing the deployed files',
    });
  }
}

App Entry Point (bin/s3-deployment-issue.ts)

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { S3DeploymentIssueStack } from '../lib/s3-deployment-issue-stack';

const app = new cdk.App();
new S3DeploymentIssueStack(app, 'S3DeploymentIssueStack', {
  env: {
    account: process.env.CDK_DEFAULT_ACCOUNT,
    region: process.env.CDK_DEFAULT_REGION,
  },
});

Setup Commands

# Create directory and initialize CDK app
mkdir s3-deployment-issue && cd s3-deployment-issue
cdk init app --language typescript

# Install specific CDK version that has the bug
npm install aws-cdk-lib@2.186.0

# Create assets directory and test file
mkdir -p assets
# (Create the test.json file as shown above)

# Build and synthesize
npm run build
npx cdk synth

Reproduction Steps

1. Check original file MD5:

md5 assets/test.json
# Output: MD5 (assets/test.json) = 5e19aa043d8c3900b087bca3a5334c74

2. Deploy the stack:

npx cdk deploy

3. Download the file from S3 and check MD5:

# Get bucket name from CDK output
BUCKET_NAME=$(aws cloudformation describe-stacks --stack-name S3DeploymentIssueStack --query 'Stacks[0].Outputs[?OutputKey==`BucketName`].OutputValue' --output text)

# Download the file
aws s3 cp s3://$BUCKET_NAME/cluster-templates/test.json ./downloaded-test.json

# Check MD5 of downloaded file
md5 downloaded-test.json
# This will show a DIFFERENT MD5 hash because the file is now single-line

4. Compare the files:

# Original (multiline)
cat assets/test.json

# Downloaded (single-line)
cat downloaded-test.json

Expected vs Actual Results

Expected: The downloaded file should have the same MD5 hash and formatting as the original.

Actual: The downloaded file will be compressed to a single line:

{"name": "test-cluster", "version": "1.0.0", "configuration": {"master": {"instanceType": "m5.xlarge", "instanceCount": 1}, "worker": {"instanceType": "m5.large", "instanceCount": 3}}, "applications": ["Spark", "Hadoop"]}

This demonstrates the regression where CDK v2.185.0+ automatically reformats JSON files during S3 deployment, breaking MD5 integrity checks that users rely on for validation.

Follow-up Actions

I'm tagging this as a regression and will escalate to the team for prioritization.

[ozelalisen]

I want to update you on our progress:

Current Status: Active Development

We have identified the root cause of the MD5 hash mismatch issue where files without markers were being unnecessarily rewritten during S3 deployment. I have developed and tested a fix that preserves original file formatting and MD5 hashes for files without markers

Next Steps:

• The fix is currently undergoing final review and testing
• Expected Release: Next week's weekly CDK release
• We will update this issue once the fix is available