Skip to content

Conversation

@ps06756
Copy link

@ps06756 ps06756 commented Dec 29, 2025

Summary

Fixes #9935

Previously, aws s3 cp --dryrun for uploads would return exit code 0 even when the user lacked permissions to the destination bucket. This was inconsistent with download operations, which fail during file discovery when HeadObject is called.

This change adds a HeadBucket call during dryrun for uploads to validate bucket access before reporting success. The validation is cached per bucket to avoid redundant API calls during recursive operations.

Root Cause

The asymmetry existed because downloads and uploads check permissions at different stages:

  • Downloads: HeadObject is called during file discovery (before dryrun check), so permission errors surface regardless of --dryrun
  • Uploads: Only local files are listed during discovery (no S3 API calls), so with --dryrun the PutObject call is skipped entirely and permissions are never validated

Implementation Options Considered

Four approaches were evaluated:

Option 1: Use CreateMultipartUpload + AbortMultipartUpload

  • Pros: Directly validates s3:PutObject permission
  • Cons: Requires separate s3:AbortMultipartUpload permission; if user has s3:PutObject but not s3:AbortMultipartUpload, this would leave orphaned multipart uploads

Option 2: Use HeadBucket (chosen)

  • Pros: No side effects, validates bucket existence and basic access
  • Cons: Doesn't specifically validate s3:PutObject permission

Option 3: Use PutObject with test key + DeleteObject

  • Pros: Directly validates s3:PutObject permission
  • Cons: Creates/deletes real objects, requires s3:DeleteObject permission, triggers S3 event notifications, potential race conditions

Option 4: No validation

  • Keep current behavior
  • Cons: Issue remains unresolved

Decision: Option 2 was chosen because it has no side effects and catches the most common access issues (wrong credentials, non-existent bucket, no bucket access). While it doesn't specifically validate s3:PutObject, it provides meaningful validation that's consistent with the spirit of --dryrun.

Changes

  • Added __init__ method to UploadRequestSubmitter to initialize per-bucket validation cache
  • Added _submit_dryrun override that calls HeadBucket before reporting success
  • Added 5 regression tests validating the new behavior

Test Plan

  • All existing tests pass (64 tests in test_s3handler.py)
  • New regression tests added:
    • test_dry_run_validates_bucket_access - verifies HeadBucket is called
    • test_dry_run_fails_on_access_denied - verifies 403 errors propagate
    • test_dry_run_fails_on_nonexistent_bucket - verifies 404 errors propagate
    • test_dry_run_caches_bucket_validation - verifies per-bucket caching
    • test_dry_run_validates_each_bucket_separately - verifies different buckets validated independently

Previously, `aws s3 cp --dryrun` for uploads would return exit code 0
even when the user lacked permissions to the destination bucket. This
was inconsistent with download operations, which fail during file
discovery when HeadObject is called.

This change adds a HeadBucket call during dryrun for uploads to validate
bucket access before reporting success. The validation is cached per
bucket to avoid redundant API calls during recursive operations.

Fixes aws#9935
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

s3 cp with dryrun flag does not properly check IAM permissions

1 participant