Fix s3 cp --dryrun to validate bucket access before reporting success #9951
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #9935
Previously,
aws s3 cp --dryrunfor uploads would return exit code 0 even when the user lacked permissions to the destination bucket. This was inconsistent with download operations, which fail during file discovery whenHeadObjectis called.This change adds a
HeadBucketcall during dryrun for uploads to validate bucket access before reporting success. The validation is cached per bucket to avoid redundant API calls during recursive operations.Root Cause
The asymmetry existed because downloads and uploads check permissions at different stages:
HeadObjectis called during file discovery (before dryrun check), so permission errors surface regardless of--dryrun--dryrunthePutObjectcall is skipped entirely and permissions are never validatedImplementation Options Considered
Four approaches were evaluated:
Option 1: Use
CreateMultipartUpload+AbortMultipartUploads3:PutObjectpermissions3:AbortMultipartUploadpermission; if user hass3:PutObjectbut nots3:AbortMultipartUpload, this would leave orphaned multipart uploadsOption 2: Use
HeadBucket(chosen)s3:PutObjectpermissionOption 3: Use
PutObjectwith test key +DeleteObjects3:PutObjectpermissions3:DeleteObjectpermission, triggers S3 event notifications, potential race conditionsOption 4: No validation
Decision: Option 2 was chosen because it has no side effects and catches the most common access issues (wrong credentials, non-existent bucket, no bucket access). While it doesn't specifically validate
s3:PutObject, it provides meaningful validation that's consistent with the spirit of--dryrun.Changes
__init__method toUploadRequestSubmitterto initialize per-bucket validation cache_submit_dryrunoverride that callsHeadBucketbefore reporting successTest Plan
test_dry_run_validates_bucket_access- verifiesHeadBucketis calledtest_dry_run_fails_on_access_denied- verifies 403 errors propagatetest_dry_run_fails_on_nonexistent_bucket- verifies 404 errors propagatetest_dry_run_caches_bucket_validation- verifies per-bucket cachingtest_dry_run_validates_each_bucket_separately- verifies different buckets validated independently