Describe the bug
Greetings,
Encountered the following error while trying to perform "cdk deploy". We've deployed many pipelines and stacks in our organization, and never had an issue. However, we are facing this issue now, when trying to deploy a very simple stack. Have spent quite a bit of time debugging and trying to resolve this but had no luck.
Regression Issue
Last Known Working CDK Version
No response
Expected Behavior
Pipeline and Stack to be deploy with success.
Current Behavior
RampTestingCicdPipelineStack | 10:25:20 AM | CREATE_FAILED | AWS::S3::Bucket | integration-pipeline/Pipeline/ArtifactsBucket (integrationpipelinePipelineArtifactsBucketE50A534C) Resource handler returned message: "User: arn:aws:sts::914081002505:assumed-role/AWSReservedSSO_AdminAccess_bfe4506b0ea61cc6/ivan_ganza@tcenergy.com is not authorized to perform: s3:PutBucketPublicAccessBlock on resource: "arn:aws:s3:::ramptestingcicdpipelinest-integrationpipelinepipel-pjtqpacpxufo" with an explicit deny in an identity-based policy (Service: S3, Status Code: 403, Request ID: P2THEHWFC9ZJ8MYW, Extended Request ID: gCNyvR4utriG+TuLbC9RiW/kIGZGaBWdRSG5O42jsnClAmGrs3wQZl34SRIL7dG1g9k4vn6YCjg=)" (RequestToken: 4169c8a4-34a4-cb0e-84c7-34f7483b9a67, HandlerErrorCode: GeneralServiceException)
cdk deploy RampTestingCicdPipelineStack -v
✨ Synthesis time: 7.4s
[10:24:51] Checking for previously published assets
[10:24:51] 0 total assets, 0 still need to be published
[10:24:51] Reading existing template for stack RampTestingCicdPipelineStack.
Lookup role does not exist, hence was not assumed. Proceeding with default credentials.
RampTestingCicdPipelineStack: deploying... [1/1]
[10:24:53] Found existing stack RampTestingCicdPipelineStack that had previously failed creation. Deleting it before attempting to re-create it.
[10:24:53] Waiting for stack RampTestingCicdPipelineStack to finish creating or updating...
[10:24:54] Stack RampTestingCicdPipelineStack has an ongoing operation in progress and is not stable (DELETE_IN_PROGRESS (User Initiated))
[10:24:59] Call failed: describeStacks({"StackName":"RampTestingCicdPipelineStack"}) => Stack with id RampTestingCicdPipelineStack does not exist (code=ValidationError)
[10:24:59] Stack RampTestingCicdPipelineStack does not exist
[10:24:59] RampTestingCicdPipelineStack: checking if we can skip deploy
[10:24:59] RampTestingCicdPipelineStack: no existing stack
[10:24:59] RampTestingCicdPipelineStack: deploying...
[10:24:59] Attempting to create ChangeSet with name cdk-deploy-change-set to create stack RampTestingCicdPipelineStack
RampTestingCicdPipelineStack: creating CloudFormation changeset...
[10:25:00] Initiated creation of changeset: arn:aws:cloudformation:us-west-2:914081002505:changeSet/cdk-deploy-change-set/6d792a62-2992-4ed6-8d3e-025544de3da4; waiting for it to finish creating...
[10:25:00] Waiting for changeset cdk-deploy-change-set on stack RampTestingCicdPipelineStack to finish creating...
[10:25:00] Changeset cdk-deploy-change-set on stack RampTestingCicdPipelineStack is still creating
[10:25:05] Changeset cdk-deploy-change-set on stack RampTestingCicdPipelineStack is still creating
[10:25:11] Initiating execution of changeset arn:aws:cloudformation:us-west-2:914081002505:changeSet/cdk-deploy-change-set/6d792a62-2992-4ed6-8d3e-025544de3da4 on stack RampTestingCicdPipelineStack
[10:25:12] Execution of changeset arn:aws:cloudformation:us-west-2:914081002505:changeSet/cdk-deploy-change-set/6d792a62-2992-4ed6-8d3e-025544de3da4 on stack RampTestingCicdPipelineStack has started; waiting for the update to complete...
[10:25:12] Waiting for stack RampTestingCicdPipelineStack to finish creating or updating...
[10:25:12] Stack RampTestingCicdPipelineStack has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS (User Initiated))
[10:25:17] Stack RampTestingCicdPipelineStack has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
RampTestingCicdPipelineStack | 0/24 | 10:25:00 AM | REVIEW_IN_PROGRESS | AWS::CloudFormation::Stack | RampTestingCicdPipelineStack User Initiated
RampTestingCicdPipelineStack | 0/24 | 10:25:11 AM | CREATE_IN_PROGRESS | AWS::CloudFormation::Stack | RampTestingCicdPipelineStack User Initiated
RampTestingCicdPipelineStack | 0/24 | 10:25:15 AM | CREATE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/Role (integrationpipelinePipelineRole1B17CC51)
RampTestingCicdPipelineStack | 0/24 | 10:25:15 AM | CREATE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/Build/Synth/CdkBuildProject/Role (integrationpipelinePipelineBuildSynthCdkBuildProjectRole3E4C2195)
RampTestingCicdPipelineStack | 0/24 | 10:25:15 AM | CREATE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/UpdatePipeline/SelfMutation/Role (integrationpipelineUpdatePipelineSelfMutationRole6DD8D6B9)
RampTestingCicdPipelineStack | 0/24 | 10:25:15 AM | CREATE_IN_PROGRESS | AWS::S3::Bucket | integration-pipeline/Pipeline/ArtifactsBucket (integrationpipelinePipelineArtifactsBucketE50A534C)
RampTestingCicdPipelineStack | 0/24 | 10:25:15 AM | CREATE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/EventsRole (integrationpipelinePipelineEventsRole80B1923D)
RampTestingCicdPipelineStack | 0/24 | 10:25:15 AM | CREATE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/dev/Deploy/CodePipelineActionRole (integrationpipelinePipelinedevDeployCodePipelineActionRoleB9B53201)
RampTestingCicdPipelineStack | 0/24 | 10:25:15 AM | CREATE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/dev/Deploy/Role (integrationpipelinePipelinedevDeployRole7038C305)
RampTestingCicdPipelineStack | 0/24 | 10:25:15 AM | CREATE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/Source/ramp-testing/CodePipelineActionRole (integrationpipelinePipelineSourceramptestingCodePipelineActionRole28676500)
RampTestingCicdPipelineStack | 0/24 | 10:25:15 AM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata)
RampTestingCicdPipelineStack | 0/24 | 10:25:16 AM | CREATE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/Role (integrationpipelinePipelineRole1B17CC51) Resource creation Initiated
RampTestingCicdPipelineStack | 0/24 | 10:25:16 AM | CREATE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/Source/ramp-testing/CodePipelineActionRole (integrationpipelinePipelineSourceramptestingCodePipelineActionRole28676500) Resource creation Initiated
RampTestingCicdPipelineStack | 0/24 | 10:25:16 AM | CREATE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/Build/Synth/CdkBuildProject/Role (integrationpipelinePipelineBuildSynthCdkBuildProjectRole3E4C2195) Resource creation Initiated
RampTestingCicdPipelineStack | 0/24 | 10:25:16 AM | CREATE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/UpdatePipeline/SelfMutation/Role (integrationpipelineUpdatePipelineSelfMutationRole6DD8D6B9) Resource creation Initiated
RampTestingCicdPipelineStack | 0/24 | 10:25:16 AM | CREATE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/dev/Deploy/CodePipelineActionRole (integrationpipelinePipelinedevDeployCodePipelineActionRoleB9B53201) Resource creation Initiated
RampTestingCicdPipelineStack | 0/24 | 10:25:16 AM | CREATE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/EventsRole (integrationpipelinePipelineEventsRole80B1923D) Resource creation Initiated
RampTestingCicdPipelineStack | 0/24 | 10:25:16 AM | CREATE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/dev/Deploy/Role (integrationpipelinePipelinedevDeployRole7038C305) Resource creation Initiated
RampTestingCicdPipelineStack | 0/24 | 10:25:16 AM | CREATE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata) Resource creation Initiated
RampTestingCicdPipelineStack | 1/24 | 10:25:16 AM | CREATE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata)
RampTestingCicdPipelineStack | 1/24 | 10:25:17 AM | CREATE_IN_PROGRESS | AWS::S3::Bucket | integration-pipeline/Pipeline/ArtifactsBucket (integrationpipelinePipelineArtifactsBucketE50A534C) Resource creation Initiated
[10:25:23] Stack RampTestingCicdPipelineStack has an ongoing operation in progress and is not stable (ROLLBACK_IN_PROGRESS)
RampTestingCicdPipelineStack | 0/24 | 10:25:23 AM | DELETE_COMPLETE | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata)
RampTestingCicdPipelineStack | 0/24 | 10:25:20 AM | CREATE_FAILED | AWS::S3::Bucket | integration-pipeline/Pipeline/ArtifactsBucket (integrationpipelinePipelineArtifactsBucketE50A534C) Resource handler returned message: "User: arn:aws:sts::914081002505:assumed-role/AWSReservedSSO_AdminAccess_bfe4506b0ea61cc6/ivan_ganza@tcenergy.com is not authorized to perform: s3:PutBucketPublicAccessBlock on resource: "arn:aws:s3:::ramptestingcicdpipelinest-integrationpipelinepipel-pjtqpacpxufo" with an explicit deny in an identity-based policy (Service: S3, Status Code: 403, Request ID: P2THEHWFC9ZJ8MYW, Extended Request ID: gCNyvR4utriG+TuLbC9RiW/kIGZGaBWdRSG5O42jsnClAmGrs3wQZl34SRIL7dG1g9k4vn6YCjg=)" (RequestToken: 4169c8a4-34a4-cb0e-84c7-34f7483b9a67, HandlerErrorCode: GeneralServiceException)
RampTestingCicdPipelineStack | 0/24 | 10:25:20 AM | CREATE_FAILED | AWS::IAM::Role | integration-pipeline/Pipeline/Build/Synth/CdkBuildProject/Role (integrationpipelinePipelineBuildSynthCdkBuildProjectRole3E4C2195) Resource creation cancelled
RampTestingCicdPipelineStack | 0/24 | 10:25:20 AM | CREATE_FAILED | AWS::IAM::Role | integration-pipeline/Pipeline/Role (integrationpipelinePipelineRole1B17CC51) Resource creation cancelled
RampTestingCicdPipelineStack | 0/24 | 10:25:20 AM | CREATE_FAILED | AWS::IAM::Role | integration-pipeline/Pipeline/Source/ramp-testing/CodePipelineActionRole (integrationpipelinePipelineSourceramptestingCodePipelineActionRole28676500) Resource creation cancelled
RampTestingCicdPipelineStack | 0/24 | 10:25:20 AM | CREATE_FAILED | AWS::IAM::Role | integration-pipeline/Pipeline/dev/Deploy/Role (integrationpipelinePipelinedevDeployRole7038C305) Resource creation cancelled
RampTestingCicdPipelineStack | 0/24 | 10:25:20 AM | CREATE_FAILED | AWS::IAM::Role | integration-pipeline/UpdatePipeline/SelfMutation/Role (integrationpipelineUpdatePipelineSelfMutationRole6DD8D6B9) Resource creation cancelled
RampTestingCicdPipelineStack | 0/24 | 10:25:20 AM | CREATE_FAILED | AWS::IAM::Role | integration-pipeline/Pipeline/EventsRole (integrationpipelinePipelineEventsRole80B1923D) Resource creation cancelled
RampTestingCicdPipelineStack | 0/24 | 10:25:20 AM | CREATE_FAILED | AWS::IAM::Role | integration-pipeline/Pipeline/dev/Deploy/CodePipelineActionRole (integrationpipelinePipelinedevDeployCodePipelineActionRoleB9B53201) Resource creation cancelled
RampTestingCicdPipelineStack | 0/24 | 10:25:21 AM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack | RampTestingCicdPipelineStack The following resource(s) failed to create: [integrationpipelinePipelineRole1B17CC51, integrationpipelineUpdatePipelineSelfMutationRole6DD8D6B9, integrationpipelinePipelinedevDeployCodePipelineActionRoleB9B53201, integrationpipelinePipelinedevDeployRole7038C305, integrationpipelinePipelineSourceramptestingCodePipelineActionRole28676500, integrationpipelinePipelineEventsRole80B1923D, integrationpipelinePipelineBuildSynthCdkBuildProjectRole3E4C2195, integrationpipelinePipelineArtifactsBucketE50A534C]. Rollback requested by user.
RampTestingCicdPipelineStack | 0/24 | 10:25:23 AM | DELETE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/EventsRole (integrationpipelinePipelineEventsRole80B1923D)
RampTestingCicdPipelineStack | 0/24 | 10:25:23 AM | DELETE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/dev/Deploy/Role (integrationpipelinePipelinedevDeployRole7038C305)
RampTestingCicdPipelineStack | 0/24 | 10:25:23 AM | DELETE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/Source/ramp-testing/CodePipelineActionRole (integrationpipelinePipelineSourceramptestingCodePipelineActionRole28676500)
RampTestingCicdPipelineStack | 0/24 | 10:25:23 AM | DELETE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/UpdatePipeline/SelfMutation/Role (integrationpipelineUpdatePipelineSelfMutationRole6DD8D6B9)
RampTestingCicdPipelineStack | 0/24 | 10:25:23 AM | DELETE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/Role (integrationpipelinePipelineRole1B17CC51)
RampTestingCicdPipelineStack | 0/24 | 10:25:23 AM | DELETE_IN_PROGRESS | AWS::CDK::Metadata | CDKMetadata/Default (CDKMetadata)
RampTestingCicdPipelineStack | 0/24 | 10:25:23 AM | DELETE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/Build/Synth/CdkBuildProject/Role (integrationpipelinePipelineBuildSynthCdkBuildProjectRole3E4C2195)
RampTestingCicdPipelineStack | 0/24 | 10:25:23 AM | DELETE_IN_PROGRESS | AWS::IAM::Role | integration-pipeline/Pipeline/dev/Deploy/CodePipelineActionRole (integrationpipelinePipelinedevDeployCodePipelineActionRoleB9B53201)
RampTestingCicdPipelineStack | 0/24 | 10:25:23 AM | DELETE_SKIPPED | AWS::S3::Bucket | integration-pipeline/Pipeline/ArtifactsBucket (integrationpipelinePipelineArtifactsBucketE50A534C)
[10:25:28] Stack RampTestingCicdPipelineStack has an ongoing operation in progress and is not stable (ROLLBACK_IN_PROGRESS)
[10:25:34] Stack RampTestingCicdPipelineStack has an ongoing operation in progress and is not stable (ROLLBACK_IN_PROGRESS)
RampTestingCicdPipelineStack | 1/24 | 10:25:35 AM | DELETE_COMPLETE | AWS::IAM::Role | integration-pipeline/Pipeline/EventsRole (integrationpipelinePipelineEventsRole80B1923D)
RampTestingCicdPipelineStack | 2/24 | 10:25:35 AM | DELETE_COMPLETE | AWS::IAM::Role | integration-pipeline/Pipeline/dev/Deploy/Role (integrationpipelinePipelinedevDeployRole7038C305)
RampTestingCicdPipelineStack | 3/24 | 10:25:35 AM | DELETE_COMPLETE | AWS::IAM::Role | integration-pipeline/Pipeline/Source/ramp-testing/CodePipelineActionRole (integrationpipelinePipelineSourceramptestingCodePipelineActionRole28676500)
RampTestingCicdPipelineStack | 4/24 | 10:25:35 AM | DELETE_COMPLETE | AWS::IAM::Role | integration-pipeline/Pipeline/Role (integrationpipelinePipelineRole1B17CC51)
RampTestingCicdPipelineStack | 5/24 | 10:25:35 AM | DELETE_COMPLETE | AWS::IAM::Role | integration-pipeline/Pipeline/Build/Synth/CdkBuildProject/Role (integrationpipelinePipelineBuildSynthCdkBuildProjectRole3E4C2195)
RampTestingCicdPipelineStack | 6/24 | 10:25:35 AM | DELETE_COMPLETE | AWS::IAM::Role | integration-pipeline/UpdatePipeline/SelfMutation/Role (integrationpipelineUpdatePipelineSelfMutationRole6DD8D6B9)
RampTestingCicdPipelineStack | 7/24 | 10:25:36 AM | DELETE_COMPLETE | AWS::IAM::Role | integration-pipeline/Pipeline/dev/Deploy/CodePipelineActionRole (integrationpipelinePipelinedevDeployCodePipelineActionRoleB9B53201)
RampTestingCicdPipelineStack | 8/24 | 10:25:36 AM | ROLLBACK_COMPLETE | AWS::CloudFormation::Stack | RampTestingCicdPipelineStack
Failed resources:
RampTestingCicdPipelineStack | 10:25:20 AM | CREATE_FAILED | AWS::S3::Bucket | integration-pipeline/Pipeline/ArtifactsBucket (integrationpipelinePipelineArtifactsBucketE50A534C) Resource handler returned message: "User: arn:aws:sts::914081002505:assumed-role/AWSReservedSSO_AdminAccess_bfe4506b0ea61cc6/ivan_ganza@tcenergy.com is not authorized to perform: s3:PutBucketPublicAccessBlock on resource: "arn:aws:s3:::ramptestingcicdpipelinest-integrationpipelinepipel-pjtqpacpxufo" with an explicit deny in an identity-based policy (Service: S3, Status Code: 403, Request ID: P2THEHWFC9ZJ8MYW, Extended Request ID: gCNyvR4utriG+TuLbC9RiW/kIGZGaBWdRSG5O42jsnClAmGrs3wQZl34SRIL7dG1g9k4vn6YCjg=)" (RequestToken: 4169c8a4-34a4-cb0e-84c7-34f7483b9a67, HandlerErrorCode: GeneralServiceException)
Reproduction Steps
cdk deploy RampTestingCicdPipelineStack
from aws_cdk import Stack, Tags
from aws_cdk.aws_codecommit import Repository
from aws_cdk.aws_codebuild import BuildEnvironment, BuildSpec, Cache, ComputeType, LocalCacheMode
from aws_cdk.aws_ecr import Repository as EcrRepo
from aws_cdk.aws_secretsmanager import Secret
from aws_cdk.pipelines import (
CodeBuildOptions,
CodePipeline,
CodePipelineSource,
DockerCredential,
ManualApprovalStep,
ShellStep
)
from aws_cdk import (
aws_s3 as s3,
)
from constructs import Construct
from library_layer.config import ConfigFactory
from infra.pipeline_app_stage import AwsAccountInfo, PipelineAppStage
config = ConfigFactory()()
class CicdPipelineStack(Stack):
"""
CICD pipeline for building and deploying stacks and docker images
to all environments.
"""
def __init__(
self,
scope: Construct,
construct_id: str,
aws_accounts: dict[str, AwsAccountInfo],
**kwargs
) -> None:
super().__init__(scope, construct_id, **kwargs)
# Import the code commit repository that will be the source for
# the pipeline
code_repository = Repository.from_repository_name(self, 'repo',
repository_name=config.REPO_NAME
)
python_base_ecr_repo = EcrRepo.from_repository_name(self, 'python-base-ecr-repo',
repository_name="python-base-image"
)
docker_hub_secret = Secret.from_secret_name_v2(self, 'docker-hub-secret',
secret_name="/ramp/core/docker-build-token"
)
# Create the CICD pipeline
pipeline = CodePipeline(self, 'integration-pipeline',
pipeline_name='ramp-testing-integration-pipeline',
# artifact_bucket=existing_bucket,
synth=ShellStep('Synth',
input=CodePipelineSource.code_commit(
repository=code_repository,
branch='main' # Branch that you want to set as the trigger for the build
),
commands=[
"npm install -g aws-cdk",
"pip install --user poetry",
"python -m poetry install",
"python -m poetry run cdk synth"
]
),
docker_enabled_for_synth=True,
docker_enabled_for_self_mutation=True,
use_change_sets=False,
docker_credentials=[
DockerCredential.docker_hub(docker_hub_secret),
DockerCredential.ecr([python_base_ecr_repo])
],
code_build_defaults=CodeBuildOptions(
build_environment=BuildEnvironment(
compute_type=ComputeType.MEDIUM,
),
cache=Cache.local(LocalCacheMode.DOCKER_LAYER, LocalCacheMode.CUSTOM),
partial_build_spec=BuildSpec.from_object({
'cache': {'paths': ['/root/.cache/**/*']}
})
)
)
dev_stage = PipelineAppStage(self, 'dev',
env=aws_accounts['nonprod_account'].env,
aws_account=aws_accounts['nonprod_account'],
config=ConfigFactory('dev')()
)
pipeline.add_stage(dev_stage)
Possible Solution
No response
Additional Information/Context
No response
CDK CLI Version
2.154.1 (build febce9d)
Framework Version
No response
Node.js Version
v18.20.3
OS
Linux D-403852 5.15.153.1-microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:14:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Language
Python
Language Version
Python 3.11.3
Other information
No response
Describe the bug
Greetings,
Encountered the following error while trying to perform "cdk deploy". We've deployed many pipelines and stacks in our organization, and never had an issue. However, we are facing this issue now, when trying to deploy a very simple stack. Have spent quite a bit of time debugging and trying to resolve this but had no luck.
Regression Issue
Last Known Working CDK Version
No response
Expected Behavior
Pipeline and Stack to be deploy with success.
Current Behavior
Reproduction Steps
cdk deploy RampTestingCicdPipelineStackPossible Solution
No response
Additional Information/Context
No response
CDK CLI Version
2.154.1 (build febce9d)
Framework Version
No response
Node.js Version
v18.20.3
OS
Linux D-403852 5.15.153.1-microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:14:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Language
Python
Language Version
Python 3.11.3
Other information
No response