eks: a destructive update of the eks cluster and fails to do so

[watany-dev]

Describe the bug

When updating an eks Cluster, there is no permission to DeleteCluster from Role if there are destructive changes to the cluster.

https://github.com/aws/aws-cdk/blob/main/packages/%40aws-cdk/aws-eks/lib/cluster-resource.ts#L152-L166

The following rolls failed. Probably because of the following

1. The "resource" in the IAM policy is changed to the new cluster.
2. Attempting to "eks:DeleteCluster" on the old cluster
3. the deletion fails because the access rights to the old cluster were revoked in 1.

Expected Behavior

Successful deletion

Current Behavior

Failure to do this.

Reproduction Steps

First deployed as follows.

    new eks.Cluster(this, 'hello-eks', {
      version: eks.KubernetesVersion.V1_24,
      kubectlLayer: new KubectlV24Layer(this, 'kubectl'),
    });

Then update by rewriting it like this (changing the cluster name is a destructive change)

    new eks.Cluster(this, 'hello-eks', {
      clusterName: "sample",
      version: eks.KubernetesVersion.V1_24,
      kubectlLayer: new KubectlV24Layer(this, 'kubectl'),
    });

Possible Solution

EKS:DeleteCluster's Statement separation, Resouce='*'

Additional Information/Context

No response

CDK CLI Version

2.62.2

Framework Version

No response

Node.js Version

any

OS

any

Language

Typescript

Language Version

No response

Other information

No response

[pahud]

Hi @watany-dev , what is the error message when you delete the cluster?

[watany-dev]

@pahud
Thanks for bringing this to my attention.

I logged it again. This is when the cluster name is changed from "sample" to "sample2”.

Received response status [FAILED] from custom resource. Message returned: User: arn:aws:sts::aaaaaaaaaaaaa:assumed-role/eks-dev-helloe
ksCreationRole0B71AB61-DS5EL31H7W8/AWSCDK.EKSCluster.Delete.39114338-b340-46b9-b82a-fa7b17826f66 is not authorized to perform: eks:De
leteCluster on resource: arn:aws:eks:ap-northeast-1:aaaaaaaaaaaaa:cluster/sample

Logs: /aws/lambda/eks-dev-awscdkawseksCluster-OnEventHandler42BEBAE0-MGF8s2fAMvvq

The following are CloudWatch logs

INIT_START Runtime Version: nodejs:14.v28	Runtime Version ARN: arn:aws:lambda:ap-northeast-1::runtime:be4a32da6aeaa90583d476cb548ced79ae2c9aa7b14a45129198bbd13ea5822d
START RequestId: 510218c4-6150-47d9-9aa9-31fda0cab101 Version: $LATEST
2023-02-16T14:39:53.756Z	510218c4-6150-47d9-9aa9-31fda0cab101	INFO	
{
    "assumeRole": {
        "RoleArn": "arn:aws:iam::aaaaaaaaaaaaa:role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8",
        "RoleSessionName": "AWSCDK.EKSCluster.Delete.39114338-b340-46b9-b82a-fa7b17826f66"
    }
}

2023-02-16T14:39:53.847Z	510218c4-6150-47d9-9aa9-31fda0cab101	INFO	onDelete: deleting cluster sample
2023-02-16T14:39:54.428Z	510218c4-6150-47d9-9aa9-31fda0cab101	INFO	[AWS sts 200 0.539s 0 retries] assumeRole({
  RoleArn: 'arn:aws:iam::aaaaaaaaaaaaa:role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8',
  RoleSessionName: 'AWSCDK.EKSCluster.Delete.39114338-b340-46b9-b82a-fa7b17826f66'
})
2023-02-16T14:39:54.638Z	510218c4-6150-47d9-9aa9-31fda0cab101	INFO	[AWS eks 403 0.752s 0 retries] deleteCluster({ name: 'sample' })
2023-02-16T14:39:54.639Z	510218c4-6150-47d9-9aa9-31fda0cab101	ERROR	Invoke Error 	
{
    "errorType": "AccessDeniedException",
    "errorMessage": "User: arn:aws:sts::aaaaaaaaaaaaa:assumed-role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8/AWSCDK.EKSCluster.Delete.39114338-b340-46b9-b82a-fa7b17826f66 is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:ap-northeast-1:aaaaaaaaaaaaa:cluster/sample",
    "code": "AccessDeniedException",
    "message": "User: arn:aws:sts::aaaaaaaaaaaaa:assumed-role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8/AWSCDK.EKSCluster.Delete.39114338-b340-46b9-b82a-fa7b17826f66 is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:ap-northeast-1:aaaaaaaaaaaaa:cluster/sample",
    "time": "2023-02-16T14:39:54.637Z",
    "requestId": "4cdd26b0-1aa0-4fc2-9a10-156e806f5d75",
    "statusCode": 403,
    "retryable": false,
    "retryDelay": 25.084285757492264,
    "stack": [
        "AccessDeniedException: User: arn:aws:sts::aaaaaaaaaaaaa:assumed-role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8/AWSCDK.EKSCluster.Delete.39114338-b340-46b9-b82a-fa7b17826f66 is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:ap-northeast-1:aaaaaaaaaaaaa:cluster/sample",
        "    at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)",
        "    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8)",
        "    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)",
        "    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)",
        "    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)",
        "    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)"
    ]
}

END RequestId: 510218c4-6150-47d9-9aa9-31fda0cab101
REPORT RequestId: 510218c4-6150-47d9-9aa9-31fda0cab101	Duration: 910.85 ms	Billed Duration: 911 ms	Memory Size: 128 MB	Max Memory Used: 89 MB	Init Duration: 623.59 ms	
START RequestId: 6d5d5d74-a4be-4a03-b14b-69b0a70057c6 Version: $LATEST
2023-02-16T14:43:00.287Z	6d5d5d74-a4be-4a03-b14b-69b0a70057c6	INFO	
{
    "assumeRole": {
        "RoleArn": "arn:aws:iam::aaaaaaaaaaaaa:role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8",
        "RoleSessionName": "AWSCDK.EKSCluster.Delete.226eefe1-a557-4a70-b7c7-a61390d14d8f"
    }
}

2023-02-16T14:43:00.288Z	6d5d5d74-a4be-4a03-b14b-69b0a70057c6	INFO	onDelete: deleting cluster sample
2023-02-16T14:43:00.777Z	6d5d5d74-a4be-4a03-b14b-69b0a70057c6	INFO	[AWS sts 200 0.488s 0 retries] assumeRole({
  RoleArn: 'arn:aws:iam::aaaaaaaaaaaaa:role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8',
  RoleSessionName: 'AWSCDK.EKSCluster.Delete.226eefe1-a557-4a70-b7c7-a61390d14d8f'
})
2023-02-16T14:43:00.967Z	6d5d5d74-a4be-4a03-b14b-69b0a70057c6	INFO	[AWS eks 403 0.679s 0 retries] deleteCluster({ name: 'sample' })
2023-02-16T14:43:00.968Z	6d5d5d74-a4be-4a03-b14b-69b0a70057c6	ERROR	Invoke Error 	
{
    "errorType": "AccessDeniedException",
    "errorMessage": "User: arn:aws:sts::aaaaaaaaaaaaa:assumed-role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8/AWSCDK.EKSCluster.Delete.226eefe1-a557-4a70-b7c7-a61390d14d8f is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:ap-northeast-1:aaaaaaaaaaaaa:cluster/sample",
    "code": "AccessDeniedException",
    "message": "User: arn:aws:sts::aaaaaaaaaaaaa:assumed-role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8/AWSCDK.EKSCluster.Delete.226eefe1-a557-4a70-b7c7-a61390d14d8f is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:ap-northeast-1:aaaaaaaaaaaaa:cluster/sample",
    "time": "2023-02-16T14:43:00.967Z",
    "requestId": "32ffabd4-e25b-4fc7-ab35-1576aa68b2db",
    "statusCode": 403,
    "retryable": false,
    "retryDelay": 45.26429123253239,
    "stack": [
        "AccessDeniedException: User: arn:aws:sts::aaaaaaaaaaaaa:assumed-role/eks-dev-helloeksCreationRole0B71AB61-DS5EL31H7W8/AWSCDK.EKSCluster.Delete.226eefe1-a557-4a70-b7c7-a61390d14d8f is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:ap-northeast-1:aaaaaaaaaaaaa:cluster/sample",
        "    at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)",
        "    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8)",
        "    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)",
        "    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)",
        "    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)",
        "    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)"
    ]
}

END RequestId: 6d5d5d74-a4be-4a03-b14b-69b0a70057c6
REPORT RequestId: 6d5d5d74-a4be-4a03-b14b-69b0a70057c6	Duration: 940.47 ms	Billed Duration: 941 ms	Memory Size: 128 MB	Max Memory Used: 92 MB	

[pahud]

@watany-dev Thank you for the details. I am making it as p2 now but making EKS:DeleteCluster on * doesn't seem to be a good idea. I am leaving this issue open and any ideas are welcome here.

[maxjiang153]

@pahud Is it possible to add a specific tag to all EKS clusters created by the CDK stack? So adding EKS:DeleteCluster based on tag policy might be a good idea

[pahud]

related to #29282