(AwsCustomResource): fails to use latest SDK version

[frfavoreto]

Describe the bug

When setting AwsCustomResource with installLatestAwsSdk: true it fails to upgrade aws-sdk to latest version (at this moment v3.556.0) and fails back to the default (currently v3.515.0 in Lambda).

Examples with DynamoDB:

 INFO	Installing latest AWS SDK v3: @aws-sdk/client-dynamodb
 Task timed out after 120.10 seconds
        .
        .
        .
 INFO	Installing latest AWS SDK v3: @aws-sdk/client-dynamodb
 INFO	Failed to install latest AWS SDK v3. Falling back to pre-installed version. Error: Error: Cannot find module 'mnemonist/lru-cache'
 Require stack:
 - /tmp/node_modules/@aws-sdk/endpoint-cache/dist-cjs/index.js
 - /tmp/node_modules/@aws-sdk/middleware-endpoint-discovery/dist-cjs/index.js
 - /tmp/node_modules/@aws-sdk/client-dynamodb/dist-cjs/index.js
 - /var/task/index.js
 - /var/runtime/index.mjs
          .
          .

and SSM modules:

INFO	Installing latest AWS SDK v3: @aws-sdk/client-ssm
Task timed out after 120.16 seconds 
          .
          .
 INFO	Installing latest AWS SDK v3: @aws-sdk/client-ssm
 INFO	Failed to install latest AWS SDK v3. Falling back to pre-installed version. Error: Error: Cannot find module '@smithy/shared-ini-file-loader'
Require stack:
- /tmp/node_modules/@smithy/node-config-provider/dist-cjs/index.js
- /tmp/node_modules/@smithy/middleware-endpoint/dist-cjs/adaptors/getEndpointFromConfig.js
- /tmp/node_modules/@smithy/middleware-endpoint/dist-cjs/index.js
- /tmp/node_modules/@smithy/core/dist-cjs/index.js
- /tmp/node_modules/@aws-sdk/client-ssm/dist-cjs/index.js
- /var/task/index.js
- /var/runtime/index.mjs
          .
          . 

The custom resource eventually succeeds, after failing back to the default aws-sdk.

Expected Behavior

Be able to update the Lambda Nodejs18 runtime with the latest SDKv3 version.

Current Behavior

Unable to retrieve and upgrade Lambda Nodejs18 for SDKv3 latest version

Reproduction Steps

Deploy a CDK App with a Custom Resource that has installLatestAwsSdk: true:

const myTable = new dynamodb.Table(this, 'myTable', {
      partitionKey: {
        name: 'id',
        type: dynamodb.AttributeType.STRING,
      },
      removalPolicy: cdk.RemovalPolicy.DESTROY,
    });

const myCustomResource = new cr.AwsCustomResource(this, 'myCR', {​​
      policy: cr.AwsCustomResourcePolicy.fromSdkCalls({​​
          resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE,
      }​​),
      installLatestAwsSdk: true,
      onCreate: {​​
          service: 'DynamoDB',
          action: 'PutItem',
          parameters: {​​
              Item: {​​
                id: {"S": 'test-value'}
              }​​,
              TableName: myTable.tableName,
          }​​,
          physicalResourceId: cr.PhysicalResourceId.of('myCRphysicalResourceID'),
      }​​,
      onUpdate: {​​
          service: 'DynamoDB',
          action: 'PutItem',
          parameters: {​​
            service: 'DynamoDB',
            action: 'PutItem',
            parameters: {​​
                Item: {​​
                  id: {"S": 'test-value'}
                }​​,
                TableName: myTable.tableName,
            }​​,
          physicalResourceId: cr.PhysicalResourceId.of('myCRphysicalResourceID'),
      }​​,
      },
  }​​);

Check the underlying Lambda Function's logs to see the messages above.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.137.0

Framework Version

No response

Node.js Version

18

OS

Mac

Language

TypeScript

Language Version

No response

Other information

No response

[khushail]

@frfavoreto , thanks for reaching out. Team is already Tracking this and working on it.

[dtczest]

I'm also running into this, but in my case, the custom resource never succeeds.

[jakekarnes42]

Please note that this is currently causing deployments to fail for custom resources that that use installLatestAwsSdk: true. The resources are timing out and/or failing when the lambda falls back to using the older SDK. Some resources may succeed upon retrying, but if you have multiple custom resources, then it's likely that at least one of them will fail during each deployment.

[colifran]

@frfavoreto @jakekarnes42 What version of the CDK are you using and do you know what version you started encountering this in?

[jakekarnes42]

@colifran Today I upgraded from CDK 2.131.0 to 2.138.0 (the current latest version) and that's when the issue began. Previously successful deployments began to fail. Investigating the failures came to exact conclusion shared by @frfavoreto in the original issue description.

Each deployment contains about 15 custom resources which would fail intermittently. Upon reviewing the Lambda logs, it appears that sometimes the Lambda would succeed after falling back to the default SDK. Infrequently, it would timeout before successfully falling back. That could cause the Custom Resource update to fail, and cascade the failure to the rest of the deployment. Since I'm deploying multiple custom resources, I found that at least one would fail on each deployment attempt.

I rolled back to CDK 2.131.0 and the issue is no longer present. I'm back to successful deployments. This appears to be regression sometime between those two versions.

I hope this helps and thanks for the quick support!

[colifran]

@jakekarnes42 Thanks for the clarification. What is strange is that I have also gone back to 2.131.0, but I'm still getting the time out when trying to install the latest SDK version. I've looked through some of our recent changes and I'm not seeing anything that would make me think this is something on the CDK side. I'm wondering if this could be an SDK related issue? We will continue to investigate / monitor!