CustomResource Table Create Fails On First Attempt When Backups Are Enabled

by ADMIN 76 views

Introduction

When working with Amazon Web Services (AWS) and the AWS CDK, creating custom resources can be a complex process. However, when backups are enabled on the backend tables, creating a new table using the CustomResource can fail on the first attempt. This issue can be frustrating, especially when trying to deploy a new application to production. In this article, we will explore the problem, provide a detailed explanation of the issue, and offer guidance on how to resolve it.

Environment Information

Before we dive into the issue, let's take a look at the environment information:

System Information

  • OS: Linux 5.15 Ubuntu 24.04.2 LTS 24.04.2 LTS (Noble Numbat)
  • CPU: (12) arm64 unknown
  • Memory: 28.50 GB / 31.17 GB
  • Shell: /bin/bash

Binaries

  • Node: 22.15.0 - ~/.local/share/mise/installs/node/22.15.0/bin/node
  • Yarn: undefined - undefined
  • npm: 11.3.0 - ~/.local/share/mise/installs/node/22.15.0/bin/npm
  • pnpm: undefined - undefined

NPM Packages

  • @aws-amplify/auth-construct: 1.6.0
  • @aws-amplify/backend: 1.14.0
  • @aws-amplify/backend-auth: 1.5.0
  • @aws-amplify/backend-cli: 1.4.8
  • @aws-amplify/backend-data: 1.4.0
  • @aws-amplify/backend-deployer: 1.1.15
  • @aws-amplify/backend-function: 1.12.1
  • @aws-amplify/backend-output-schemas: 1.4.0
  • @aws-amplify/backend-output-storage: 1.1.4
  • @aws-amplify/backend-secret: 1.1.5
  • @aws-amplify/backend-storage: 1.2.4
  • @aws-amplify/cli-core: 1.2.3
  • @aws-amplify/client-config: 1.5.5
  • @aws-amplify/deployed-backend-client: 1.5.0
  • @aws-amplify/form-generator: 1.0.3
  • @aws-amplify/model-generator: 1.0.12
  • @aws-amplify/platform-core: 1.6.0
  • @aws-amplify/plugin-types: 1.8.0
  • @aws-amplify/sandbox: 1.2.10
  • @aws-amplify/schema-generator: 1.2.7
  • aws-amplify: 6.12.2
  • aws-cdk: 2.177.0
  • aws-cdk-lib: 2.177.0
  • typescript: 5.5.4

Describe the Bug

We have enabled Point-in-Time Recovery (PITR) on our backend tables (table.pointInTimeRecoveryEnabled = true;), however when new tables are added, the resource regularly fails to create with the following error:

Received response status [FAILED] from custom resource. Message returned: Backups are being enabled for the table: Todo-...-NONE. Please retry later (RequestId: ...)

This issue has come up during development, but a redeploy usually succeeds. We're currently trying to make our first deployment from dev to production where we're now creating several tables for the first time. We've re-tried the deployment several times and a different table seems to fail with this error each time. It seems that there's something in the CustomResource that isn't playing nice with PITR.

Reproduction Steps

To reproduce this issue, follow these steps:

  1. Deploy a new Model with PITR enabled.

Troubleshooting

To troubleshoot this issue, we can try the following:

  1. Check the AWS CDK version: Ensure that you are using the latest version of the AWS CDK.
  2. Verify the PITR configuration: Double-check that PITR is enabled on the backend tables.
  3. Check the CustomResource configuration: Review the CustomResource configuration to ensure that it is correctly set up.
  4. Try redeploying: Redeploy the application to see if the issue persists.
  5. Break down the stack: If the issue persists, try breaking down the stack and deploying one table at a time.

Conclusion

In conclusion, creating a new table using the CustomResource can fail on the first attempt when backups are enabled on the backend tables. This issue can be frustrating, especially when trying to deploy a new application to production. By following the troubleshooting steps outlined above, you should be able to resolve this issue and successfully deploy your application.

Additional Resources

For more information on the AWS CDK and CustomResources, please refer to the following resources:

Introduction

In our previous article, we explored the issue of CustomResource table create failing on the first attempt when backups are enabled on the backend tables. We provided a detailed explanation of the issue and offered guidance on how to resolve it. In this article, we will answer some frequently asked questions related to this issue.

Q: What is the cause of the CustomResource table create failure?

A: The cause of the CustomResource table create failure is due to the fact that the CustomResource is trying to enable backups on the table at the same time as it is creating the table. This can cause a conflict and result in the failure of the table creation.

Q: How can I resolve this issue?

A: To resolve this issue, you can try the following:

  1. Check the AWS CDK version: Ensure that you are using the latest version of the AWS CDK.
  2. Verify the PITR configuration: Double-check that PITR is enabled on the backend tables.
  3. Check the CustomResource configuration: Review the CustomResource configuration to ensure that it is correctly set up.
  4. Try redeploying: Redeploy the application to see if the issue persists.
  5. Break down the stack: If the issue persists, try breaking down the stack and deploying one table at a time.

Q: What is the difference between PITR and backups?

A: PITR (Point-in-Time Recovery) and backups are two different concepts in AWS. PITR allows you to restore your database to a specific point in time, while backups allow you to restore your database to a specific point in time and also provide a snapshot of your database at that point in time.

Q: Can I enable PITR on my existing tables?

A: Yes, you can enable PITR on your existing tables. However, you will need to follow the instructions provided by AWS to do so.

Q: What are the benefits of using CustomResources?

A: CustomResources provide a way to extend the functionality of the AWS CDK and allow you to create custom resources that are not provided by the AWS CDK. They also provide a way to create resources that are specific to your application and cannot be created using the AWS CDK.

Q: Can I use CustomResources with other AWS services?

A: Yes, you can use CustomResources with other AWS services. However, you will need to ensure that the CustomResource is compatible with the other AWS service.

Q: What are some common issues that can occur when using CustomResources?

A: Some common issues that can occur when using CustomResources include:

  • CustomResource table create failure: This is the issue that we are discussing in this article.
  • CustomResource update failure: This can occur when trying to update a CustomResource that is already created.
  • CustomResource deletion failure: This can occur when trying to delete a CustomResource that is already created.

Conclusion

In conclusion, CustomResource table create failure on the first attempt when backups are enabled on the backend tables is a common issue that occur when using the AWS CDK. By following the troubleshooting steps outlined in this article and the previous article, you should be able to resolve this issue and successfully deploy your application.

Additional Resources

For more information on the AWS CDK and CustomResources, please refer to the following resources: