Serverless Custom Resources 🚀
How can we use Custom Resources on AWS to extend our deployment and resource options without any restrictions, including non-AWS services and seeding data, all within the same IaC? This article includes visuals and code examples written in TypeScript with the CDK.
Introduction
These days all of your infrastructure should be Infrastructure as Code (IaC), commonly using tools such as the AWS CDK, Serverless Framework or Terraform. There are however times when we need to create non-AWS resources, and ideally they would be along side the rest of your architecture code.
The code for this article can be found here: https://github.com/leegilmorecode/serverless-custom-resources
“There are however times when we need to create non-AWS resources, and ideally they would be along side the rest of your architecture code.”
Terraform has the notion of Providers to extend it for this reason, and any CloudFormation backed IaC tools also have the ability through the use of either AWS Custom Resources, or through the Public CloudFormation Registry (although this is fairly limited currently).
The power comes from the fact that Custom Resources within your CloudFormation can be backed by Lambda, so you can essentially do anything at this point; including running smoke tests, populating dev environments with test data, creating database tables etc etc — this opens you up to do anything as part of your Serverless deployments!
🛑 Note: although this is a very powerful tool, there are also some caveats that you need to think about so you don’t break your stacks. The following article covers this in great detail:
“so you can essentially do anything at this point; including running smoke tests, populating dev environments with test data, creating database tables etc etc — this opens you up to do anything as part of your Serverless deployments!”
What is a Custom Resource? 💭
A Custom resource is essentially a web-hook which can be fired when working with CloudFormation (or tools that generate CloudFormation such as the CDK and Serverless Framework), that allows you to create non AWS services through invoking custom lambdas, or performing tasks such as:
- You may need to create Pager Duty alerts as part of your deployments.
- You could utilise Auth0 over Cognito, and provision these Auth0 resources as part of your stack.
- You may want to integrate with GitHub or Bitbucket over AWS Code Commit.
- You may want to use MongoDB Atlas rather than DocumentDB.
- There may be AWS resources that CloudFormation does not support yet.
- You may want to use this as a hook to add seed data into ephemeral environments.
Where does this become massively useful? 🔥
Essentially Custom Resources give you the ability to jump out to an associated Lambda as part of your CloudFormation template when extensions don’t exist in the public registry, to do whatever you want… keeping all of your AWS and non-AWS resources as IaC in the same stacks.
Let’s see where this becomes so powerful!
What are we building? 🏗️
We are going to build a very basic API which returns a list of available coffee shops for our fictitious company ‘Lee James Gilmore Coffee Shops’.
As part of the deployment we will utilise a custom resource which creates the relevant seed data in DynamoDB i.e. our locations based on an ID and a Location Name.
“As part of the deployment we will utilise a custom resource which creates the relevant seed data in DynamoDB i.e. our locations based on an ID and a Location Name.”
This is a fantastic tool to ensure seed data is added to non-production environments before running deterministic pipeline testing (smoke/unit/integration etc), but not running this in production for example.
“This is a fantastic tool to ensure seed data is added to non-production environments before running deterministic pipeline testing (smoke/unit/integration etc)”
The Serverless architecture is shown below using the CDK:
Let’s get started in the next section.
Getting Started! ✔️
To get started, clone the following repo with the following git command:
git clone https://github.com/leegilmorecode/serverless-custom-resourcesThis will pull down the example code to your local machine.
Deploying the solution! 👨💻
🛑 Note: Running the following commands will incur charges on your AWS account.
To deploy the solution we simply run npm run deploy
🛑 Note: Remember to tear down the stack when you are finished so you won’t continue to be charged, by using ‘
npm run remove’.
Testing the solution 🎯
Now that we have the solution deployed, you can simply call your endpoint which will return a list of coffee shop locations as seen below, as the data was seeded as part of the deployment!
Talking through key code 👊
OK, so we understand the architecture and premise of AWS Custom Resources, but let’s now talk through some of the key code below.
Setting up the CDK side
From an infrastructure perspective it is very easy to set up an AWS Custom Resource using the CDK as shown below:
We can see that we simply create a Provider object which has the following properties:
onEventHandler: The Lambda handler which is going to perform the work.logRetention: How long we want to keep the CloudWatch Logs for.providerFunctionName: The provider function name.
We then create a new Custom Resource based on the Provider above. This takes the following properties:
serviceToken: The serviceToken of the Provider object.properties: An object of properties that we want to pass to the Custom Resource.
Populate Locations Lambda
The Lambda which backs the AWS Custom Resource looks like the following:
We can see that from the code above that we:
- Check the event
RequestTypeand drop into the switch statement depending on whether or not it is a ‘Create’, ‘Update’ or ‘Delete’. - For the Create and Update we run a function called
seedDatawhich pushes the locations to DynamoDB. This essentially does anupserton the items in the collections (overwriting the existing items). - We swallow a ‘
Delete’ as the DynamoDB Table would also be deleted in this regard.
We can see the event structure passed through to the Lambda from the Custom Resource here for reference (including the table name which is used in the DynamoDB call):
On deployment of the CDK application using npm run deploy we invoke the Lambda through the Custom Resource Provider which seeds our location data to DynamoDB!
Summary
I hope you found that useful as an example of using Custom Resources to do some cool things in your stacks!
Go and subscribe to my Enterprise Serverless Newsletter here for more of the same content:
Wrapping up 👋
Please go and subscribe on my YouTube channel for similar content!
I would love to connect with you also on any of the following:
https://www.linkedin.com/in/lee-james-gilmore/
https://twitter.com/LeeJamesGilmore
If you enjoyed the posts please follow my profile Lee James Gilmore for further posts/series, and don’t forget to connect and say Hi 👋
Please also use the ‘clap’ feature at the bottom of the post if you enjoyed it! (You can clap more than once!!)
About me
“Hi, I’m Lee, an AWS Community Builder, Blogger, AWS certified cloud architect and Global Serverless Architect based in the UK; currently working for City Electrical Factors & City Electric Supply, having worked primarily in full-stack JavaScript on AWS for the past 6 years.
I consider myself a serverless advocate with a love of all things AWS, innovation, software architecture and technology.”
*** The information provided are my own personal views and I accept no responsibility on the use of the information. ***
You may also be interested in the following:
