Serverless Idempotency with Powertools for AWS Lambda
An example of using the Powertools for AWS Lambda to ensure our requests are idempotent, with an example written in TypeScript and the AWS CDK.
Preface
✔️ We discuss the need for idempotency.
✔️ We walk through the Powertools for AWS Lambda package.
✔️ We talk through the code repository in detail.
Introduction 👋🏽
In this article we are going to cover the need for idempotency in our solutions, and how we can achieve this using the Powertools for AWS Lambda package.
To help talk through the solution we will cover a fictitious doctor's surgery called ‘L.J Doctors’, where patients can book appointments online.
In the solution shown below, we can see that we have a basic ‘appointments’ API that has a Lambda function integration for the business logic, which ultimately persists the appointments into our DynamoDB table.
We need to ensure however that the client web application that the patient is using doesn’t try to book the same appointment more than once in a very short time period, i.e. the same patient for the same doctor on the same day/time. This can happen when the same request is made in quick succession by the client app (maybe a double click from a user or an error).
The code repository for this article can be found here:
We call this ‘idempotency’ which we will discuss in the next section.
👇 Before we go any further — please connect with me on LinkedIn for future blog posts and Serverless news https://www.linkedin.com/in/lee-james-gilmore/
What is Idempotency? 💜
The concept of idempotency ensures that operating multiple times with the same input parameters does not yield additional side effects. Idempotent operations consistently produce the same outcome upon multiple successful calls, rendering them safe for retries.
“In the context of AWS Lambda, idempotency refers to the property of an operation where executing the same operation multiple times has the same effect as executing it once.”
If we look in Wikipedia for the word ‘idempotence’ we see:
“Idempotence (UK: /ˌɪdɛmˈpoʊtəns/,[1] US: /ˈaɪdəm-/)[2] is the property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application”
In the context of AWS Lambda, idempotency refers to the property of an operation where executing the same operation multiple times has the same effect as executing it once. More specifically, when dealing with AWS Lambda functions, making a request or invoking a function is considered idempotent if performing the same operation with the same input parameters multiple times does not result in unintended side effects.
The idempotency key, a hash derived from either the entire event or a specific configured subset, encapsulates the input parameters. The results of invocations are serialized to JSON and stored in the chosen persistence storage layer. In our example, this persistence layer is Amazon DynamoDB.
An idempotency record serves as the data representation of a saved idempotent request in the preferred storage layer. It facilitates coordination to determine the idempotency status, validity, and expiration based on timestamps, among other factors.
This is where the Powertools for AWS Lambda (mainly known as Lambda Powertools) is our friend, as this package makes it easy for us to implement idempotency in our Lambda functions based on DynamoDB as the data persistence store for our idempotency records.
Our Example 💜
In our example, we want to ensure that if a patient tries to book the same appointment multiple times with the same input parameters within ten seconds we only save the appointment once, and on subsequent calls, we return the same success result.
The payload in our basic example looks like this
(note: we would usually use IDs for the doctors and patients but we wanted to make this example very obvious as to what things are):
// POST https://api.eu-west-1.amazonaws.com/prod/appointments/
{
"doctor": {
"name": "Dr. Smith",
"specialty": "Cardiology"
},
"patient": {
"name": "John Doe",
"dob": "1985-05-20"
},
"appointmentDateTime": "2023-05-10T14:00:00.000Z"
}The sequence diagram below shows how this works for successful calls:
We also want to ensure that we lock the process when we have concurrent requests in flight with the same payload, ensuring that we finish the first request, and send an error back to the client for the subsequent requests. This workflow is shown below:
Now that we know what idempotency is, and how we can utilise the Powertools for Lambda package, let’s now have a look through the basic code for this example.
Talking through key code 👨💻
Let’s start with one of the key pieces of the puzzle which is our custom L3 CDK construct for setting up the idempotency data store based on DynamoDB.
Custom Idempotency Construct
The following custom L3 construct allows us to reuse this across multiple projects and solutions if we publish it to an NPM repository so it can be installed easily from a common place:
import * as cdk from 'aws-cdk-lib';
import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';
import { Construct } from 'constructs';
interface IdempotencyTableProps
extends Pick<dynamodb.TableProps, 'removalPolicy' | 'tableName'> {
/**
* The table name
*/
tableName: string;
/**
* The removal policy for the table
*/
removalPolicy: cdk.RemovalPolicy;
}
type FixedDynamoDbTableProps = Omit<dynamodb.TableProps, 'removalPolicy'>;
export class IdempotencyTable extends Construct {
public readonly table: dynamodb.Table;
constructor(scope: Construct, id: string, props: IdempotencyTableProps) {
super(scope, id);
const fixedProps: FixedDynamoDbTableProps = {
billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
encryption: dynamodb.TableEncryption.AWS_MANAGED,
tableName: props.tableName,
pointInTimeRecovery: false,
contributorInsightsEnabled: true,
partitionKey: {
name: 'id', // this must be set for the powertools package to work
type: dynamodb.AttributeType.STRING,
},
timeToLiveAttribute: 'expiration', // this must be set for the powertools package to work
};
this.table = new dynamodb.Table(this, id + 'IdempTable', {
// fixed props
...fixedProps,
// custom props
...props,
});
}
}We can see from the code above that it sets up the DynamoDB table with a partition key of type string called ‘id’, and with a timeToLiveAttribute value of ‘expiration’, both of which need to match the Lambda Powertools package.
Stateful Stack
We can now utilise this custom construct in our stateful stack where we will set up the idempotency DynamoDB table, and our appointments DynamoDB table for persisting patient bookings:
import * as cdk from 'aws-cdk-lib';
import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';
import { Construct } from 'constructs';
import { IdempotencyTable } from '../custom-constructs';
import { config } from '../stateless/src/config';
export class DoctorsServiceStatefulStack extends cdk.Stack {
public readonly idempotencyTable: dynamodb.Table;
public readonly table: dynamodb.Table;
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
// create the idempotency table for our api using our custom construct
this.idempotencyTable = new IdempotencyTable(this, 'Idempotency', {
removalPolicy: cdk.RemovalPolicy.DESTROY,
tableName: config.get('idempotencytTableName'),
}).table;
// crteate the table which will store our appointments
this.table = new dynamodb.Table(this, 'DoctorsServiceTable', {
tableName: 'appointments',
partitionKey: {
name: 'id',
type: dynamodb.AttributeType.STRING,
},
});
this.table.applyRemovalPolicy(cdk.RemovalPolicy.DESTROY);
}
}Lambda Handler
We now create the Lambda handler for our POST endpoint as shown below:
import {
MetricUnits,
Metrics,
logMetrics,
} from '@aws-lambda-powertools/metrics';
import { Tracer, captureLambdaHandler } from '@aws-lambda-powertools/tracer';
import { errorHandler, logger, schemaValidator } from '@shared';
import {
APIGatewayProxyEvent,
APIGatewayProxyResult,
Context,
} from 'aws-lambda';
import { IdempotencyConfig } from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import { makeHandlerIdempotent } from '@aws-lambda-powertools/idempotency/middleware';
import { injectLambdaContext } from '@aws-lambda-powertools/logger';
import { config } from '@config';
import { Appointment } from '@dto/appointment';
import { ValidationError } from '@errors/validation-error';
import middy from '@middy/core';
import httpErrorHandler from '@middy/http-error-handler';
import { bookAppointmentUseCase } from '@use-cases/book-appointment';
import { schema } from './book-appointment.schema';
const tracer = new Tracer();
const metrics = new Metrics();
const persistenceStore = new DynamoDBPersistenceLayer({
tableName: config.get('idempotencytTableName'),
});
const idempotencyConfig = new IdempotencyConfig({
expiresAfterSeconds: 10, // how long to store the idempotency record
eventKeyJmesPath: 'body', // what the idempotency cache is based on
useLocalCache: false,
maxLocalCacheSize: 512,
});
export const bookAppointmentAdapter = async (
{ body }: APIGatewayProxyEvent,
context: Context
): Promise<APIGatewayProxyResult> => {
idempotencyConfig.registerLambdaContext(context);
try {
if (!body) throw new ValidationError('no payload body');
const appointment = JSON.parse(body) as Appointment;
schemaValidator(schema, appointment);
const created: Appointment = await bookAppointmentUseCase(appointment);
metrics.addMetric('SuccessfulBookAppointment', MetricUnits.Count, 1);
return {
statusCode: 201,
body: JSON.stringify(created),
};
} catch (error) {
let errorMessage = 'Unknown error';
if (error instanceof Error) errorMessage = error.message;
metrics.addMetric('BookAppointmentCreatedError', MetricUnits.Count, 1);
return errorHandler(error);
}
};
export const handler = middy()
.handler(bookAppointmentAdapter)
.use(injectLambdaContext(logger))
.use(captureLambdaHandler(tracer))
.use(logMetrics(metrics))
.use(
makeHandlerIdempotent({
persistenceStore,
config: idempotencyConfig,
})
)
.use(httpErrorHandler());We can see from the code above that we set both the persistenceStore and idempotencyConfig configurations that are then used in our makeHandlerIdempotent middy handler. In conjunction, they now make our handler idempotent!
Error Handler
One key addition to the Lambda handler code is our ‘error handler’ as show below:
import { logger } from '@shared';
import { APIGatewayProxyResult } from 'aws-lambda';
import createError from 'http-errors';
export function errorHandler(error: Error | unknown): APIGatewayProxyResult {
logger.error(`private error: ${error}`);
let errorMessage: string;
let statusCode: number;
if (error instanceof Error) {
switch (error.name) {
case 'ValidationError':
errorMessage = error.message;
statusCode = 400;
logger.error(errorMessage, {
errorName: errorMessage,
statusCode,
});
throw createError.BadRequest(errorMessage);
case 'ResourceNotFound':
errorMessage = error.message;
statusCode = 404;
logger.error(errorMessage, {
errorName: errorMessage,
statusCode,
});
throw createError.NotFound(errorMessage);
default:
errorMessage = 'An error has occurred';
statusCode = 500;
logger.error(errorMessage, {
errorName: errorMessage,
statusCode,
});
throw createError.InternalServerError(errorMessage);
}
} else {
errorMessage = 'An error has occurred';
statusCode = 500;
}
logger.error(errorMessage, {
errorName: errorMessage,
statusCode,
});
throw createError.InternalServerError(errorMessage);
}We can see from the code above that we check for the existence of certain errors and then throw the correct error (which includes the status code and error message). For example, when we have a ‘ResourceNotFound’ error we then throw a NotFound error using the ‘http-errors’ library:
throw createError.NotFound(errorMessage);This means that the middy middleware will automatically return the correct status code and message back to API Gateway in the correct shape, as we wrapped the Lambda handler with the httpErrorHandler middleware:
.use(httpErrorHandler());Trying it out 🎯
You can use the Postman collection in the repository to try out our code.
We can call it with the following payload:
// POST https://api.execute-api.eu-west-1.amazonaws.com/prod/appointments/
{
"doctor": {
"name": "Dr. Smith",
"specialty": "Cardiology"
},
"patient": {
"name": "John Doe",
"dob": "1985-05-20"
},
"appointmentDateTime": "2023-05-10T14:00:00.000Z"
}Which results in an idempotency record being added to our idempotency table (as shown below)
and our actual appointment record is stored in the appointments table.
If we now make the exact same request within ten seconds we will see that no other records are added, yet we still get the same result back on our successful call.
Wrapping up 👋🏽
I hope you enjoyed this article, and if you did then please feel free to share and feedback!
Please go and subscribe on my YouTube channel for similar content!
I would love to connect with you also on any of the following:
https://www.linkedin.com/in/lee-james-gilmore/
https://twitter.com/LeeJamesGilmore
If you enjoyed the posts please follow my profile Lee James Gilmore for further posts/series, and don’t forget to connect and say Hi 👋
Please also use the ‘clap’ feature at the bottom of the post if you enjoyed it! (You can clap more than once!!)
About me
“Hi, I’m Lee, an AWS Community Builder, Blogger, AWS certified cloud architect, and Global Head of Technology & Architecture based in the UK; currently working for City Electrical Factors (UK) & City Electric Supply (US), having worked primarily in full-stack JavaScript on AWS for the past 6 years.
I consider myself a serverless advocate with a love of all things AWS, innovation, software architecture, and technology.”
*** The information provided are my own personal views and I accept no responsibility for the use of the information. ***
You may also be interested in the following:
