Photo by Bram Van Oost on Unsplash

Serverless OpenAPI & Amazon API Gateway with the AWS CDK — Part 2

How we can utilise the “OpenAPI code first approach” alongside Amazon API Gateway in our AWS CDK TypeScript solutions.

Serverless Advocate
17 min readJun 20, 2023

--

Preface

✔️ We discuss using the OpenAPI code approach. 🔩
✔️ We discuss publishing/hosting our API spec on Amazon CloudFront. ☁️
️✔️ We create a mock version of the service for other teams. 🧑🏿‍🤝‍🧑🏿
✔️ We walk through the code. 🧑🏽‍💻

Part 1OpenAPI spec first approach.
Part 2
— OpenAPI code first approach.

Introduction

This series is going to discuss the various ways we can create our API Gateways on AWS with the AWS CDK and TypeScript; and different approaches to importing and exporting our OpenAPI specifications. We will cover the advantages and disadvantages of each approach throughout the series.

✔️ Part 1 of the series covered the “OpenAPI spec first approach” to development, whereby our Amazon API Gateway and the associated integrations to Lambda functions etc all happen via the uploaded spec in JSON format.

✔️ Part 2 of the series (this part) covers the “OpenAPI code first approach” to development, whereby our Amazon API Gateway and the associated integrations to Lambda functions etc all happen via AWS CDK code.

Example of what we will build in this article

The diagram above shows:

  1. The engineering team create the API Gateway through code rather than through an OpenAPI spec.
  2. The CDK stack is deployed via the cdk deploy command which uses CloudFormation under the hood.
  3. The API Gateway is created with the relevant backing Lambda functions as integrations.
  4. A Custom Resource uses the AWS SDK and the API Gateway Export command to export the newly deployed API Gateway as an OpenAPI spec file, and this is then persisted to the S3 bucket.
  5. As part of the CDK deploy we also push the Swagger UI files to the same S3 bucket which allows us to host the spec in a visual way.
  6. We put Amazon CloudFront in front of the private S3 bucket and use origin access identity to allow the two services to work in tandem. This allows other teams to access our API spec for the Movies domain service.

In the next section lets work through the code.

👇 Before we go any further — please connect with me on LinkedIn for future blog posts and Serverless news https://www.linkedin.com/in/lee-james-gilmore/

Let’s talk through the code 🎤

Let’s now walk through the code and setup, starting with the mock service.

Our Mock API Service ✔️

In the previous article we deployed the Mock API using the OpenAPI spec first approach, whereas in this approach we generate it all through code.

Note: We cover in Part 1 why we may want to create a mock version of our API to aid development or help other teams with their e2e/integration tests.

We start by creating a new apigw.RestApi resource and then create the resources for our API i.e. ‘api/v1/movies’:

...
// create our mock api
const moviesMockApi = new apigw.RestApi(this, 'MoviesMockApi', {
description: 'A Mock API for movies (Code)',
deploy: true,
endpointTypes: [apigw.EndpointType.REGIONAL],
deployOptions: {
stageName: 'api',
loggingLevel: apigw.MethodLoggingLevel.INFO,
},
});
...
// add our mock service resources
const moviesMockRootVersion: apigw.Resource =
moviesMockApi.root.addResource(version);

const moviesMockResource: apigw.Resource =
moviesMockRootVersion.addResource('movies');
const movieMockResource: apigw.Resource =
moviesMockResource.addResource('{id}');
...

We then integrate the mock methods with the apigw.MockIntergration, with the code below showing a GET on the /movies/ resource as an example:

// add our methods to the mock api
moviesMockResource.addMethod(
'GET',
new apigw.MockIntegration({
passthroughBehavior: apigw.PassthroughBehavior.WHEN_NO_MATCH,
requestTemplates: {
'application/json': '{ "statusCode": 200 }',
},
integrationResponses: [
{
statusCode: '200',
responseTemplates: {
'application/json': JSON.stringify(movies),
},
},
],
}),
{
methodResponses: [
{
statusCode: '200',
},
{
statusCode: '400',
},
],
}
);

We can see from the code above that we return a 200 status code and the response template returns the mock movies we have in the movie-service-code/src/data/movies.ts file:

// note: these are hard coded movies for now
// for our basic example i.e. no datastores
export const movies: Movies = [
{
id: '180aba94-88e8-4f83-9dcb-0dd437d93ff8',
title: 'movie one',
year: '2019',
rating: Rating.EIGHTEEN,
},
{
id: 'cbbccaba-3ab9-4af0-a157-0e5cfd654e20',
title: 'movie two',
year: '2020',
rating: Rating.PG,
},
{
id: '31adad13-c876-4231-b8b1-cb45926ba1c8',
title: 'movie three',
year: '2007',
rating: Rating.U,
},
];

This is slightly different for the POST request of a new movie, which works in the same way as the OpenApi spec approach, but this time it is in code rather than in the JSON OpenAPI spec definition:

...
moviesMockResource.addMethod(
'POST',
new apigw.MockIntegration({
passthroughBehavior: apigw.PassthroughBehavior.WHEN_NO_MATCH,
requestTemplates: {
'application/json':
'#set($context.requestOverride.path.body = $input.body)\n{\n "statusCode": 200,\n}',
},
integrationResponses: [
{
statusCode: '200',
responseTemplates: {
'application/json':
'#set($body = $util.parseJson($context.requestOverride.path.body))\n{"id": "c4887ba4-0782-471c-bddc-af50265c96b9",\n "title": "$body.title",\n "rating": "$body.rating",\n "year": "$body.year"\n}',
},
},
],
}),
{
methodResponses: [
{
statusCode: '200',
},
{
statusCode: '400',
},
],
}
);
...

You can see from the code above that we return the same payload back to the consumer with a deterministic hardcoded ID, which can aid other teams in their own integration or e2e tests. There would be nothing stopping us using VTL to return a new UUID each time too.

Our Prod API Service ✔️

Now let’s cover the more interesting part which is our actual production REST API (non-mock). We start by creating a new apigw.RestApi as shown below:

// create the rest api through code integrations
const moviesApi: apigw.RestApi = new apigw.RestApi(this, 'MoviesApi', {
description: 'An API for movies (Code)',
endpointTypes: [apigw.EndpointType.REGIONAL],
retainDeployments: false,
deploy: true,
deployOptions: {
stageName: 'api',
loggingLevel: apigw.MethodLoggingLevel.INFO,
},
});

Another thing which we need to setup in the code itself since we are not using the OpenAPI spec first approach is the various models. An example of the Movie model is shown below:

// add our api models
const movieModel = new apigw.Model(this, 'Movie', {
restApi: moviesApi,
contentType: 'application/json',
modelName: 'Movie',
schema: {
schema: apigw.JsonSchemaVersion.DRAFT4,
type: apigw.JsonSchemaType.OBJECT,
required: ['id', 'rating', 'title', 'year'],
properties: {
id: {
type: apigw.JsonSchemaType.STRING,
description: 'The movie ID (numeric characters only)',
pattern:
'^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-4[0-9a-fA-F]{3}-[89aAbB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}$',
},
year: {
type: apigw.JsonSchemaType.STRING,
description: 'The release year of the movie',
pattern: '^\\d{4}$',
},
rating: {
type: apigw.JsonSchemaType.STRING,
description: 'The rating of the movie',
pattern: '^[UPG]|1[258]$',
enum: ['U', 'PG', '12', '15', '18'],
},
title: {
type: apigw.JsonSchemaType.STRING,
maxLength: 100,
minLength: 1,
pattern: '^[a-zA-Z0-9 ]*$',
description:
'The movie title (alphanumeric characters and spaces only)',
},
},
},
});

One key thing to note here is due to the specific API Gateway types for the model (JsonSchemaVersion, JsonSchemaType etc), it doesn’t allow us to reuse the schemas natively like we did in Part 1 i.e. utilising JSON based schemas:

schema: apigw.JsonSchemaVersion.DRAFT4,
...
type: apigw.JsonSchemaType.STRING,

We could probably work around this with some TypeScript magic, but it’s yet more changes we need to make to get this working smoothly i.e. over and above the simple option of the OpenAPI spec like we did in Part 1.

Note: Ideally we want to be able to use that model schema above in any of our code where we use a movie model. In a DDD approach this could be in a Use Case (business logic essentially), but we don’t want to leak technology or framework specific code into this schema — i.e. like the apigw.JsonSchema types! We want to keep the Movie model agnostic of technology or frameworks.

One other key aspect to this is the actual Lambda Integrations themselves, as in the OpenAPI spec first approach this is done in the JSON definition itself, whereas in this code first approach we need to add the following code (example of list movies):

// add our methods to the prod api
moviesResource.addMethod(
'GET',
new apigw.LambdaIntegration(listMoviesLambda, {
proxy: true,
}),
{
requestValidator,
methodResponses: [
{
statusCode: '200',
responseModels: {
'application/json': arrayOfMoviesModel,
},
},
{
statusCode: '400',
responseModels: {
'application/json': errorResponseModel,
},
},
],
}
);

This is an example purely for the GET method on the /movies/ resource, which we can see uses the arrayOfMoviesModel for the return schema with a 200 status code, and an errorResponseModel when the lambda function creates a 400 response.

You will also see that we state that this integration uses a requestValidator, which again we need to define in code explicitly (unlike the OpenAPI spec first approach we used previously where it was part of the JSON definition):

// create our request validator
const requestValidator = new apigw.RequestValidator(
this,
'RequestValidator',
{
requestValidatorName: 'RequestValidator',
restApi: moviesApi,
validateRequestBody: true,
validateRequestParameters: true,
}
);

The code above ensures that API Gateway will perform basic validation for the request body and request parameters (the link below provides more information on this):

Documentation Versions ✔️

Documentation versions are again very manual with the code first approach, and in the OpenAPI spec first approach we get this automatically generated for us!

“In reality do people use the documentation version feature these days?”

In reality do people use the documentation version feature these days? I would assume not when we have detailed OpenAPI specs on offer as an industry standard.

That being said, I wanted to show how you could create the Documentation parts through code for brevity — starting with the DocumentationVersion:

// we create some documentation which will be pushed to apigw
const documentation = new apigw.CfnDocumentationVersion(
this,
'ApiDocumentation' + version, // this ensures we get a new version each time
{
restApiId: moviesApi.restApiId,
documentationVersion: version,
description: 'api schema',
}
);
documentation.applyRemovalPolicy(RemovalPolicy.RETAIN);

Note: I haven’t done this for the full API, and have only showed examples of how to do this, as in reality I don’t think many people actually use this feature.

The next step is to create the DocumentationParts for the DocumentationVersion above, starting with the API:

new apigw.CfnDocumentationPart(this, 'MoviesApiPart', {
location: {
type: 'API',
},
properties: JSON.stringify({
info: {
description: 'An API for movies (Code)',
},
}),
restApiId: moviesApi.restApiId,
});

And we then start creating these for our methods, models etc too:

new apigw.CfnDocumentationPart(this, 'ListMoviesDocPart', {
location: {
method: 'GET',
path: `/${version}/movies`,
type: 'METHOD',
},
properties: JSON.stringify({
summary: 'Get all movies',
}),
restApiId: moviesApi.restApiId,
});

As I say, in reality I don’t ‘think’ teams (in my experience) actually use this feature, but it is kinda neat that we get this out of the box with the OpenAPI spec first approach!

What does ChatGPT have to say about the differences between OpenAPI specs and Documentation Versions on AWS:

🤖 “Regarding OpenAPI documentation, API Gateway supports the OpenAPI Specification (formerly known as Swagger) to define and document APIs. OpenAPI is a widely adopted industry standard for describing RESTful APIs. API Gateway leverages OpenAPI to generate documentation automatically based on the API configuration.

While the OpenAPI documentation generated by API Gateway provides a basic level of documentation, the Documentation Version and Documentation Parts in API Gateway offer additional features and flexibility to manage, version, and customise your API documentation within the API Gateway service.”

Deploying the OpenAPI spec for other teams ✔️

Now let’s discuss the important part of actually exporting our deployed API as an OpenAPI specification document (JSON), and pushing this to an S3 bucket fronted by CloudFront for our consumers.

The blue circle shows where this part of the article focuses

We can see from the diagram above that we use a Custom Resource as part of the CDK deployment to export the latest API Gateway definition that has just been deployed as an OpenAPI spec, and to save that in an S3 bucket along with the Swagger UI distribution files.

We start by creating a custom resource and a provider as shown in the code below:

// create our custom resource which will pull down the generated
// openapi spec and push it to the s3 bucket with the correct name
const provider: cr.Provider = new cr.Provider(
this,
'PublishOpenApiSpecCustomResource',
{
onEventHandler: publishOpenApiSpecLambda,
logRetention: logs.RetentionDays.ONE_DAY,
providerFunctionName: `publish-openapi-${props.stageName}-cr-lambda`,
}
);

// use the custom resource provider
const customResource = new CustomResource(
this,
`OpenApiSpecCustomResource`,
{
serviceToken: provider.serviceToken,
removalPolicy: RemovalPolicy.DESTROY,
properties: {
version,
restApiId: moviesApi.restApiId,
bucket: bucket.bucketName,
stageName: 'api',
deploymentId: deployment.deploymentId,
// force the custom resource to run on every stack deploy so the openapi spec is always up to date
changeId: uuid(),
},
}
);

One key thing to note in the code above is that we pass in an arbitrary property that we don’t actually use called ‘changeId’ which forces the custom resource to run on every CDK deployment, as the value is different each time (that of a autogenerated random UUID).

This means that we will always export the latest API definition and push to the S3 bucket to ensure it is up to date!

Next, we ensure that the Lambda behind the custom resource can export the API from API Gateway, as well as having the permissions to get the current deployment:

// ensure the custom resource lambda can get the current rest api and export it as openapi json
publishOpenApiSpecLambda.addToRolePolicy(
new PolicyStatement({
resources: [
`arn:aws:apigateway:eu-west-1::/restapis/${moviesApi.restApiId}/stages/api/exports/*`,
`arn:aws:apigateway:eu-west-1::/restapis/${moviesApi.restApiId}/stages/api`,
],
actions: ['apigateway:GET'],
})
);
bucket.grantReadWrite(publishOpenApiSpecLambda);

customResource.node.addDependency(moviesApi);

Note: we also ensure that the the custom resource is a dependency of the deployment which forces this to run after it within CloudFormation.

If we now focus on the code within the movie-service-code/src/handlers/publish-openapi-spec/publish-openapi-spec.ts lambda function which sits behind the custom resource, we can see that it performs the following functionality:

1. Ensures that the API deployment is complete before trying to export the latest OpenAPI specification document:

// ensure that the deployment of the stage and restapi
// is complete before exporting the version of the openapi doc
async function ensureDeploymentCompleted(
restApiId: string,
deploymentId: string,
stageName: string
): Promise<void> {
let count = 0;
const maxRetries = 10;
const delayInSeconds = 10;

const getStageParams: GetStageCommandInput = {
restApiId: restApiId,
stageName: stageName,
};

while (count < maxRetries) {
const getStageCommand: GetStageCommand = new GetStageCommand(
getStageParams
);

try {
const response: GetStageCommandOutput = await apigwClient.send(
getStageCommand
);
// get the associated deploymentId for the restApi and stage
const associatedDeploymentId = response.deploymentId;

if (associatedDeploymentId === deploymentId) {
console.log(
`Stage '${stageName}' is associated with the desired deployment ID of ${associatedDeploymentId}`
);
return; // Exit the function if the stage is associated with the desired deployment ID
} else {
console.log(
`Stage '${stageName}' is not associated with the desired deployment ID of ${associatedDeploymentId}. Retrying in ${delayInSeconds} seconds...`
);
await delay(delayInSeconds * 1000); // Delay for the specified time in milliseconds
}
} catch (error) {
console.error('Error retrieving stage details: ', error);
}

count++;
}
}

This is fairly belts and braces stuff, but I wanted to ensure that the Custom Resource Lambda function is actually going to export the correct version based on the Rest API ID, the Deployment ID, and the Stage.

2. Next, we export the OpenAPI documentation now that we know that the correct API version for a given deployment and stage has successfully deployed:

// export the openapi spec document from the api gateway with backoff and retry
async function generateOpenApiSpec(
restApiId: string,
stageName: string
): Promise<GetExportCommandOutput> {
let retryCount = 0;
const maxRetries = 5;
const delayInSeconds = 10;

const getExportRequest: GetExportRequest = {
restApiId,
stageName,
exportType: 'oas30',
parameters: {},
accepts: 'application/json',
};
const getExportCommand: GetExportCommand = new GetExportCommand(
getExportRequest
);

while (retryCount < maxRetries) {
try {
const response: GetExportCommandOutput = await apigwClient.send(
getExportCommand
);
console.log(`Request statuscode: ${response.$metadata.httpStatusCode}`);

if (response.$metadata.httpStatusCode !== 200) {
console.log(
`Error response metadata: ${JSON.stringify(response.$metadata)}`
);
throw new Error('Unsuccessful attempt at downloading the schema');
}
return response;
} catch (error) {
console.error(`Retry ${retryCount + 1}: ${error}`);
retryCount++;
await delay(delayInSeconds * 1000); // Delay in milliseconds
}
}

throw new Error(
`Exceeded maximum retries (${maxRetries}) for getExportCommand.`
);
}

Again this is fairly belts and braces stuff here, but I want to ensure that this works successfully when other teams will be relying on this specification being up to date.

3. Finally, now that we have exported the OpenAPI spec into memory, we push the file to the S3 bucket so consumers of the documentation can read it (via CloudFront of course):

// write the openapi spec to the specified s3 bucket and then check that
// it was successfully saved by retrieving it once written.
async function saveOpenApiSpecToBucket(
bucket: string,
fileBody: Uint8Array | undefined,
fileName: string
): Promise<void> {
if (!fileBody) throw new Error('no openapi spec body');

let retryCount = 0;
const delayInSeconds = 10;
const maxRetries = 5;

const input: PutObjectCommandInput = {
Body: fileBody,
ContentType: 'application/json',
Bucket: bucket,
Key: fileName,
};

const command = new PutObjectCommand(input);

while (retryCount < maxRetries) {
try {
const response: PutObjectCommandOutput = await s3Client.send(command);

console.log(
`Writing to ${command.input.Bucket} for file ${command.input.Key}`
);
console.log(`S3 response: ${JSON.stringify(response)}`);

// Check if the file is actually persisted by performing a getObject operation
const getObjectInput: GetObjectCommandInput = {
Bucket: bucket,
Key: fileName,
};
const getObjectResponse: GetObjectCommandOutput = await s3Client.send(
new GetObjectCommand(getObjectInput)
);

if (getObjectResponse.Body) {
console.log(
`File ${fileName} successfully persisted in the S3 bucket.`
);
break; // Exit the loop if the file is successfully written and retrieved
} else {
console.log(
`File ${fileName} is not yet persisted in the S3 bucket. Retry attempt ${
retryCount + 1
} in 10 seconds...`
);
}
} catch (error) {
console.error(error);
retryCount++;
console.log(
`Failed to write file. Retry attempt ${retryCount} in 10 seconds...`
);
}

await delay(delayInSeconds * 1000); // Delay in milliseconds
}

if (retryCount === 10) {
console.log(
`File could not be written and persisted after ${maxRetries} attempts.`
);
}
}

We once again go over and above to ensure that the OpenAPI spec has been written to the S3 bucket, by retrieving it once it has been successfully written to the bucket.

We now have our deployment always generating the latest OpenAPI spec which is exported from Amazon API Gateway, deployed to S3, and available to consumers using CloudFront!

One annoying bug 🐛

One key thing to note in this setup is one annoying bug with CloudFormation/API Gateway which is discussed below:

This issue manifests itself when we perform a deployment of the API Gateway where we have deleted previous paths or methods, and a quirk means that the previous definition and the new one are merged at the stage level.

An example would be completely replacing the V1 resources with V2 versions, which essentially means a deletion of the V1 resources and creations of new V2 resources in CloudFormation. From an API perspective this looks fine as shown below (we don’t see the deleted V1 resources):

Only the stage shows the redundant resources, the parent API resources are correct. This is why we need to force the deployment of the API.

However, if we look at the deployed stage we will see both, i.e. the previously deleted resources and the new ones (V1 and V2):

An example of where the stage still reflects the previous API’s resources, even though they have been amended/removed

This means that when we export the OpenAPI definition we see both versions as shown below:

An example of where the stage still reflects the previous API’s resources, even though they have been amended/removed

In reality with versioned APIs we should ensure that the original versions are not removed entirely anyway (i.e. no breaking changes), but this is a notable quirk regardless and has to be mentioned.

Conclusion

Now let’s talk about the advantages and disadvantages of the two approaches:

Part 1 — OpenAPI spec first approach:

Diagram showing the OpenAPI spec first approach with Amazon API Gateway

Part 2 — Code first approach:

Diagram showing the code first approach with Amazon API Gateway

Advantages and Disadvantages

OK, so before moving onto the advantages and disadvantages of this approach, a recent poll on LinkedIn showed the majority of teams follow the approach in Part 1:

I’m not actually surprised by this, but it is great to get this somewhat validated by the community too! 🙌🏽

What are the advantages and disadvantages of this code first approach?

Advantages ✔️

  1. Some teams may feel the code first approach is less verbose compared to working with large JSON objects, i.e. writing it all in CDK code.
  2. Working with the ‘x-amazon-apigateway-integration’ definitions is not intuitive and the documentation is also not great. When looking at direct integrations (over and above basic Lambda integrations) it can be a bit cumbersome and error prone.

Disadvantages ❌

  1. There are many more lines of code in this approach compared to that of the OpenAPI spec first approach. For example, the code in the stack in this instance is 590 lines long compared to 225 (and we also didn’t generate all of the documentation parts too!).
  2. There is more complexity I feel in this approach when it comes to exporting the API documentation from API Gateway using a Custom Resource, and the complexities in code of working around edge cases (like checking the correct API has been deployed before exporting, checking that the file has actually been written etc).
  3. We can’t reuse the base models in the simplistic way we did in the OpenAPI spec first approach, as in this approach the models need API Gateway specific types in there..
  4. There is the added quirks around previous deleted resources still showing when exporting the schemas from API Gateway.
  5. There is the added complexities of ensuring the Custom Resource is ran on every change to the stack. This may feel wasteful to some people (although we are talking fractions of pennies here with serverless of course!)
  6. We had to create the documentation versions and parts in this approach manually, whereas in the OpenAPI spec first approach this is generated automatically for us.

Conclusion

My own preference is the OpenAPI spec first approach when working with Amazon API Gateway as you are defining your contracts up front with other teams, it does a lot of the heavy lifting for us, and we don’t have to deal with the quirks of deleted resources still showing in the spec.

That being said, there is nothing stopping us creating a hybrid solution of the two approaches too.

Wrapping up 👋

Please go and subscribe on my YouTube channel for similar content!

I would love to connect with you also on any of the following:

https://www.linkedin.com/in/lee-james-gilmore/
https://twitter.com/LeeJamesGilmore

If you enjoyed the posts please follow my profile Lee James Gilmore for further posts/series, and don’t forget to connect and say Hi 👋

Please also use the ‘clap’ feature at the bottom of the post if you enjoyed it! (You can clap more than once!!)

About me

Hi, I’m Lee, an AWS Community Builder, Blogger, AWS certified cloud architect and Global Serverless Architect based in the UK; currently working for City Electrical Factors (UK) & City Electric Supply (US), having worked primarily in full-stack JavaScript on AWS for the past 6 years.

I consider myself a serverless advocate with a love of all things AWS, innovation, software architecture and technology.

*** The information provided are my own personal views and I accept no responsibility on the use of the information. ***

You may also be interested in the following:

--

--

Global Head of Technology & Architecture | Serverless Advocate | Mentor | Blogger | AWS x 7 Certified 🚀