Rapid AWS API dev on your laptop with Localstack, Pulumi, PostgreSQL, and Docker

Rapid API Dev

Rapid API Dev Diagram

1. Deploy containers on Docker Desktop for Localstack and PostgreSQL

2. Deploy AWS services onto Localstack using Pulumi, some code you’d use to deploy to AWS

3. Run Jest tests. Tests create DB schema/tables, hit APIGWs

4. Tests assert data in DB is correct

This Blueprint / Tutorial shows you how to use Pulumi as your Infrastructure as Code (IaC) to develop RESTful APIs deployed on AWS API Gateway (APIGW) , serverless Lambda integrations, and a PostgreSQL DB backend, all on your laptop. Developer workflow round-trip times are dramatically reduced while employing test-driven development (TDD) methodologies. APIs and DB schemas can be fully flushed out before being promoted to CI environments and then production. The beauty of this approach is the DevOps integration in the IaC; the same pulumi IaC that’s used to develop solutions locally is used to deploy to AWS environments. So let’s get into it.

Setup

This BluePrint is done on a Mac so some of the instructions are Mac / Linux specific. It should be doable on a Windows machine also somehow.

1. Install Homebrew on your Mac. https://docs.brew.sh/Installation

2. Install Docker Desktop. https://www.docker.com/products/docker-desktop

3. Install NVM (node version manager). If you’re on Windows, install the latest stable Node version.

https://github.com/nvm-sh/nvm

4. Run these commands to install the latest node and use it

> nvm install node

> nvm use node

5. Install Yarn.

> npm install — global yarn

6. Install AWS CLI. https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html

7. Clone the repository here https://github.com/cabeaulac/apigw-lambda-localstack-bp

Configuration

1. Create a .env file with the following variables:

export APP_NAME=bp-api

export REGION=us-east-1

export STAGE=bp.local

export STACK=${APP_NAME}-${STAGE}

export LS_VERSION=latest

To deploy to AWS just change the STAGE value to anything that doesn’t end in .local

Login to pulumi locally to manage the project stack states on the local file system

To see more document on Pulumi backends, see this: https://www.pulumi.com/docs/intro/concepts/state/#backends

We’re storing the pulumi state to the current directory you’re in. A .pulumi directory will be created.

$ pulumi login file://`pwd`

In the terminal run the following commands

The project uses secrets. Set a pulumi project passphrase

export PULUMI_CONFIG_PASSPHRASE=<choose some passphrase to encrypt project secrets>

Install required packages

$ make setup

Start localStack and PosgreSQL docker

$ make up

Deploy AWS Resources with Pulumi

Run

$ make deploy

If you want to run pulumi up from the command line, you need to source the .env file. source .env

Test the API

Install all Packages

$ yarn install

Run all the tests

All test files end with test.ts. Go find them. Might have to mess with the test and Lambda timeouts to get this to pass on the first try while Lambda Docker containers are launching. If all the tests don’t pass on the first try, run yarn test again.

$ yarn test

Destroy

Don’t do this unless you want to completely start from scratch again. The process is super-fast though. This deletes your pulumi stack and you’ll have to start back at make setup.

Run

$ make destroy

Destroy pulumi stack, recreate it, redeploy

Delete stack, stop Docker containers, cleanup Docker.

$ make redo

Deploy the services

$ make deploy

Developer SDLC

The developer Software Development LifeCycle is the ecosystem the developer is operating in and the workflows they can perform. The better the ecosystem, the faster the developer worksflows can be run, maximizing developer velocity.

Change a Lambda function

Let’s assume you have the system running. Let’s change a Python Lambda

1. Open a Lambda function.

src/profile/lambda/python/get_profile_by_id.py

2. You see that this references the Profile class, and that’s what you really want to change to add some more error handling or something. src/profile/lambda/python/lib42/db_util.py

3. Update unit tests in src/test/db-api.test.ts

4. Run `make deploy` to deploy the new Lambda

5. Run yarn test to run all the unit tests

Add route to the REST API and new Lambda

This Blueprint uses AWS API Gateway Version 1 because I want to be able to quickly switch between API Gateway types regional, edge, and private.

Every REST operation needs

- resource to hit. We’re creating a Lambda

- give APIGW permission to invoke the Lambda

- APIGW needs a resource (URI path), HTTP Method, and Integration

- good automated tests

This is where to do all of that in this project.

1. Let’s create a Python Lambda. See the src/profile/lambda/python/get_profile_by_id.py file and how it defines a Lambda. Also the makefile in that the root directory and the src/profile/lambda/python directory handle packaging the Lambda into a zip file inside a Docker container targeted to the Python 3.8 runtime.

Now we’re going to modify the Pulumi Typescript resource definitions in src/profile/profileApi.ts

2. Define the Lambda Function. See examples around line 105. Every resource must have a unique name.

const getProfilePythonLambda = new aws.lambda.Function

3. Give APIGW permission to invoke the Lambda. See example around line 168.

let invokeGetPythonPermission = new aws.lambda.Permission

4. Add resource path, method, and integration. See example around line 284.

// GetProfile APIGW config

// PATH: /profile/python

const getProfilePythonResourceBase = new aws.apigateway.Resource

// PATH: /profile/python/{id}

const getProfilePythonResource = new aws.apigateway.Resource

const getProfilePythonMethod = new aws.apigateway.Method

const getProfilePythonInteg = new aws.apigateway.Integration

5. Create tests that hit the APIGW endpoint and assert results.

See src/test/db-api.test.ts

Run yarn test until your tests pass.

Misc

After you reboot

The Docker containers are not running and you’ve lost all your Localstack deployed resources.

Login to Pulumi backend.

$ pulumi login file://`pwd`

Set your Pulumi project passphrase. The one you use for this project.

$ export PULUMI_CONFIG_PASSPHRASE=<choose some passphrase to encrypt project secrets>

Wipe out the pulumi state and start the Docker containers

$ make redo

Deploy your resources again

$ make deploy

Login to PostgreSQL DB running in Docker

If you want to get onto the PostgreSQL docker instance, you can do the following.

$ docker-compose run database bash

Tailing Lambda Logs on Localstack

Export some fake AWS Credentials

$ source testcreds.sh

Get Lambda names

$ aws — endpoint-url=http://localhost:4566 lambda list-functions

Tail Lambda Log in real-time

$ aws — endpoint-url=http://localhost:4566 logs tail /aws/lambda/<Lambda name> — follow

There’s also a convenient make target to tail the createProfile Lambda log

$ make lambda-log

And

$ make lambda-log-get

Deploy to AWS

You have to change the pulumi stack name to not end in .local and run pulumi up with some valid AWS credentials. You’d also have to alter this solution to point to a DB that exists that’s reachable. Actual solutions that integrate this approach can use other solutions to deploy AWS RDS Aurora with AWS RDS Proxy and store connection credentials in AWS Secrets Manager.

This was my first tech article on medium.com. Hope you experience some rapid REST API dev targeted for AWS!