AWS Lambda, a level up (part 2): Automation

On this occasion, I’d like to share a couple of ways to enhance the deployment of your Lambda functions, aiming for its complete automation:

• Handling tasks not achiveable with ClofudFormation.
• Use of CF resources in your code via environment variables.

Is Cloud Formation enough?

I guess this is the first question to ask when deploying Lambda functions via Serverless.

Through the Serverless configuration system, which in its most basic form consists of a single serverless.yml file, we are able to deploy our whole stack. Essentially, the configuration of our serverless project will work as a wrap of a CloudFormation template that will be used to deploy our functions and all the resources we need:

The Serverless Framework translates all syntax in serverless.yml to a single AWS CloudFormation template. By depending on CloudFormation for deployments, users of the Serverless Framework get the safety and reliability of CloudFormation.

Reference: https://serverless.com/framework/docs/providers/aws/guide/deploying/

That’s great but we still have to face a couple of issues when trying to deploy our projects:

• Sometimes, as part of our deployment process, we need to carry out some tasks that are independent of AWS e.g: we need to create a webhook in a third party service that will request our function through an API Gateway.
• Or, at times, there are AWS related tasks that can’t be executed via CloudFormation templates (or at least entirely). We’ll procede with an example.

Serverless plugin scripts

To solve the problem of having to deal with these tedious tasks, we’ll use the plugin Serverless plugin scripts.

This plugin will allow you to hook one of the lifecycle events belonging to the different framework commands to run custom scripts. This tool will exponentially elevate the capabilities of your automated actions.

In our case, we’ll locally invoke a Lambda function to create a Cognito user with a permanent password (the user that would request the AppSync API of our first article) when deploying our project via sls deploy.

We just need to include what event we want to hook and the script to run in our serverless.yml

This approach will have an additional advantage, we can make use of the same environment variables defined in our project configuration to create the user and subsequently use it in other functions:

Handler create-our-user.ts

Class OurUserFactory.example.ts

Next time we deploy we’ll be able to see our user is created, saving us any manual step!

Serverless resources in your Lambda functions

Now, that we can control complex steps in our deployments, let’s get back to CloudFormation and the relationship between our created resources and its utilisation in Lambda functions.

A typical scenario when making use of the Serverless framework could be presented as:

1. Create a resource with CloudFormation via serverless.yml
2. Use the resource, generally through the AWS SDK, in your functions code. A standard approach would be accessing the desired resource properties, like the ARN, through environment variables.

Problems

• Serverless doesn’t have (yet) a system to reference our created resources and be assigned to environment variables.
• Though in some places is advised you could use Ref, it looks like there is currently a bug when referencing CF resources and functions in a development environment in Serverless. Besides, sometimes, we’d need to access to properties other than the resource ARN, eg. the URL of the Graphql endpoint.
• We'd like to save any manual step involving the deploy of the stack to manually set our enviroment variables with the resource properties through some copy/pasting.

CloudFormation Outputs

Serverless provides an alternative solution for these problems allowing to reference CloudFormation Outputs in your configuration files and, this is the most interesting part, assigning them to environment variables that can be used in your code.

CloudFormation Outputs is a way to store data derived from the resources created by your template in the stack. This data will be available as a variable to be used from external sources or be referenced from a different CF stack, which is called cross stack references.

So, in our project configuration file, after the Resources section, we’ll add the outputs we need to use in our code. Following our first example, where we needed to work with a Cognito Client, we'll create a variable containing the id of the App Client:

Ok, now we can set one of the environment variables to retrieve the value in one of our functions:

serverless.yml

Extracted code from our first article 

But… oh wait, something wrong is going on:

This is the kind of message returned by Serverless when you haven’t deployed your resources yet or add a new Output. And it makes sense… Serverless will try to resolve CF variables but will find that the value doesn’t exist in the stack yet.

Multiple environment configuration files

A normal practice when working on projects that rely on external services for development (as it is normally the case with AWS) is splitting up your environment variables and other settings into 4 different environments: dev, staging and production which would reference the different stages of your external services and local. The local environment can be used as a way to mock those external services or, for example, work offline.

In our case, we’ll create of a separate cli option --our-build local when wanting to make use of our local variables instead of including it as another stage environment. This will allow us to use the stage option (needed for some actions like deployment) and set the environment variables with our local configuration when dealing with Serverless:

Extracting environment variables into separate files will stop Serverless from trying to resolve CF Outputs not created yet. We’ll use the syntax ${opt:our-build, self:provider.stage} to load the local configuration file, if the option --our-build local is detected, or the corresponding to the stage otherwise.

serverless.yml

Using our local settings:

local.yml

With this approach, we'll be able to carry out the first deployment of our stack without Serverless complaining about referencing non existing resources with:

Once the stack is created (and our Outputs) we can make use of its resources, if we need it, in our functions starting sls as normal:

development.yml

To ensure your stage environment use the created Output values, you just need to deploy getting rid of the local flag:

Conclusion

Hopefully, these solutions can improve the automation of some of your tasks and be integrated as part of your CI/CD processes.

This is the last of a series of articles that started with Operating with AppSync from AWS Lambda, which raised some questions about possible improvements and resulted, firstly, in Lambda, a level up (1/2): Performance.