AWS Lambda & Shared layers for Python

At Flaneer, we are working on a new kind of workplace: a digital and remote one. Thanks to Flaneer, everyone can now turn its workstation (be it a laptop, a tablet or even a phone) into a powerful cloud-computer (or DaaS).

If you’d rather work with teammates, you can join their session and collaborate seamlessly. Talk with each other, annotate each other’s work live, send snapshots… Flaneer is much more than a simple VDI/virtualization service.

To make our platform as fast and scalable as possible, we built most of it on top of the Serverless Framework, with the help of AWS Lambda. You can find the code related to this article in the following Github repository.

AWS Lambda lets you run code in response to events and automatically manages the underlying compute resources for you. You can use it to create and deploy application back-ends quickly and seamlessly integrate other AWS services. You won't have to manage any servers, and can simply focus on the code.

If your lambda functions are written in Python, you may need to use public libraries or private custom package. The goal of this article is to focus on AWS Lambda Shared Layers, a common place where you can store common functions and libraries for all of your lambda functions. We provide an overview of how to build such layers in this article! Our shared layers will be based on both public libraries, and on custom internal code. Let's start with the architecture of shared layers!

Serverless & Shared Layers Architecture

Our shared layer will follow the simple structure:

We will then package this python folder as a shared layer that each lambda function will be able to use in the following manner:

Add External Libraries to Serverless Projects

Installing the external libraries is pretty easy, you can use 2 methods. The first one is to directly install them in the right location with the following command:

You will have to repeat this operation for each python version your lambda functions will have to use. You can also make this process quicker by building a requirements.txt file like the following one:

and install them with

Add Internal codes to Serverless Projects

You can build internal library as you would build any usual python project: simply use the folder "internal_lib" as the root folder of your project. You will then be able to use it in your lambda functions without any troubles.

This is is something that is super useful when your project starts to get bigger and bigger. Most of your lambda functions will start to reuse the same objects or functions. It can also allow you to make custom class for other AWS services, such as DynamoDB or Cognito for example.

Publish your Shared Layer

Now that you have a complete layer (with internal & external libraries), you can publish it to AWS. You can use 2 methods: from the AWS Lambda Console, or with a custom script.

AWS Serverless Console

First, let's start by zipping the python folder:

(you can use the method of your choice. We also provide a custom function for Windows user in the script method)

We can now go to the AWS Lambda Console, and on the left, go to Additional resources > Layers.  Here, you can create a new layer (or make a new version of an existing one, depending on if you already built your layer).

Here, you can give a name to your layer, upload the zip file we created earlier and that's it! You can now use your shared layer with every lambda functions you want to code.

If your zip file is too large, you can create an s3 bucket and upload your zip file there. (this is what we do in the second method)

Custom script to package Serverless Shared Layers

The goal of this script is to quickly:

  1. Zip all of the packages you want to use
  2. Upload the zip file to a bucket within your AWS account
  3. Use the bucket to create a new layer, or update an existing one with a new version

This is a simplified version of real scripts. It doesn't deal with any AWS settings, and will simply use your ~/. aws/config variables. You can find it here.

For example, let's say you created a bucket called "flaneer-backend-shared-layers" where you store the zip files, and that you want to update the version of a layer called "test-shared-layer-1", you can use the following (as a macOS user):

Use a Serverless Shared Layer

To use a share layer, you simply have to tell it to the lambda function that should use it. Again, this depends on how you usually build your lambda functions. If you use the AWS console, you can simply go to the page of a lambda function, click on the Layers > Add a layer button, and choose your newly built one.

If you usually use SAM or CloudFormation to build your functions, you can simply tell what layer to use by adding the following line: (if I set global variables for all of my lambda functions)

To go further

  1. Tired of manually updating layer version? You can use the serverless-latest-layer-version plugin to easily manage your version. Simply install the plugin, and replace your version number by the string `latest`.
  2. Make your environnement easier to use by reducing the impact of shared layers by using command code. This will let you build internal libraries like you would on any Python project. See the serverless-package-external plugin!

Get started today