SAM build - does it also build layers?

aws::serverless::layerversion
lambda layers
sam layers
sam build template file not found
sam build command failed
sam build hello world
aws sam local
sam build npm

I'm new to both lambda's and also SAM - so if I've screwed anything simple up don't yell :D.

Summary: I can't get sam build to build a layer specified in template.yaml, it only builds the lambda function.

Background: I'm trying to build a lambda function in python3.7 that uses the skimage (scikit-image) module. To do that, I'm trying to use SAM to build and deploy it all. ...this is working

I'm trying to deploy the scikit-image module as a layer (and also build with SAM), rather than having it included in the lambda function direction ...this isn't working


As a start, I'm simply extending the standard SAM Hello World app.

I've got skimage working by simply add it to requirements.txt , then using sam build -u, then manually removing the numpy/scipy dependencies from the built package directory (I've got the AWS scipy/numpy layer included).

(I added import numpy, scipy.ndimage and skimage.draw to the standard hello world app, and included some test function calls to each)

requirements.txt:

requests
scikit-image

After that, everything works fine (running locally and/or on AWS).


However, I'd now like to move the skimage module out of my app and into a new custom layer (I'd like to have skimage in a layer to re-use for a few functions)

To set that up, I've created a dependencies directory and moved requirements.txt into there (leaving empty requirements.txt in the app directory). I then updated template.yaml to also specify the new layer:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-app

  Sample SAM Template for sam-app

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.7
      Layers:
              - arn:aws:lambda:us-west-2:420165488524:layer:AWSLambda-Python37-SciPy1x:2
              - !Ref SkimageLayer
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: get

  SkimageLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: Skimage
      Description: Skimage module layer
      ContentUri: dependencies/
      CompatibleRuntimes:
              - python3.7
      RetentionPolicy: Retain
    DependsOn:
            - Skimage

directory structure:

▾ dependencies/                
    requirements.txt (responses and scikit-image)          
▸ events/                      
▾ hello_world/                 
    __init__.py                
    app.py                     
    requirements.txt (now empty)          
▸ tests/                       
  README.md                    
  template.yaml                 

However, when I run sam build -u with that template file, nothing gets built for the layer specified in ./dependencies: SkimageLayer in the template.yml file. However the HelloWorldFunction still gets built correctly (now of course without any included modules)

Quick answer - No, currently SAM does not build layers you define in a SAM template.yaml file.

It will only build any functions you define.

However (curiously) it will package (upload to S3) and deploy (setup within AWS, assign ARN so it can be used etc) any layers you define.


There is a feature request on the SAM github issues to implement layer building with SAM.


This can actually be hacked right now to get SAM to build your layers as well, by creating a dummy function in your SAM template file, as well as a layer entry, and having the layer ContentUri entry point to the .aws-sam build directory that gets created for the function.

See my post here on that.

That approach actually seems to work pretty well for twisting SAM right now to build your layers for you.

Building Layers, Builds a serverless application, and prepares it for subsequent steps in your workflow, like locally testing the application, or deploying it to the AWS Cloud. The AWS Serverless Application Model (AWS SAM) is an open-source framework that you can use to build serverless applications on AWS.. A serverless application is a combination of Lambda functions, event sources, and other resources that work together to perform tasks.

I'm not sure if something changed recently but I'm able to do this without issue. My template file and structure is very similar to the OP except I've put all my common code into...

/dependencies/python/lib/python3.7/site-packages/

I didn't include a requirements.txt file in that directory... just the __init__.py file and various .py files that I need to import into my functions.

SAM then finds the code and builds the layer. You don't even need to zip the contents of the directory as some tutorials tell you to do.

The best part is Layers: is able to be put into the Globals: section of the template file so that the layer is available to all of your functions!

Globals:
  Function:
    Handler: main.lambda_handler
    Timeout: 10
    Runtime: python3.7
    Layers: 
        - !Ref HelperFunctions

Resources:
  HelperFunctions:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: MyHelperFunctions
      Description: My Lambda Layer with Helper Functions for accessing RDS, Logging, and other utilities.
      ContentUri: dependencies/
      CompatibleRuntimes:
        - python3.6
        - python3.7
      LicenseInfo: MIT
      RetentionPolicy: Delete

sam build - AWS Serverless Application Model, Execute sam build --template template.yaml. Observed result: Package is built but no requirements are downloaded for dependency layer. You can execute subsequent deployments of your application by simply executing sam deploy and the needed parameters will be retrieved from the AWS SAM CLI configuration file. Deploying Lambda functions through AWS CloudFormation requires an Amazon S3 bucket for the Lambda deployment package.

I got it to work with the following script. Tested with Ubuntu 18 and CodeBuild

It pip install's the layer's requirements to .aws-sam/build/layername/python/. Then you can run sam package and sam deploy as normal

build-layers.py:

import yaml
import subprocess
import sys
import shutil

SAM_BUILD_PATH = ".aws-sam/build"

with open("template.yaml", "r") as f:
    template = yaml.safe_load(f)

for key, resource in template["Resources"].items():
    if resource["Type"] not in ["AWS::Serverless::LayerVersion", "AWS::Lambda::LayerVersion"]:
        continue
    properties = resource["Properties"]
    content_uri = properties["ContentUri"]
    layer_name = properties["LayerName"]
    requirements_path = f"{content_uri}/requirements.txt"

    subprocess.check_call([sys.executable, "-m", "pip", "install", "-r", requirements_path, "-t", f"{SAM_BUILD_PATH}/{layer_name}/python"])

shutil.copyfile("template.yaml", f"{SAM_BUILD_PATH}/template.yaml")

template.yaml:

Transform: AWS::Serverless-2016-10-31

Resources:
  pandas:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: pandas
      ContentUri: pandas
      CompatibleRuntimes:
        - python3.6
        - python3.7
        - python3.8
  sqlparse:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: sqlparse
      ContentUri: sqlparse
      CompatibleRuntimes:
        - python3.6
        - python3.7
        - python3.8

so call python build-layers.py first, then sam package then sam deploy

my directories look like this:

lambda
  layers
    pandas
      requirements.txt (content = pandas)
    sqlparse
      requirements.txt (content = sqlparse)
  template.yaml
  build-layers.py

buildspec.yml:

---  # build spec for AWS CodeBuild

version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.8
    commands:
      - pip install aws-sam-cli
  build:
    commands:
      - cd lambda/layers
      - python build-layers.py
      - sam package --s3-bucket foo --s3-prefix sam/lambda/layers | sam deploy --capabilities CAPABILITY_IAM -t /dev/stdin --stack-name LAYERS

Add dependency build support for layers · Issue #848 · awslabs/aws , GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together. Sign up. Initializes a serverless application with an AWS SAM template. The template provides a folder structure for your Lambda functions, and is connected to an event source such as APIs, S3 buckets, or DynamoDB tables. This application includes everything you need to get started and to eventually extend it into a production-scale application.

sam build to support packaging lambda layers · Issue #1750 , See where this is going? sam build will install all the dependencies for our Layer and copy its code into the build directory, and then when we call  #Step 1 - Download a sample application sam init #Step 2 - Build your application cd sam-app sam build #Step 3 - Deploy your application sam deploy --guided Prerequisites. This guide assumes that you've completed the steps in the Installing the AWS SAM CLI for your OS. It assumes that you've done the following:

Trick Sam into building your Lambda Layers – Bryson Tyrrell, We just need it so that the sam build currently has something to work with. Inside the layer folder can be any combination of local and third-party  Each runtime looks for libraries in a different location under /opt, depending on the language. Structure your layer so that function code can access libraries without additional configuration. You can also use AWS Serverless Application Model (AWS SAM) to manage layers and your function's layer configuration.

Building Lambda Layers by Cheating (a bit) - Jim O'Neil, The basic steps for deploying AWS SAM apps is: This strategy has the benefit of building your function code artifacts in your AWS account. when working with Lambda Layers stored as Stackery environment variables, is an Invalid Lambda  r/ar15: /r/ar15 is here for your favorite black gun links, build pics, questions and other tactical or practical information. Press J to jump to the feed. Press question mark to learn the rest of the keyboard shortcuts

Comments
  • I'm working a bit based on this: noise.getoto.net/2019/02/06/…. About halfway down he implies with how he does it that SAM build wont build a layer itself, as he manually runs npm install inside his node dependencies directory to install the node dependencies
  • Appreciate this answer -- I'll give this approach a try.
  • Problem with this right now is you build all the dependencies on your local. Some might not work after deploy. numpy for example would have this issue.
  • The path is important. The directory name of 'dependencies' is just one I chose but everything beneath it is important and must follow this configuration. If you do, the site packages should be found when uploading the layer. I haven't tried doing it with numpy but I fail to understand how it wouldn't work.
  • @Karuhanga - when you use SAM, there is the option with sam build (I think it's -u flag?) to build inside an AWS lambda environment docker container. That way it builds locally, but uses the correct AWS environment. Therefore everything builds 'natively' for lambda. I use this without problems for numpy etc.
  • @Karuhanga - also NOTE - AWS supplies a pre-built numpy/scipy layer for you to use - you don't need to build your own layer for numpy/scipy if that's all you need a layer for.
  • Hey Neil - I'll have to check that. ...you got it to work without specifying a function? The only way previously to 'fool' SAM into building a layer for you was to associate it with a dummy function. You couldn't specify only a layer in the template yaml file. But you seem to be able to do that! That would be good if they've changed that now
  • @Richard correct no function required
  • Sweet - I'll have to check it out then