Issue using M2Crypto on lambda (works on EC2)

aws lambda region
lambda function console
ujson aws lambda
aws lambda lxml
aws lambda python cryptography
cython aws lambda
cannot import name 'etree' from lxml lambda
aws lambda pillow

I am trying to install a python function using M2Crypto in AWS Lambda.

I spun up an EC2 instance with the Lambda AMI image, installed M2Crypto into a virtualenv, and was able to get my function working on EC2.

Then I zipped up the site-package and uploaded to Lambda. I got this error

Unable to import module 'epd_M2Crypto': /var/task/M2Crypto/_m2crypto.cpython-36m-x86_64-linux-gnu.so: symbol sk_deep_copy, version libcrypto.so.10 not defined in file libcrypto.so.10 with link time reference

There are similar questions and hints here and here. I tried uploading the offending lib (libcrypto.so.10) in the zip file, but still get the same error. I am assuming the error means that the EC2 version of libcrypto.so.10 (used to install M2Crypto) is different than the version on Lambda (that I trying to run with), so M2Crypto complains.

If I look at the versions of openssl they are different:

  • OpenSSL 1.0.0-fips 29 Mar 2010 (lambda version)
  • OpenSSL 1.0.2k-fips 26 Jan 2017 (ec2 version)

I don't think the answer is to downgrade openssl on ec2 as the 1.0.0 version is obsolete (AWS applies security patches but the version still shows as 1.0.0). (Also the yum doesn't have versions this old)

Here's the steps i used on the EC2 instance to get it working on EC2:

$ sudo yum -y update
$ sudo yum -y install python36
$ sudo yum -y install python-virtualenv
$ sudo yum -y groupinstall "Development Tools"
$ sudo yum -y install python36-devel.x86_64
$ sudo yum -y install openssl-devel.x86_64

$ mkdir ~/forlambda
$ cd ~/forlambda
$ virtualenv -p python3 venv
$ source venv/bin/activate

$ cd ~
$ pip install M2Crypto -t ~/forlambda/venv/lib/python3.6/site-packages/

$ cd ~/forlambda/venv/lib/python3.6/site-packages/
$ (create python function that uses M2Crypto)
$ zip -r9 ~/forlambda/archive.zip .

Then added to the zip file

  • /usr/bin/openssl
  • /usr/lib64/libcrypto.so.10
  • /usr/lib64/libssl.so.10

And uploaded to Lambda, which is where I am now stuck.

Do I need to do something to get Lambda to use the version of libcrypto.so.10 that I have included in the uploaded zip?

My function:

"""
Wrapper for M2Crypto
https://github.com/mcepl/M2Crypto
https://pypi.org/project/M2Crypto/
"""

from __future__ import print_function
from M2Crypto import RSA
import base64
import json

def decrypt_string(string_b64):
    rsa = RSA.load_key('private_key.pem')
    string_encrypted = base64.b64decode(string_b64)
    bytes = rsa.private_decrypt(string_encrypted, 1)
    string_plaintext = bytes.decode("utf-8")

    response = {
        's': string_plaintext,
        'status': "OK",
        'statuscode': 200
    };
    return response


def lambda_handler(event, context):

    response = ""
    action = event['action']

    if action == "decrypt":
        string_b64 = event['s']
        response = decrypt_string(string_b64)

    return response

First I ran this command on the EC2 instance to make sure I had included the correct .so file in my .zip:

$ ldd -v _m2crypto.cpython-36m-x86_64-linux-gnu.so 

The output of the ldd command (edited for brevity):

    libssl.so.10 => /lib64/libssl.so.10 (0x00007fd5f1892000)
    libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fd5f1433000)

Based on the output above, I included /lib64/libcrypto.so.10 in my .zip.

Also (at the suggestion of AWS Support), on the Lambda console, under 'Environment variables', I added a key 'LD_LIBRARY_PATH' with value '/var/task'.

I'm not sure if I needed both those changes to fix my problem, but it works right now and after three days of troubleshooting I am afraid to touch it to see if it was one or the other that made it work.

Running Python with compiled code on AWS Lambda, This is especially nice for sporadic work loads in response to events like Under the hood, your Lambda functions are running on EC2 with  Issue using M2Crypto on lambda (works on EC2) I am trying to install a python function using M2Crypto in AWS Lambda. I spun up an EC2 instance with the Lambda AMI image, installed M2Crypto into a virtualenv, and was able to get my function

It is perhaps too brutal, but would it be possible to use LD_PRELOAD to force using your preferred version of OpenSSL library?

Troubleshoot Lambda Invoke API Errors 502 and 500, When I try to invoke my AWS Lambda function, the request fails with a 502 or Use Amazon CloudWatch to review the amount of Amazon EC2 API If the issue persists, contact AWS Support from the AWS Support Center. Amazon EC2: In comparison to AWS Lambda vs EC2, the later one have pretty flexible options. You can definitely work with long running tasks since instances are available for different types of requirements with different configurations.

AWS support provided a resolution, upgrading to use Python 3.7 where the issue is resolved:

Our internal team has confirmed that the issue is with Lambda's Python runtime. In a few rare cases, when the Lambda function is being initialised, Lambda is unable to link against the correct OpenSSL libraries - instead linking against Lambda's own in-built OpenSSL binaries.

The team suggests trying this out in the Python3.7 environment where this behaviour has been fixed. Also, python3.7 is compiled with the newer openssl 1.0.2 and you should not have to include the binaries in the Lambda package. ... still had to include the OpenSSL binaries in the package and could not get it working with the default libraries.

Amazon Linux AMI 2017.09 Packages, This page lists the source RPMs comprising the Amazon Linux AMI 2017.09 release on 2017-10-03. docker-storage-setup-0.6.0 python-m2crypto-0.21.1 Today I was stuck finding a solution on a very specific problem: find a way to execute one or more shell command on a Linux EC2 on AWS based on a particular event, let’s say a file upload on a S3…

AWS lambda runs code on an old version of amazon linux (amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2) as mentioned in the official documentation https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html

So to run a code that depends on shared libraries, it needs to be compiled in the same environment so it can link correctly.

What I usually do in such cases is that I create virtualenv using docker container. The virtualenv can than be packaged with lambda code.

Please note that if you need install anything using yum (in the docker container), you must use same release server as the amazon linux version:

yum --releasever=2017.03 install ...

virtualenv can be built using an EC2 instance as well instead of docker container (though, I find docker method easier). Just make sure that the AMI used for EC2 is same as the one used by lambda.

Questions for tag m2crypto, installing m2crypto for python on windows · Error installing m2crypto (Centos 7.5.​1804, python 2.7.5, pip 8.1.2) · Issue using M2Crypto on lambda (works on EC2). Hi, I received EC2ThrottledException with message: Lambda was throttled while using the Lambda Execution Role to set up for the Lambda function. when using lambda client.

sk_deep_copy requirement causes M2Crypto to fail to load in AWS , We were able to eventually get around this issue by packaging OpenSSL 1.0.1k with our Lambda package, but I noticed these two commits:  By using run_instances we can quickly and easily launch an EC2 instance. The trick, however, is getting the newly launched instance to actually do something . To do that, we need to use the

Changelog, PR #495 - @monkeysecurity - issue #494 - Refactoring to work with the new EC2::Instance; ENI; KMS::Grant; KMS::Key; Lambda; RDS::ClusterSnapshot; RDS:: to remove swig/python-m2crypto and add libffi-dev; Issue #220 - SQS Auditor  AWS Lambda is a service for running code in response to events, such as changes to data in an Amazon S3 bucket and Amazon DynamoDB tables, or as compute services to run your code in response to HTTP requests using Amazon API gateway or API calls made by using AWS SDKs.

Scalr Enterprise Edition :: Release notes, With 7.16.9 release, Scalr no longer supports CentOS 6. [SCALRCORE-15034] - AWS > EC2 i3en, x1e, m5ad, r5ad Instances should be populated to all [​SCALRCORE-12779] - UI > AWS Lambda Execution Profile should respect "0" does not work; [SCALRCORE-12435] - Cost Analytics > Azure > Issues with billing in  "Execution failed due to configuration error: Lambda was not able to access EC2's API using the Lambda Execution Role to set up the Lambda function." Can someone tell us what basically might be going wrong? Details: We are working from eu-west-1.

Comments
  • My function worked for a few hours, and then the next day stopped working (no changes). I have opened another ticket with AWS support. My guess is that the issue is related to the underlying OS instance that is picked up -- and it works on some and not others.
  • Thanks for the suggestion! I tried it out but still no luck :( In the Lambda console, under key I added LD_PRELOAD with value '/var/task/libcrypto.so.10' (without the quotes). I also tried value '/var/task/openssl:/var/task/libcrypto.so.10:/var/task/libssl.so.10'. I would have expected this to work, so now I'm wondering if I'm misunderstanding the error message from Lambda 'Unable to import module 'wz_M2Crypto': /var/task/M2Crypto/_m2crypto.cpython-36m-x86_64-linux-gnu.so: symbol sk_deep_copy, version libcrypto.so.10 not defined in file libcrypto.so.10 with link time reference'
  • I really don't know, if you find a solution, please, let us know on gitlab.com/m2crypto/m2crypto/issues .
  • Moving to Python3.7 worked for me with the default libraries, there was no need to include them in the package. Could it be that you accidentally left the "/var/task" library mapping in the environment variables definition for LD_LIBRARY_PATH?