Unable to import module 'lambda_locust': No module named 'greenlet' about invokust HOT 9 CLOSED

Hi Chris,
What OS are running pip3 install invokust --target=python-packages on? And where do you see that error message? In AWS Lambda console? Or are you trying to run it locally?

from invokust.

Hey @FutureSharks

I can confirm that error. I am doing the build on a centos7 machine (latest minimal build) with python3 installed. Ran the exact build commands you've documented (I've removed their output as they succeed)

$ git clone [email protected]:FutureSharks/invokust.git
$ cd invokust/
$ pip3 install invokust --target=python-packages
$ zip -q -r lambda_locust.zip lambda_locust.py locustfile_example.py python-packages
$ aws lambda update-function-code --function-name test-invokust-lambda --zip-file fileb://lambda_locust.zip
$ aws lambda invoke --function-name test-invokust-lambda --invocation-type RequestResponse --payload '{"locustfile": "locustfile_example.py", "host":"https://example.com", "num_requests":"20", "num_clients": "1", "hatch_rate": "1"}' output.txt
{
    "FunctionError": "Unhandled",
    "StatusCode": 200
}
$ cat output.txt
{"errorMessage": "Unable to import module 'lambda_locust'"}

The error from the lambda logs in CloudWatch says:

Unable to import module 'lambda_locust': No module named 'greenlet'

I suspect the greenlet module expects some C header file (since it's a C module) that is not present in the AWS lambda environment. (I'll keep this thread updated if I have findings or solutions to this)

from invokust.

I can also add that doing the build using mock to create a new clean centos7 chroot and virtualenv inside that has the same result (and the built zip package is identical).

from invokust.

Hmmm interesting. Could you use my docker example to create python-packages for the zip file? If this works, then it's a problem with libraries/dependencies.

from invokust.

Yes, doing the build inside the docker container you specified in your docs does produce a package that works inside the lambda environment:

$ docker run -it --volume=$PWD/python-packages:/python-packages python:3.6 bash -c "pip install invokust --target=/python-packages"
$ zip -q -r lambda_locust.zip lambda_locust.py locustfile_example.py python-packages
$ aws lambda update-function-code --function-name test-invokust-lambda --zip-file fileb://lambda_locust.zip
$ aws lambda invoke --function-name test-invokust-lambda --invocation-type RequestResponse --payload '{"locustfile": "locustfile_example.py", "host":"https://example.com", "num_requests":"20", "num_clients": "1", "hatch_rate": "1"}' output.txt
{
    "StatusCode": 200
}

Really annoying. Ideally AWS should provide an image with the exact lambda environment for these exact scenarios where you are using modules that need to be compiled against that environment.

Not sure how we can address this...other than put a note in the docs that warns against this problem

from invokust.

That is strange.

I would have thought it would be fine any Linux x86_64 host. There is some into here:
https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html

Perhaps I should just only include the Docker way in my notes?

from invokust.

Yeah, I've seen those docs (we use lambdas extensively) and we build them inside virtualenvs copying the site-packages.

I'll try to verify this, but it might be some low level incompatibility between python3.4 and python3.6 (because on centos7 you can only get up to python3.4 in a sane way). If that's the case you can at least document that you have a hard dependency on python3.6 since that's the runtime AWS lambda uses (there's no 3.4 runtime).

Thank you for your responses.

from invokust.

I'll close this. Feel free to reopen if you still have a problem.

from invokust.

I have been having the same exact issue whether I use docker or not. I am using Amazon Linux 2.

from invokust.