Bug: python 3.12 with setuptools installed and distutils still being required. about aws-sam-cli HOT 5 OPEN

Hi there,

It looks like this is coming from your lambda function, can you confirm that you build your function with all its requirements?

In the meantime, if you can share a reproducible, we can test on our side as well.

from aws-sam-cli.

@mndeveci

In attention to your observation on the build:
I have reviewed the build process and resolved all appearing warnings by updating the referred packages now having the following versions:

chardet.  5.2.0
psycopg2-binary.  2.9.9
requests.  2.32.3
urllib3.  2.2.2

This is the build command:

sam build --use-container --build-image docker.artifactory.my.domain.com/sam/build-python3.12 --template sam-templates/my-template.yaml

Here is the requirements.txt file with the following content:


psycopg2-binary==2.9.9
setuptools==60.5.0

This is the excerpt of the settings in the template file:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  my-app
  SAM Template for my-app server-less stack
Globals:
  Function:
    Handler: app.lambda_handler
    Runtime: python3.12
    Layers:
      - !Ref UtilsLayer
      - !Ref StaticLayer
    Timeout: 900
    Environment:
      Variables:
        
Resources:
  # common modules
  UtilsLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      ContentUri: ../layer-adp/python
      CompatibleRuntimes:
        - python3.12
    Metadata:
      BuildMethod: python3.12
  # static modulea
  StaticLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      ContentUri: ../../layer-static/python
      CompatibleRuntimes:
        - python3.12
    Metadata:
      BuildMethod: python3.12
  # function
  TemplatesForUserAPIFunction:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: ../app-src
      Handler: my_api.get_app
      Events:
        GroupsApi:
          Type: Api 
          Properties:
            Path: /app
            Method: GET
 

As you can see above, I have 2 lambda layers and as the build goes perfect with no errors or warnings and when calling the service after the application server is initialized then the referred error message comes up.

As a side note:

I noticed is that even though I have LOCALLY installed this psycopg2-binary==2.9.9 package, it turns out that when run the build while being network disconnected I get the following error:
PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: psycopg2-binary==2.9.9

Could this indicate that while having locally the package installed it is retrieved from somewhere else?
Once connected back to internet the dependency gets satisfied and the build goes unblemished.

from aws-sam-cli.

Thanks for providing more details.

I've tried to build a python3.12 lambda function with the requirements below;

psycopg2-binary==2.9.9
setuptools==60.5.0

I was able to build it with no issues.

I saw that you are using a custom build image, may I ask why you need it? Can you just run sam build --use-container --template sam-templates/my-template.yaml to see if it is going to succeed?

I noticed is that even though I have LOCALLY installed this psycopg2-binary==2.9.9 package, it turns out that when run the build while being network disconnected I get the following error:
PythonPipBuilder:ResolveDependencies - Could not satisfy the requirement: psycopg2-binary==2.9.9

AWS SAM CLI uses PIP cli to manage dependencies. And it downloads them depending on the architecture of your lambda function. It can't use local dependencies all the time since OS or architecture might be different from what you are targeting. If you prefer, you can run your own build process by defining a Makefile, the process is explained here: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/building-custom-runtimes.html

from aws-sam-cli.

the public.ecr.aws/lambda/python:3.11 (and public.ecr.aws/sam/build-python3.11) image has setuptools installed by default

❯ docker run -it --entrypoint="" public.ecr.aws/lambda/python:3.11 pip list
Package         Version
--------------- -------
awslambdaric    2.0.10
boto3           1.34.42
botocore        1.34.42
jmespath        1.0.1
pip             24.0
python-dateutil 2.8.2
s3transfer      0.10.0
setuptools      65.5.0
simplejson      3.17.2
six             1.16.0
urllib3         1.26.18
wheel           0.43.0

whereas, the public.ecr.aws/lambda/python:3.12 (and public.ecr.aws/sam/build-python3.12) doesn't

❯ docker run -it --entrypoint="" public.ecr.aws/lambda/python:3.12 pip list
Package         Version
--------------- -------
awslambdaric    2.0.10
boto3           1.34.42
botocore        1.34.42
jmespath        1.0.1
pip             24.0
python-dateutil 2.8.2
s3transfer      0.10.0
simplejson      3.17.2
six             1.16.0
urllib3         1.26.18
wheel           0.43.0

from aws-sam-cli.

Finally, I had the application now running on python 3.12.

Having that mndeveci  was able to run it, what I did was to start from a clean slate with a type of 'hello world' application then added the modules one by one and turned out that never had an issue! It's like something somehow was not getting updated. Don't know what that something was.

I was considering to migrate the implementation to python 3.11 runtime as a last resort but had to try the virtual 'start from scratch' out for a last try.

Appreciate your help guys

from aws-sam-cli.