bug: lambda could not connect to rds mysql cluster about localstack HOT 5 CLOSED

Hi @steffyP,

after using a reserved port everything works for me. Thanks a lot 🙏 and sorry for stealing your time ...

from localstack.

Hi @InvokedLambda,

thanks for reaching out.

From the screenshot and the logs that you provided:

1. the database connection test with MySQLWorkbench does not use the port 3306 (which is used by the lambda), but the container port. Please consider checking that 3306 is also reachable 🙏

2. the logs suggest that you are creating a database connection string using the SQLAlchemy syntax. It seems your database connection string results into mysql+pymysql://lambda:<passwd>@192.168.176.2:3306/mydb. However, your password contains some special characters. You may need to do some URL escaping for the password.

If this doesn't solve your issue already, please provide a small complete sample, e.g with a fully-working lambda, and explanation how the ENVs are set or retrieved, so we can try to reproduce the issue.

from localstack.

Hi @steffyP,

thanks for your response.

I updated my code that the password will be escaped:
Code:

hostname = urlparse(os.environ.get("AWS_ENDPOINT_URL")).hostname

def get_engine():
    url_object = URL.create(
        os.environ.get('DB_INSTANCE_DRIVER'), 
        username=os.environ.get('DB_INSTANCE_USER'),
        password=os.environ.get('DB_INSTANCE_PASSWORD'),  # plain (unescaped) text
        host=hostname,
        port=os.environ.get('DB_INSTANCE_PORT'),
        database=os.environ.get('DB_INSTANCE_DATABASE_NAME'),
    )

    print("Url-Object: " + url_object.render_as_string(False))

    return create_engine(url_object)

Output:
2024-04-17T07:26:20.605 DEBUG --- [ asgi_gw_5] l.s.l.i.version_manager : [cmpr-lambda-function-organization-type-get-all-dev-81d388a5-936a-4916-83e2-a149fbf93cfd] Url-Object: mysql+pymysql://lambda:%26%26_-S_%[email protected]:3306/dbcmpr

I also exposed the 3306 port of my LocalStack docker container. But I still got this error:
2024-04-17T07:26:20.608 DEBUG --- [ asgi_gw_5] l.s.l.i.version_manager : [cmpr-lambda-function-organization-type-get-all-dev-81d388a5-936a-4916-83e2-a149fbf93cfd] sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.192.2' ([Errno 111] Connection refused)")

I create the rds cluster, my db and the lambda function like this with terraform:

resource "aws_rds_cluster" "my_rds_cluster" {
  cluster_identifier                  = "${var.project}-rds-instance-${var.namespace}"
  engine                              = "aurora-mysql"
  engine_version                      = "8.0"
  engine_mode                         = "provisioned"
  database_name                       = "db${var.project}"
  master_username                     = var.database_master_user
  master_password                     = var.db_master_pass_result
  db_subnet_group_name                = aws_db_subnet_group.private_rds_group.id
  vpc_security_group_ids              = [var.sg_rds_id]
  iam_database_authentication_enabled = true
  apply_immediately                   = true
  skip_final_snapshot                 = true
  storage_encrypted                   = true
  port                                = 3306

  serverlessv2_scaling_configuration {
    min_capacity = 0.5
    max_capacity = 1.0
  }

  tags = {
    Name = "${var.project}-rds-cluster-${var.namespace}"
  }
}

resource "aws_rds_cluster_instance" "rds_instance" {
  cluster_identifier   = aws_rds_cluster.my_rds_cluster.id
  instance_class       = "db.serverless"
  engine               = aws_rds_cluster.my_rds_cluster.engine
  engine_version       = aws_rds_cluster.my_rds_cluster.engine_version
  apply_immediately    = true
  db_subnet_group_name = aws_db_subnet_group.private_rds_group.id
  publicly_accessible  = false

  tags = {
    Name = "${var.project}-rds-instance-${var.namespace}"
  }
}


data "archive_file" "archive_src_lambda" {
  type        = "zip"
  source_dir  = "${var.lambdas_relative_path}src/${var.lambda_src_path}"
  output_path = "${var.lambdas_relative_path}zipped/${var.lambda_function_name}.zip"
}

resource "aws_s3_object" "s3_object_lambda" {
  bucket = var.s3_bucket_lambdas_id

  key    = "lambda/${var.lambda_function_name}"
  source = data.archive_file.archive_src_lambda.output_path

  etag = filemd5(data.archive_file.archive_src_lambda.output_path)

  tags = {
    Name = "${var.project}-s3-object-${var.lambda_function_name}-${var.namespace}"
  }
}

resource "aws_lambda_function" "lambda_function" {
  function_name           = "${var.project}-lambda-function-${var.lambda_function_name}-${var.namespace}"
  description             = ""
  code_signing_config_arn = ""
  source_code_hash        = filebase64sha256(data.archive_file.archive_src_lambda.output_path)

  s3_bucket = var.s3_bucket_lambdas_id
  s3_key    = aws_s3_object.s3_object_lambda.key

  handler = "lambda_function.lambda_handler"
  runtime = "python3.9"
  role    = var.iam_role_lambda_arn

  layers = var.aws_lambda_layer_arn_list

  timeout = 30

  environment {
    variables = {
        DB_INSTANCE_DRIVER          = "mysql+pymysql"
        DB_INSTANCE_ENDPOINT        = aws_rds_cluster.rds_cluster.endpoint
        DB_INSTANCE_PORT            = aws_rds_cluster.rds_cluster.port
        DB_INSTANCE_DATABASE_NAME   = aws_rds_cluster.rds_cluster.database_name
        DB_INSTANCE_USER            = "lambda"
        DB_INSTANCE_PASSWORD        = module.secrets_manager.db_lambda_pass_result
      }
  }
  vpc_config {
    subnet_ids         = var.private_subnets_ids
    security_group_ids = var.security_group_id_list_lambda
  }

  tags = {
    Name = "${var.project}-lambda-function-${var.lambda_function_name}-${var.namespace}"
  }
}

awslocal rds describe-db-clusters

{
    "DBClusters": [
        {
            "AllocatedStorage": 1,
            "DatabaseName": "dbcmpr",
            "DBClusterIdentifier": "cmpr-rds-instance-dev",
            "DBClusterParameterGroup": "default.aurora-mysql8.0",
            "DBSubnetGroup": "cmpr-rds-cluster-subnet-group-dev",
            "Status": "available",
            "Endpoint": "localhost.localstack.cloud",
            "ReaderEndpoint": "localhost.localstack.cloud",
            "MultiAZ": false,
            "Engine": "aurora-mysql",
            "EngineVersion": "8.0",
            "Port": 3306,
            "MasterUsername": "dbroot",
            "DBClusterMembers": [
                {
                    "DBInstanceIdentifier": "tf-20240417071438684800000016",
                    "IsClusterWriter": true,
                    "DBClusterParameterGroupStatus": "in-sync",
                    "PromotionTier": 0
                }
            ],
            "VpcSecurityGroups": [
                {
                    "VpcSecurityGroupId": "sg-0e7b739211a535482",
                    "Status": "active"
                }
            ],
            "StorageEncrypted": true,
            "DbClusterResourceId": "cluster-efaa4385",
            "DBClusterArn": "arn:aws:rds:eu-central-1:000000000000:cluster:cmpr-rds-instance-dev",
            "IAMDatabaseAuthenticationEnabled": true,
            "EngineMode": "provisioned",
            "DeletionProtection": false,
            "CopyTagsToSnapshot": false,
            "TagList": [
                {
                    "Key": "Name",
                    "Value": "cmpr-rds-cluster-dev"
                }
            ],
            "ServerlessV2ScalingConfiguration": {
                "MinCapacity": 0.5,
                "MaxCapacity": 1.0
            }
        }
    ]
}

aws_rds_cluster.rds_cluster.endpoint returns localhost.localstack.cloud but this is wrong for me because I need to connect over the docker network. Which would be accessible over the env "AWS_ENDPOINT_URL" as I already do in my code. When I use "localhost.localstack.cloud" Im getting a error that the network address could not be found.

For me it looks like that localstack starts the rds cluster in the localstack container correctly as you can see in the "awslocal rds describe-db-clusters" result. But localstack also starts the instance as a new docker container. This new docker container is not in the same network as the localstack container. When I now invoke the lambda function, the lambda function will be started in a new docker container as well but in the same network as the local stack docker container and now the lambda function tries to connect over the network on port 3306 to the rds cluster. It looks like that the cluster could be reached but the connection is refused because the rds instance is not accessible.

Docker container overview:

LocalStack network conf:

MySql network conf:

Lambda network conf:

So as you can see the lambda function could never reach the mysql instance because they are in different networks. The lambda function and localstack could reach out each other over 192.168.192.0 but the mysql container is in another network 172.17.0.0 . So beside the lambda function, the rds cluster itself is not able to reach out its own instances.

As I read here #6782 there was the Mariadb only first and then the mysql was added. The MariaDB instance was started directly in the localstack container but for the mysql engine a new docker container will be started. So I don't know what the exact behaviour should be. Maybe the lambda function should connect to the rds cluster in the localstack container (which is working currently) but then the rds cluster should redirect the lambda function to the instance (based on load distribution) but the instance is not reachable from the localstack or the lambda container. And that's why the connection is refused. But this is only my thoughts, I don't know what the behaviour should be in detail.

from localstack.

In addition to check if there are any permission issues I also tried it with the root user-

Output:
2024-04-17T11:19:52.543 DEBUG --- [ asgi_gw_10] l.s.l.i.version_manager : [cmpr-lambda-function-organization-type-get-all-dev-270b17ff-0207-4ab0-8579-cdf7895ccb7a] Url-Object: mysql+pymysql://root:%26%7D%7D%7B%29M%26%3Ed5+oSY%7B3wsfXTTW%[email protected]:3306/dbcmpr

MySqlDeveloper:

Edit: port has changed because meanwhile I redeployed everything from scratch

from localstack.

Hi @InvokedLambda,

I think there might be a few misunderstandings.

In your initial Dockerfile you have the following port-configuration

ports:
      - "4566:4566"
      - "4510-4559:4510-4559"     
      - "53:53"                # DNS config (only required for Pro)
      - "53:53/udp"            # DNS config (only required for Pro)
      - "443:443"

Which means that for the port range of external services (like RDS mysql) you have ports 4510-4559 reserved.
Whenever you request e.g. a new database cluster for MySQL, LocalStack will search a free port from the range 4510-4559 and select one. The MySQL container starts as a separate docker container, but LocalStack will proxy the connection. E.g. if LocalStack selects port 4510 it will automatically forward the traffic to the MySQL container.

I also exposed the 3306 port of my LocalStack docker container. But I still got this error:

This doesn't make any sense in that case :) You are exposing a port that is not used on LocalStack.

In your terraform-script you are requesting port 3306 but as this port is not configured for LocalStack, it should actually select and return a different one (I also tested and verified this).

So this is where things get complicated, because you posted the output of describe-db-clusters which I believe cannot have the port 3306 assigned.
Please verify the output here. If you still see this port assigned, we would need to get the logs from LocalStack (with DEBUG=1 enabled).
If you want to be able to select the port, you either need to specify one from the range you specified for LocalStack (e.g. 4510-4559), or adapt the port range in the Dockerfile. Please follow the docs for instructions.

Your terraform snippet might contain some issues as well. In your environment section you reference aws_rds_cluster.rds_cluster.port but I think it should be aws_rds_cluster.my_rds_cluster.port.
Also it seems you are using a different password, then for the actual creation of the cluster var.db_master_pass_result vs module.secrets_manager.db_lambda_pass_result.

environment {
    variables = {
        DB_INSTANCE_DRIVER          = "mysql+pymysql"
        DB_INSTANCE_ENDPOINT        = aws_rds_cluster.rds_cluster.endpoint
        DB_INSTANCE_PORT            = aws_rds_cluster.rds_cluster.port
        DB_INSTANCE_DATABASE_NAME   = aws_rds_cluster.rds_cluster.database_name
        DB_INSTANCE_USER            = "lambda"
        DB_INSTANCE_PASSWORD        = module.secrets_manager.db_lambda_pass_result
      }
  }

The connection-test that you are running with the MySqlDeveloper should use the port that is returned by LocalStack (which should be in the range 4510-4559).

Could you please double check your settings and verify that 1) the returned port from LocalStack is within the range, and 2) that the variables you set for the Lambda match the actual database configuration 🙏

from localstack.