Hello,

Thanks for a great distribution!

So, AWS Lambda now supports a Python v3.8 runtime. I am writing "an FYI," to let you know out of gratitude for the project.

Pardon, this is not an "issue" per se, however I did not know of a better location to state the above.

―James

For lighter deployment archive size.

When I run my app.py locally, it works perfectly fine: psycopg2 is successfully imported and used. However when I try to invoke my handler on AWS it returns the following error:

{
"errorMessage": "Unable to import module 'app'"
}

The log says:
Unable to import module 'app': No module named psycopg2

the psycopg2 directory is placed in the root directory of my lambda package (alongside my app.py). The same procedure works with other libraries (e.g. requests). I tried with and without virtualenv. Any ideas?

I have used your psycopg2 builds before and never had any issues, however since amazon upgraded Lambda to python 3.8, I cannot get it to work. Where can I find a built psycopg2 library for python 3.8?

The latest version of psycopg2 has improvements like ability to fetch with extras.executevalues(). It would be awesome if you could update this repo to support the latest version.

Many thanks for producing this library.

I got the following error when using the psycopg2-3.7 library. I copied and renamed as per the instructions (I noticed all files were executable with perm 755 which is different to the psycopg2-3.6 version).

{"errorMessage": "Unable to import module 'db_test': /var/task/psycopg2/_psycopg.so: undefined symbol: _PyInterpreterState_Get", "errorType": "Runtime.ImportModuleError"}

The solution for me was to switch to the 3.6 version of the library, which worked fine.

I tried to use this package inside of my lambda function and I am still getting the following error:

"Unable to import module 'redshift_loader': No module named psycopg2"

I also tried building this from scratch per your instructions, but I am unsure of which directory is the newly created package. Some guidance would really be appreciated.

Thank you,

Tyler

No matter which runtime (Python 2.7, 3.6, or 3.7, with or without ssl) I use, I keep getting the following error.

 "errorMessage": "Unable to import module 'lambda_function': /var/task/psycopg2/_psycopg.cpython-37m-x86_64-linux-gnu.so: ELF file's phentsize not the expected size",

I see someone ran into the same message in Issue 36 and was able to fix after this response:

Sounds like you were using a custom compiled library that was compiled on a different platform and imported into lambda

I'm not sure I understand, though. I thought that the intention was that these binaries were precompiled specifically to run in lambda?

When I was trying to make a connection, it gave me error that "module 'psycopg2' has no attribute 'connect'

I followed the instruction, and rename the package to psycopg2.

Here is the sample code:

try:
    conn = psycopg2.connect(connection_string )
except Exception as e:
    print('Error {}'.format(str(e)))

I tried to both use the compiled binary as well as using the following the manual build instruction but I still get a No module named 'psycopg2._psycopg' error on the lambda function. It seem like there is a _psycopg module that is attempting to be imported here:

cd0f954#diff-1166ada1a9105364bc73bd85f308d8e9R50

but that file doesn't seem to exist in the compiled directory. Am I missing something?

I've also gone ahead and dockerized the build process, I was going to submit it in a PR with some instructions in the readme if that would be useful:

FROM lambci/lambda:build-python3.7

ENV \
 POSTGRES_VER=11.4 \
 PSYCOPG_VER=2.8 \
 PSYCOPG_VER_PATH=PSYCOPG-2-8 \
 PYTHON_VER=3.7

RUN yum install -y wget tar makecache fast automake16

#setup output dir for library extraction
RUN mkdir /var/output

# PSYCOPG2
WORKDIR /var/psycopg

RUN wget -nv https://ftp.postgresql.org/pub/source/v${POSTGRES_VER}/postgresql-${POSTGRES_VER}.tar.gz
RUN tar -zxf postgresql-${POSTGRES_VER}.tar.gz
RUN wget -nv http://initd.org/psycopg/tarballs/${PSYCOPG_VER_PATH}/psycopg2-${PSYCOPG_VER}.tar.gz
RUN tar -zxf psycopg2-${PSYCOPG_VER}.tar.gz

# build postgres
RUN cd postgresql-${POSTGRES_VER} && \
./configure --prefix /var/psycopg/postgresql-${POSTGRES_VER} --without-readline --without-zlib --with-openssl && \
make && \
make install

# build psycopg2
# need to replace some values in the config file so they point to postgres install and enable ssl
RUN cd psycopg2-${PSYCOPG_VER} && \
sed -ie "s/pg_config =/pg_config = \/var\/psycopg\/postgresql-$POSTGRES_VER\/bin\/pg_config/g" setup.cfg && \
sed -i 's/static_libpq = 0/static_libpq = 1/g' setup.cfg && \
sed -i 's/libraries =/libraries = ssl crypto/g' setup.cfg
RUN cat psycopg2-${PSYCOPG_VER}/setup.cfg
RUN ls /var/psycopg/postgresql-${POSTGRES_VER}
RUN cd psycopg2-${PSYCOPG_VER} && python setup.py build

# copy compiled library to output to deliever to host
RUN cp -r /var/psycopg/psycopg2-2.8/build/lib.linux-x86_64-${PYTHON_VER}/psycopg2 /var/output

Also, I'm aware of this issue: #23 but seeing as I'm having the issue on lambda, and the change noted above is recent, I figured I'd make an issue.

when importing version 3.7 and running natively, it issues and exception:

import psycopg2 File "/opt/psycopg2/__init__.py", line 68, in <module> import psycopg2.extensions as _ext File "/opt/psycopg2/extensions.py", line 211, in <module> from psycopg2. _range import Range # noqa File "/opt/psycopg2/_range.py", line 502, in <module> oid=3904, subtype_oid=23, array_oid=3905) File "/opt/psycopg2/_range.py", line 283, in __init__ self._create_ranges(pgrange, pyrange) File "/opt/psycopg2/_range.py", line 302, in _create_ranges if isinstance(pgrange, basestring): NameError: name 'basestring' is not defined
a solution is to replace basestring with str

I have an extremely simple app, of which I've made a zip (literally nothing but your psycopg2 dir + a small python script) and yet I'm still getting the import error:

Unable to import module 'lambda_function': No module named psycopg2

Has anything changed here? I'm zipping on a mac and the relevant code looks like:

from __future__ import print_function
from datetime import datetime
from urllib2 import urlopen
import os
import psycopg2

def logData():
    #Define our connection string
    conn_string =  {{connection details ommitted}}

    # get a connection, if a connect cannot be made an exception will be raised here
    conn = psycopg2.connect(conn_string)

    # conn.cursor will return a cursor object, you can use this cursor to perform queries
    cursor = conn.cursor()

def lambda_handler(event, context):
    try:
        logData()
    except:
        print('log data failed!')
        raise
    else:
        print('log data passed!')
        return event['time']
    finally:
        print('data log complete at {}'.format(str(datetime.now())))

lambda function will drop the connection after 5 mins, with error message:
"errorType": "ReadTimeout",
"errorMessage": "HTTPSConnectionPool(host='lambda.us-east-1.amazonaws.com ', port=443): Read timed out. (read timeout=60)"

Hello there -
I am running python 3.6 with sqlalchemy and using this repo as well. I copied over the psycopg2-3.6 directory and renamed it psycopg2.

I am then zipping the file and uploading it to Lambda. However, when I try to run it, I get the following error:
cannot import name 'PGDialect_psycopg2'

Do you know what is happening here?

I didn't have any other way of contacting you. I am having a similar problem with the lxml library and was wondering if you had any thoughts on how to compile that for aws lambda as well. Thanks for the psycopg2 that was a huge help as well.

If I just copy & bundle up the existing psycopg2 directory, then I get errors trying to import it in my Lambda function.

Turns out the permissions on the psycopg2 directory were rwx------ by default.

Fixed by chmod 775 psycopg2

Feedback: It'd be ideal if it either came down with the right permissions, or if the README.md file documented this issue so that new users don't have to experience and debug this issue every.

I'm on Ubuntu 16.04, if it matters.

Trying to get up and running with lambda and postgres. However, running into the below stack trace. I'm using python 3.6 and SQLAlchemy 1.1.14.

/var/task/psycopg2/_psycopg.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8: ImportError
Traceback (most recent call last):
  File "/var/task/hello_world.py", line 9, in handler
    connect_args={'application_name': 'lambda-test'}
  File "/var/task/sqlalchemy/engine/__init__.py", line 391, in create_engine
    return strategy.create(*args, **kwargs)
  File "/var/task/sqlalchemy/engine/strategies.py", line 80, in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
  File "/var/task/sqlalchemy/dialects/postgresql/psycopg2.py", line 554, in dbapi
    import psycopg2
  File "/var/task/psycopg2/__init__.py", line 50, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: /var/task/psycopg2/_psycopg.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8

The artifact I'm deploying to lambda goes something roughly like this:

pip install sqlalchemy
zip site-packages
add awslambda-psycopg2 shim to zip
add source code
deploy

Some of the zip output:

  adding: psycopg2/ (stored 0%)
  adding: psycopg2/__init__.py (deflated 56%)
  adding: psycopg2/_ipaddress.py (deflated 56%)
  adding: psycopg2/_json.py (deflated 65%)
  adding: psycopg2/_psycopg.so (deflated 66%)
  adding: psycopg2/_range.py (deflated 72%)
  adding: psycopg2/errorcodes.py (deflated 65%)
  adding: psycopg2/extensions.py (deflated 59%)
  adding: psycopg2/extras.py (deflated 73%)
  adding: psycopg2/pool.py (deflated 71%)
  adding: psycopg2/psycopg1.py (deflated 60%)
  adding: psycopg2/sql.py (deflated 70%)
  adding: psycopg2/tz.py (deflated 60%)

Contents of the .zip:

Any insight into what I could be missing would be appreciated.

Hi all,
I have connected to AWS RDS Postgres with psycopg2-3.6 but can't use cusor or execute command.

Anyone resolve this issue?
START RequestId: 1932d209-b708-4adb-8690-ce77059c867e Version: $LATEST
'psycopg2.extensions.connection' object has no attribute 'cusor': AttributeError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 13, in lambda_handler
cur = conn.cusor()
AttributeError: 'psycopg2.extensions.connection' object has no attribute 'cusor'

1. Python Runtime: 3.6
2. psycopg2-3.6

Here is my code very simple
`import sys
import logging
import psycopg2
import psycopg2.extras

def lambda_handler(event, context):
conn=psycopg2.connect("dbname=*** host=*** port=5439 user=*** password=***")

#Connect and execute scripts
cur = conn.cusor()
cur.execute("TRUNCATE TABLE dwh_ods.rpt12_bcbs_payroll_report;")
conn.commit()
conn.close()
return "Executed!"`

Hi,
Like many other users I am experiencing issue with Postgres connectivity on Lambda. I have downloaded your psycopg2-3.6 directory and copied it in the root directory of my function. I also did a CHMOD 777 on the directory as it seems that some users reported that to have solved the issue. But I still get the error
Unable to import module 'rds-to-stream': No module named 'psycopg2' on Lambda. also, when I run it locally now I get the error
ImportError: No module named 'psycopg2._psycopg'

Please definitely care about how large the resulting so is. For an interesting discussion on making files smaller: https://serverlesscode.com/post/deploy-scikitlearn-on-lamba/

Hi, I am using Python 3.6 with pyscopg2 and I am trying to run this code on a lambda function. I tried adding both of the folders in this repo to my zip file but neither worked, I keep getting this error:

Unable to import module 'lambda_function': /var/task/psycopg2/_psycopg.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8

Is there something I am doing wrong? Any help would be appreciated, thank you in advance.

When I do import psycopg2, it still says No module named psycopg2. I have even tried adding current dir to syspath and same error.

Thanks for putting this together. So, I'm having a bit of problem getting the connection finalized. I have cloned the repo and copied the psycopg2 directory into the Lambda deployment package per the instructions. When I attempt to connect using host, database, user and password values (which I have verified from another machine are valid), the following error is thrown:

no pg_hba.conf entry for host

I've also attempted to use the with_ssl_support/psycopg2 files and received the following error:

FATAL: no pg_hba.conf entry for host "12.34.56.789", user "my_user", database "foobar", SSL off
: OperationalError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 47, in lambda_handler
conn = psycopg2.connect(host="foobar.com", database="foobar", user="my_user", password="secret")
File "/var/task/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
OperationalError: FATAL: no pg_hba.conf entry for host "12.34.56.789", user "my_user", database "foobar", SSL off

./psycopg/psycopg.h:31:10: fatal error: libpq-fe.h: No such file or directory

Hey there

I've given this a shot but nothing I do will get psycopg2 to run on my Lambda. It never gets past trying to establish the connection and I can’t even get any sort of error message - just complete silence from that point in my code. Ugh.

Would you be willing to upload to github a zipfile of a an AWS Lambda function that you know works and uses psycopg2? Then I could examine it and see what you are doing to make it work.

thanks heaps!

having the same issue with AWS lambda as posted here #32
w
Using sqlalchemy==1.2.14
I copied the exact same zip file to a linux image (ubuntu 18.04.1 and was not able to reproduce there)

any help appreciated!

module 'psycopg2' has no attribute 'paramstyle': AttributeError Traceback (most recent call last): File "/var/task/lambda_function.py", line 264, in lambda_handler create_sql_objects(ENV_VARS['db_connection']) File "/var/task/lambda_function.py", line 257, in create_sql_objects ENGINE = create_engine(engine_string) File "/var/task/sqlalchemy/engine/__init__.py", line 425, in create_engine return strategy.create(*args, **kwargs) File "/var/task/sqlalchemy/engine/strategies.py", line 89, in create dialect = dialect_cls(**dialect_args) File "/var/task/sqlalchemy/dialects/postgresql/psycopg2.py", line 549, in __init__ PGDialect.__init__(self, **kwargs) File "/var/task/sqlalchemy/dialects/postgresql/base.py", line 2212, in __init__ default.DefaultDialect.__init__(self, **kwargs) File "/var/task/sqlalchemy/engine/default.py", line 200, in __init__ self.paramstyle = self.dbapi.paramstyle AttributeError: module 'psycopg2' has no attribute 'paramstyle'

Hi there,

The compilation fails at the step where it involves
python3 setup.py build

for python3.7. Is there any way to run it for python .

Here is the error:

Hello,

Thank you for this project. It has been very useful for our team who are facing a similar problem with AWS Glue containers which don't come with preinstalled C libraries required for psycopg2.

I am interested in knowing more about the contents of the .so file which can be found in psycopg2-3.7/_psycopg.cpython-37m-x86_64-linux-gnu.so. What are the C files packaged in the shared libraries? Our team is looking to create our own .so file to use and maintain so sharing this with us will be extremely valuable.

Thanks a lot for your time!

Unable to import module 'lambda_function': /var/task/psycopg2/_psycopg.so: ELF file's phentsize not the expected size

While trying to catch an error this exception is thrown:

File "/var/task/lambda.py", line 90, in process_asset_id
except psycopg2.errors.UniqueViolation:

This code executes on my windows machine with this verions of psychopg2 installed

pip show psycopg2
Name: psycopg2
Version: 2.8.3
Summary: psycopg2 - Python-PostgreSQL Database Adapter
Home-page: http://initd.org/psycopg/
Author: Federico Di Gregorio
Author-email: [email protected]
License: LGPL with exceptions or ZPL
Location:
Requires:
Required-by:

Hi, I try to run a script try to connect my RDS Postgresql, but there is a error that I can not solve. What should I to do something to deal with it?

[1511405008073] The above exception was the direct cause of the following exception:
[1511405008073] Traceback (most recent call last):
[1511405008073] File "/var/task/flask/app.py", line 1982, in wsgi_app
[1511405008073] response = self.full_dispatch_request()
[1511405008073] File "/var/task/flask/app.py", line 1614, in full_dispatch_request
[1511405008073] rv = self.handle_user_exception(e)
[1511405008073] File "/var/task/flask/app.py", line 1517, in handle_user_exception
[1511405008073] reraise(exc_type, exc_value, tb)
[1511405008073] File "/var/task/flask/_compat.py", line 33, in reraise
[1511405008073] raise value
[1511405008073] File "/var/task/flask/app.py", line 1612, in full_dispatch_request
[1511405008073] rv = self.dispatch_request()
[1511405008073] File "/var/task/flask/app.py", line 1598, in dispatch_request
[1511405008073] return self.view_functions[rule.endpoint](**req.view_args)
[1511405008073] File "/var/task/main.py", line 43, in hello_world
[1511405008073] db.create_all()
[1511405008073] File "/var/task/flask_sqlalchemy/__init__.py", line 963, in create_all
[1511405008073] self._execute_for_all_tables(app, bind, 'create_all')
[1511405008073] File "/var/task/flask_sqlalchemy/__init__.py", line 955, in _execute_for_all_tables
[1511405008073] op(bind=self.get_engine(app, bind), **extra)
[1511405008073] File "/var/task/sqlalchemy/sql/schema.py", line 3949, in create_all
[1511405008073] tables=tables)
[1511405008073] File "/var/task/sqlalchemy/engine/base.py", line 1928, in _run_visitor
[1511405008073] with self._optional_conn_ctx_manager(connection) as conn:
[1511405008073] File "/var/lang/lib/python3.6/contextlib.py", line 82, in __enter__
[1511405008073] return next(self.gen)
[1511405008073] File "/var/task/sqlalchemy/engine/base.py", line 1921, in _optional_conn_ctx_manager
[1511405008073] with self.contextual_connect() as conn:
[1511405008073] File "/var/task/sqlalchemy/engine/base.py", line 2112, in contextual_connect
[1511405008073] self._wrap_pool_connect(self.pool.connect, None),
[1511405008073] File "/var/task/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect
[1511405008073] e, dialect, self)
[1511405008073] File "/var/task/sqlalchemy/engine/base.py", line 1465, in _handle_dbapi_exception_noconnection
[1511405008073] exc_info
[1511405008073] File "/var/task/sqlalchemy/util/compat.py", line 203, in raise_from_cause
[1511405008073] reraise(type(exception), exception, tb=exc_tb, cause=cause)
[1511405008073] File "/var/task/sqlalchemy/util/compat.py", line 186, in reraise
[1511405008073] raise value.with_traceback(tb)
[1511405008073] File "/var/task/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
[1511405008073] return fn()
[1511405008073] File "/var/task/sqlalchemy/pool.py", line 387, in connect
[1511405008073] return _ConnectionFairy._checkout(self)
[1511405008073] File "/var/task/sqlalchemy/pool.py", line 766, in _checkout
[1511405008073] fairy = _ConnectionRecord.checkout(pool)
[1511405008073] File "/var/task/sqlalchemy/pool.py", line 516, in checkout
[1511405008073] rec = pool._do_get()
[1511405008073] File "/var/task/sqlalchemy/pool.py", line 1138, in _do_get
[1511405008073] self._dec_overflow()
[1511405008073] File "/var/task/sqlalchemy/util/langhelpers.py", line 66, in __exit__
[1511405008073] compat.reraise(exc_type, exc_value, exc_tb)
[1511405008073] File "/var/task/sqlalchemy/util/compat.py", line 187, in reraise
[1511405008073] raise value
[1511405008073] File "/var/task/sqlalchemy/pool.py", line 1135, in _do_get
[1511405008073] return self._create_connection()
[1511405008073] File "/var/task/sqlalchemy/pool.py", line 333, in _create_connection
[1511405008073] return _ConnectionRecord(self)
[1511405008073] File "/var/task/sqlalchemy/pool.py", line 461, in __init__
[1511405008073] self.__connect(first_connect_check=True)
[1511405008073] File "/var/task/sqlalchemy/pool.py", line 651, in __connect
[1511405008073] connection = pool._invoke_creator(self)
[1511405008073] File "/var/task/sqlalchemy/engine/strategies.py", line 105, in connect
[1511405008073] return dialect.connect(*cargs, **cparams)
[1511405008073] File "/var/task/sqlalchemy/engine/default.py", line 393, in connect
[1511405008073] return self.dbapi.connect(*cargs, **cparams)
[1511405008073] File "/var/task/psycopg2/__init__.py", line 130, in connect
[1511405008073] conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
[1511405008073] sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "dev.******.rds.amazonaws.com" to address: Temporary failure in name resolution
[1511405008074] [INFO] 2017-11-23T02:43:28.74Z 0997510b-cff8-11e7-afe5-e96d3959eab7 47.52.141.165 - - [23/Nov/2017:02:43:28 +0000] "GET / HTTP/1.1" 500 291 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36" 0/20245.424

I've lambda function with python 3.6 When I deploy it, I get this error:
"errorMessage": "Unable to import module 'lambda_function': /var/task/psycopg2/_psycopg.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8"

python 3.6
psycopg2 for python 3.6 (renamed as psycopg2)

I copied content into my ASW lambda folder. Running python in this folder leads to the following error.

import json
Traceback (most recent call last):
File "", line 1, in
File "/home/ay49514/miniconda3/lib/python3.7/json/init.py", line 106, in
from .decoder import JSONDecoder, JSONDecodeError
File "/home/ay49514/miniconda3/lib/python3.7/json/decoder.py", line 5, in
from json import scanner
File "/home/ay49514/miniconda3/lib/python3.7/json/scanner.py", line 5, in
from _json import make_scanner as c_make_scanner
File "/mnt/home/ay49514/rmbits/src/datamanager/mcfly/_json.py", line 32, in
from psycopg2._psycopg import ISQLQuote, QuotedString
File "/mnt/home/ay49514/rmbits/src/datamanager/mcfly/psycopg2/init.py", line 68, in
from psycopg2 import extensions as _ext
File "/mnt/home/ay49514/rmbits/src/datamanager/mcfly/psycopg2/extensions.py", line 199, in
JSON, JSONARRAY = register_default_json()
File "/mnt/home/ay49514/rmbits/src/datamanager/mcfly/psycopg2/_json.py", line 142, in register_default_json
loads=loads, oid=JSON_OID, array_oid=JSONARRAY_OID)
File "/mnt/home/ay49514/rmbits/src/datamanager/mcfly/psycopg2/_json.py", line 122, in register_json
oid, array_oid, loads=loads, name=name.upper())
File "/mnt/home/ay49514/rmbits/src/datamanager/mcfly/psycopg2/_json.py", line 161, in _create_json_typecasters
loads = json.loads
AttributeError: module 'json' has no attribute 'loads'

Hey @jkehler

I’m getting the error below when I try to run on AWS Lambda:

Unable to import module 'handler': /var/task/psycopg2/_psycopg.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8

I am facing the same issue what other people have faced.
Extracted and Copied psycopg2 in windows into my AWS Lambda zip package along-with my python file and site packages.
As suggested in #3 , changed the permission, transferred file back to windows and zipped again into my AWS Lambda package.
Below is my code snippet

`#!/usr/bin/python
import psycopg2
import sys
import pprint
import datetime

def lambda_handler(event, context):

Connect to RedShift

conn_string = "dbname='XXXX' port='5439' user='XXX' password='XXXX' host='XXXXXXXXXXXX'";

conn = psycopg2.connect(conn_string);
cursor = conn.cursor();

cursor.execute("begin transaction");
cursor.execute("truncate table XXXX");
cursor.execute("truncate table XXXX");
cursor.execute("truncate table XXXX");
cursor.execute("delete from XXXX");
cursor.execute("insert into XXXX");
cursor.execute("truncate table XXXX");
cursor.execute("truncate table XXXX");
cursor.execute("truncate table XXXX");
cursor.execute("end transaction");

conn.commit();
conn.close();`

My zip structure
. +-- python_script.py +-- psycopg2 +-- psycopg2 files

Is it something that I am doing wrong, did I miss anything?

cant seem to get this worked out. I saw that it was opened and promptly closed... any thoughts ?

I get this error when trying to run the lambda script locally with your folder included in the root

now that Lambda supports python 3.6 https://aws.amazon.com/about-aws/whats-new/2017/04/aws-lambda-supports-python-3-6/

is there a chance you could make a 2nd package that we could use in our 3.6 lambdas?

Thanks

Hello,
i called my .py script : "lambda_function.py" ( the handler in the config tab of lambda console is : lambda_function.lambda_handler so i guess it's ok ) ... my zip structure of the package to upload is the same as shown by kylebuckingham in another issue closed...
i have : 1 file + psycopg2 lib ... the goal of my simple script is to trigger a scheduled vaacuum on a pgsql table

my zip structure in detail :

.
+-- lambda_function.py
+-- psycopg2
    +-- psycopg2 files

here is a snippet of my code [lambda_function.py]:

from __future__ import print_function
from datetime import datetime
import os
import psycopg2

con = None
query = ''

def lambda_handler(event, context):
    print('Scheduled maintenance started at {}'.format(str(datetime.now())))
    try:
        host = "THE DB HOST"
        database = "DB NAME"
        user = "DBUSER"
        password = "DBPASS"
        days = "7"

        con = psycopg2.connect(host=host, database=database, user=user, password=password) 
        cur = con.cursor()
        cur.execute('SELECT ..... ' + 
                    'FROM pg_stat_all_tables ' +
                    "WHERE schemaname = 'foo' " +
                    "AND relname = 'bar' " +
                    'AND ((last_analyze is NULL ' +
                    'AND last_autoanalyze is NULL)' +
                    'OR ((last_analyze < last_autoanalyze OR last_analyze is null) ' +
                    "AND last_autoanalyze < now() - interval %s) " +
                    'OR ((last_autoanalyze < last_analyze OR last_autoanalyze is null) ' +
                    "AND last_analyze < now() - interval %s));", [days + ' day', days + ' day'])
        rows = cur.fetchall()
        con.set_isolation_level(0)
        for row in rows:
            query = 'VACUUM ANALYZE %s;' % (row[0])
            cur.execute(query)

    except psycopg2.DatabaseError, e:
        print('Error: %s' % e)
        raise
    else:
        print('Vaacuum done!')
        return event['time']
    finally:
        print('Scheduled maintenance completed at {}'.format(str(datetime.now())))
        if con:
            con.close()

The error when testing in lambda: "Unable to import module 'lambda_function': No module named psycopg2"

Any clue on why is not working for me ? Is this the correct way to import the module ? Thank you very much !

When I deploy it, I get this error.

I checked _psycopg module but there is no this module.
I used psycopg2-3.7

I hope your help.

https://github.com/Miserlou/lambda-packages/

Using sqlalchemy 1.1.14 and psycopg2-3.6 package renamed as psycopg2 in aws lambda and keep seeing:

File "/var/task/sqlalchemy/engine/__init__.py", line 391, in create_engine
return strategy.create(*args, **kwargs)
File "/var/task/sqlalchemy/engine/strategies.py", line 88, in create
dialect = dialect_cls(**dialect_args)
File "/var/task/sqlalchemy/dialects/postgresql/psycopg2.py", line 521, in __init__
PGDialect.__init__(self, **kwargs)
File "/var/task/sqlalchemy/dialects/postgresql/base.py", line 2083, in __init__
default.DefaultDialect.__init__(self, **kwargs)
File "/var/task/sqlalchemy/engine/default.py", line 196, in __init__
self.paramstyle = self.dbapi.paramstyle
AttributeError: module 'psycopg2' has no attribute 'paramstyle'

I've tried using different versions of sqlalchemy with no luck.

Just curious, why do you configure PostgreSQL without readline and zlib in this situation?

Getting this error when the lib is loading

 import psycopg2
   File "/.....path to lambda....../lambdas/psycopg2/__init__.py", line 50, in <module>
     from psycopg2._psycopg import (                     # noqa
 ImportError: Error loading shared library libssl.so.10: No such file or directory (needed by /.....path to lambda....../lambdas/psycopg2/_psycopg.cpython-37m-x86_64-linux-gnu.so)

just copied the psycopg2-3.7 folder into lambda directory and renamed it to psycopg2
python version 3.7

Hi @jkehler, community,

Thanks for creating and maintaining this life saver package !

I'm working with python3.6 and have copied the pyscopg2-3.6 folder to my root and renamed it to pscopg2. But even after doing so, the app was not able to import psycopg (this was the error : Driver "postgres" requires "psycopg2" package)

I fired a terminal and tried to import psycopg2 manually, but it failed with the following error :

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../psycopg2/__init__.py", line 50, in <module>
    from psycopg2._psycopg import (                     # noqa
ModuleNotFoundError: No module named 'psycopg2._psycopg'

Here are the contents of my psycopg2 folder :

What am I doing wrong here ?
How should I go about it ?

Thanks

PS : I'm on macOS running python3.6.

Would be great to have a version with SSL support. The OpenSSL libs are included on the Amazon AMI already. AFAICT something like this seems to work:

• yum install openssl-devel (OpenSSL headers)
• (for postgresql) /configure --prefix=$(pwd) --without-readline --without-zlib --with-openssl
• (for psycopg2) add libraries=ssl crypto to setup.cfg

Then maybe also bundle the Redshift CA certificate (http://docs.aws.amazon.com/redshift/latest/mgmt/connecting-ssl-support.html#connect-using-ssl) to the psycopg2/ directory?

Then can use via:

db = psycopg2.connect(
    ...,
    sslmode='verify-ca',
    sslrootcert=os.path.join(os.path.dirname(os.path.realpath(__file__)), 'psycopg2/redshift-ssl-ca-cert.pem')
)

Hi @jkehler

I get a next error when I'm trying to use the library:

START RequestId: b7d75edc-9af5-11e8-a42b-719aa1d42326 Version: 6
Unable to import module 'index': /var/task/psycopg2/_psycopg.so: undefined symbol: _Py_ZeroStruct

END RequestId: b7d75edc-9af5-11e8-a42b-719aa1d42326
REPORT RequestId: b7d75edc-9af5-11e8-a42b-719aa1d42326	Duration: 1.15 ms	Billed Duration: 100 ms 	Memory Size: 128 MB	Max Memory Used: 34 MB

I also tried to compile lib with postgres 10.4 and latest psycopg2 and get the same error.

I see the awslabs repo has an example lambda function that import psycopg2 and there's no fancy make files or things to suggest they are bundling the C library, maybe it's finally supported ?

See:
https://github.com/awslabs/rds-support-tools/blob/master/serverless/serverless-query.py.postgresql

in aws lambda, #32 #14 realted issue is still happening. i tried to move psydopg2 package folder into /application or outside, it won't work either.
always throws same error. any solutions?