Pandas on AWS
An AWS Professional Service open source initiative | [email protected]
Source | Downloads | Installation Command |
---|---|---|
PyPi | pip install awswrangler |
|
Conda | conda install -c conda-forge awswrangler |
Installation command: pip install awswrangler
import awswrangler as wr
import pandas as pd
df = pd.DataFrame({"id": [1, 2], "value": ["foo", "boo"]})
# Storing data on Data Lake
wr.s3.to_parquet(
df=df,
path="s3://bucket/dataset/",
dataset=True,
database="my_db",
table="my_table"
)
# Retrieving the data directly from Amazon S3
df = wr.s3.read_parquet("s3://bucket/dataset/", dataset=True)
# Retrieving the data from Amazon Athena
df = wr.athena.read_sql_query("SELECT * FROM my_table", database="my_db")
# Get Redshift connection (SQLAlchemy) from Glue and retrieving data from Redshift Spectrum
engine = wr.catalog.get_engine("my-redshift-connection")
df = wr.db.read_sql_query("SELECT * FROM external_schema.my_table", con=engine)
# Get MySQL connection (SQLAlchemy) from Glue Catalog and LOAD the data into MySQL
engine = wr.catalog.get_engine("my-mysql-connection")
wr.db.to_sql(df, engine, schema="test", name="my_table")
# Get PostgreSQL connection (SQLAlchemy) from Glue Catalog and LOAD the data into PostgreSQL
engine = wr.catalog.get_engine("my-postgresql-connection")
wr.db.to_sql(df, engine, schema="test", name="my_table")
- What is AWS Data Wrangler?
- Install
- Tutorials
- 001 - Introduction
- 002 - Sessions
- 003 - Amazon S3
- 004 - Parquet Datasets
- 005 - Glue Catalog
- 006 - Amazon Athena
- 007 - Databases (Redshift, MySQL and PostgreSQL)
- 008 - Redshift - Copy & Unload.ipynb
- 009 - Redshift - Append, Overwrite and Upsert
- 010 - Parquet Crawler
- 011 - CSV Datasets
- 012 - CSV Crawler
- 013 - Merging Datasets on S3
- 014 - Schema Evolution
- 015 - EMR
- 016 - EMR & Docker
- 017 - Partition Projection
- 018 - QuickSight
- 019 - Athena Cache
- 020 - Spark Table Interoperability
- 021 - Global Configurations
- 022 - Writing Partitions Concurrently
- 023 - Flexible Partitions Filter
- 024 - Athena Query Metadata
- 025 - Redshift - Loading Parquet files with Spectrum
- API Reference
- License
- Contributing
- Legacy Docs (pre-1.0.0)
Please send a Pull Request with your resource reference and @githubhandle.
- Optimize Python ETL by extending Pandas with AWS Data Wrangler [@igorborgest]
- Reading Parquet Files With AWS Lambda [@anand086]
- Transform AWS CloudTrail data using AWS Data Wrangler [@anand086]
- Rename Glue Tables using AWS Data Wrangler [@anand086]
- Getting started on AWS Data Wrangler and Athena [@dheerajsharma21]
- Simplifying Pandas integration with AWS data related services [@bvsubhash]
Knowing which companies are using this library is important to help prioritize the project internally.
Please send a Pull Request with your company name and @githubhandle if you may.
- Amazon
- AWS
- Cepsa [@alvaropc]
- Cognitivo [@msantino]
- Digio [@afonsomy]
- DNX [@DNXLabs]
- Funcional Health Tech [@webysther]
- Informa Markets [@mateusmorato]
- LINE TV [@bryanyang0528]
- M4U [@Thiago-Dantas]
- nrd.io [@mrtns]
- OKRA Technologies [@JPFrancoia, @schot]
- Pier [@flaviomax]
- Pismo [@msantino]
- ringDNA [@msropp]
- Serasa Experian [@andre-marcos-perez]
- Shipwell [@zacharycarter]
- strongDM [@mrtns]
- Thinkbumblebee [@dheerajsharma21]
- Zillow [@nicholas-miles]