What it is

AWS SDK for pandas is a Python library for working with pandas DataFrames on AWS. It is published as awswrangler and is listed under Data & Analytics / Data Engineering & Integration. The project presents itself as Pandas on AWS and is an AWS Professional Service open source initiative under the Apache-2.0 license.

The concrete problem it addresses is moving tabular data between pandas and AWS data services through library calls. The README shows calls for storing and reading Parquet data in S3, querying Athena, connecting to Redshift through a Glue connection, and writing or querying Timestream. The topic list places it in data engineering, ETL, data science, AWS, Athena, Glue, Lambda, EMR, SageMaker notebooks, Apache Arrow, and Apache Parquet.

Key capabilities

• The library reads and writes pandas DataFrames to Amazon S3 in Parquet, CSV, JSON, and Excel formats, as the README shows for Parquet datasets.
• It runs SQL queries against Amazon Athena and returns results as pandas DataFrames, as shown by wr.athena.read_sql_query.
• It connects to Amazon Redshift through AWS Glue catalog connections and reads SQL query results with wr.redshift.connect and wr.redshift.read_sql_query.
• It writes time-series records to Amazon Timestream and queries them with wr.timestream.write and wr.timestream.query.
• It integrates with services and databases named in the description, including OpenSearch, Neptune, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, and SQLServer.
• It can distribute workloads at scale by leveraging Modin and Ray over a cluster of workers.

Who uses it and how

• Data engineers use it to move data between S3, Athena, Redshift, and Timestream inside AWS ETL workflows.
• Data scientists use it in Python environments such as SageMaker notebooks, because the topic list includes amazon-sagemaker-notebook.
• AWS Lambda users can use it in Python functions, because the topic list includes aws-lambda.
• Teams building data lakes use it to write Parquet datasets to S3 and register them with database and table names in calls such as wr.s3.to_parquet.
• Teams operating EMR or AWS analytics stacks use it to query services such as Athena and Redshift from pandas code.

Getting started

Typical installation is pip install awswrangler or conda install -c conda-forge awswrangler. Starting with version 3.0, optional modules must be installed explicitly, for example pip install 'awswrangler[redshift]'.

When to use it — and when not to

Use it when the workload runs on AWS and the team works in Python with pandas. It provides calls for S3, Athena, Redshift, Timestream, Glue, and other listed AWS services. Do not use it when the environment is not AWS, because the named integrations are AWS services or catalog connections. Version 3.0 separates optional modules, so users must install extra packages for features such as Redshift.

import awswrangler as wr
import pandas as pd
from datetime import datetime

df = pd.DataFrame({"id": [1, 2], "value": ["foo", "boo"]})

# Storing data on Data Lake
wr.s3.to_parquet(df=df, path="s3://bucket/dataset/", dataset=True, database="my_db", table="my_table")

# Retrieving the data directly from Amazon S3
df = wr.s3.read_parquet("s3://bucket/dataset/", dataset=True)

# Retrieving the data from Amazon Athena
df = wr.athena.read_sql_query("SELECT * FROM my_table", database="my_db")

# Get a Redshift connection from Glue Catalog and retrieving data from Redshift Spectrum
con = wr.redshift.connect("my-glue-connection")
df = wr.redshift.read_sql_query("SELECT * FROM external_schema.my_table", con=con)
con.close()

# Amazon Timestream Write
df = pd.DataFrame(
    {
        "time": [datetime.now(), datetime.now()],
        "my_dimension": ["foo", "boo"],
        "measure": [1.0, 1.1],
    }
)
rejected_records = wr.timestream.write(
    df,
    database="sampleDB",
    table="sampleTable",
    time_col="time",
    measure_col="measure",
    dimensions_cols=["my_dimension"],
)

# Amazon Timestream Query
wr.timestream.query("""
SELECT time, measure_value::double, my_dimension
FROM "sampleDB"."sampleTable" ORDER BY time DESC LIMIT 3
""")