AWS Wrangler¶
Description:
- Pandas on AWS: read and write pandas DataFrames in Amazon S3 and other AWS data services.
Resources:
Environment variables¶
This notebook reads these environment variables. Set them before you start Jupyter:
S3_BUCKET: name of an S3 bucket you can access that holds the sample objects named in the code. AWS credentials come from the usual AWS configuration (environment variables,~/.aws, or an instance role).
Example 3 writes to and then reads from s3://$S3_BUCKET/test/.
Example 1¶
Create a session using the default Boto3 Session
In [ ]:
Copied!
import awswrangler as wr
import boto3
boto3.setup_default_session(region_name="us-east-2")
wr.s3.does_object_exist("s3://noaa-ghcn-pds/fake")
import awswrangler as wr
import boto3
boto3.setup_default_session(region_name="us-east-2")
wr.s3.does_object_exist("s3://noaa-ghcn-pds/fake")
Example 2¶
Create and customize a session using a custom Boto3 Session
In [ ]:
Copied!
import awswrangler as wr
import boto3
my_session = boto3.Session(region_name="us-east-2")
wr.s3.does_object_exist("s3://noaa-ghcn-pds/fake", boto3_session=my_session)
import awswrangler as wr
import boto3
my_session = boto3.Session(region_name="us-east-2")
wr.s3.does_object_exist("s3://noaa-ghcn-pds/fake", boto3_session=my_session)
Example 3¶
Create a pandas dataframe
In [ ]:
Copied!
import os
from datetime import date
import awswrangler as wr
import pandas as pd
bucket = os.environ["S3_BUCKET"]
path = f"s3://{bucket}/test/"
df = pd.DataFrame({
"id": [1, 2],
"value": ["foo", "boo"],
"date": [date(2020, 1, 1), date(2020, 1, 2)]
})
wr.s3.to_parquet(
df=df,
path=path,
dataset=True,
mode="overwrite"
)
data = wr.s3.read_parquet(path, dataset=True)
print(data)
import os
from datetime import date
import awswrangler as wr
import pandas as pd
bucket = os.environ["S3_BUCKET"]
path = f"s3://{bucket}/test/"
df = pd.DataFrame({
"id": [1, 2],
"value": ["foo", "boo"],
"date": [date(2020, 1, 1), date(2020, 1, 2)]
})
wr.s3.to_parquet(
df=df,
path=path,
dataset=True,
mode="overwrite"
)
data = wr.s3.read_parquet(path, dataset=True)
print(data)