Environment variables¶
This notebook reads its settings from environment variables. Set them before starting Jupyter:
S3_BUCKET: name of an S3 bucket you can access
Upload a CSV file to example/data.csv and a Parquet file to example/data.parquet in that bucket, or change the keys in the cells. AWS credentials come from the standard AWS configuration (environment variables, ~/.aws, or an instance role).
Example 1¶
Read csv data into a dask dataframe from s3
In [ ]:
Copied!
import os
import dask.dataframe as dd
df = dd.read_csv(f"s3://{os.environ['S3_BUCKET']}/example/data.csv")
df.head()
import os
import dask.dataframe as dd
df = dd.read_csv(f"s3://{os.environ['S3_BUCKET']}/example/data.csv")
df.head()
Example 2¶
Read parquet data into a dask dataframe from S3
In [ ]:
Copied!
import os
import dask.dataframe as dd
df = dd.read_parquet(f"s3://{os.environ['S3_BUCKET']}/example/data.parquet")
df.head()
import os
import dask.dataframe as dd
df = dd.read_parquet(f"s3://{os.environ['S3_BUCKET']}/example/data.parquet")
df.head()