GCSFS (and GS)¶
Description:
- List objects from google cloud blob storage
Resources:
- [Google Auth Docs] https://google-auth.readthedocs.io/en/master/user-guide.html
- [Gcfs Docs] https://gcsfs.readthedocs.io/en/latest/
- [Gcfs Docs on Credentials] https://gcsfs.readthedocs.io/en/latest/#credentials
- [Open Source Data on Google] https://cloud.google.com/storage/docs/public-datasets
Environment variables¶
This notebook reads its settings from environment variables. Set them before starting Jupyter:
GOOGLE_APPLICATION_CREDENTIALS: path to a Google Cloud service-account JSON key fileGCS_PROJECT: Google Cloud project IDGCS_BUCKET: name of a Cloud Storage bucket in that project
Example 2 reads an object named example.jpeg from the bucket.
Example 1¶
Listing objects in a bucket
In [ ]:
Copied!
import os
import gcsfs
gcsfs_creds = os.environ["GOOGLE_APPLICATION_CREDENTIALS"] # path to a service-account JSON key file
gcsfs_project = os.environ["GCS_PROJECT"]
gcsfs_bucket = os.environ["GCS_BUCKET"]
fs = gcsfs.GCSFileSystem(project=gcsfs_project, token=gcsfs_creds)
fs.ls(gcsfs_bucket)
# Note: the above is equivalent to fsspec.filesystem('gs', ...), which returns the same GCSFileSystem:
# from fsspec import filesystem
# fs = filesystem("gs", project=gcsfs_project, token=gcsfs_creds)
# fs.ls(gcsfs_bucket)
import os
import gcsfs
gcsfs_creds = os.environ["GOOGLE_APPLICATION_CREDENTIALS"] # path to a service-account JSON key file
gcsfs_project = os.environ["GCS_PROJECT"]
gcsfs_bucket = os.environ["GCS_BUCKET"]
fs = gcsfs.GCSFileSystem(project=gcsfs_project, token=gcsfs_creds)
fs.ls(gcsfs_bucket)
# Note: the above is equivalent to fsspec.filesystem('gs', ...), which returns the same GCSFileSystem:
# from fsspec import filesystem
# fs = filesystem("gs", project=gcsfs_project, token=gcsfs_creds)
# fs.ls(gcsfs_bucket)
Example 2¶
Read an object from google cloud blob storage
In [ ]:
Copied!
import os
import gcsfs
gcsfs_creds = os.environ["GOOGLE_APPLICATION_CREDENTIALS"] # path to a service-account JSON key file
gcsfs_project = os.environ["GCS_PROJECT"]
gcsfs_bucket = os.environ["GCS_BUCKET"]
fs = gcsfs.GCSFileSystem(project=gcsfs_project, token=gcsfs_creds)
with fs.open(f'{gcsfs_bucket}/example.jpeg', 'rb') as f:
print(f.read())
import os
import gcsfs
gcsfs_creds = os.environ["GOOGLE_APPLICATION_CREDENTIALS"] # path to a service-account JSON key file
gcsfs_project = os.environ["GCS_PROJECT"]
gcsfs_bucket = os.environ["GCS_BUCKET"]
fs = gcsfs.GCSFileSystem(project=gcsfs_project, token=gcsfs_creds)
with fs.open(f'{gcsfs_bucket}/example.jpeg', 'rb') as f:
print(f.read())