GDriveFS¶
Description:
- Work with objects in google drive through fsspec, with the
gdrive_fsspecpackage (pip install gdrive-fsspec; the unrelated PyPI packagegdrivefsis a FUSE tool)
Resources:
- https://github.com/fsspec/gdrive-fsspec
- https://github.com/fsspec/gdrive-fsspec/blob/master/src/gdrive_fsspec/core.py
- https://stackoverflow.com/questions/45492703/google-drive-api-oauth-and-service-account
Notes:
- You can get the
root_file_idfrom the URL of that folder in Google Drive, for examplehttps://drive.google.com/open?id=<FILE_ID>.
NOTE: on a headless machine (a remote server or a cluster node) you have to provide credentials directly and cannot rely on token="browser", because no browser is available to finish the sign-in. The resources above describe the options for getting credentials.
Environment variables¶
This notebook reads its settings from environment variables. Set them before starting Jupyter:
GDRIVE_SERVICE_ACCOUNT_JSON: path to a Google service-account JSON key file with access to the folderGDRIVE_ROOT_FILE_ID: ID of the Google Drive folder to use as the root
Example 1¶
List objects in google drive
In [ ]:
Copied!
import json
import os
import xarray as xr
from gdrive_fsspec import GoogleDriveFileSystem
# Service-account key (JSON) for an account that can read the folder
with open(os.environ["GDRIVE_SERVICE_ACCOUNT_JSON"]) as f:
gdrive_creds = json.load(f)
token = 'service_account'
root_file_id = os.environ["GDRIVE_ROOT_FILE_ID"]
gdfs = GoogleDriveFileSystem(token=token, root_file_id=root_file_id, creds=gdrive_creds)
gdfs.ls("", detail=True)
import json
import os
import xarray as xr
from gdrive_fsspec import GoogleDriveFileSystem
# Service-account key (JSON) for an account that can read the folder
with open(os.environ["GDRIVE_SERVICE_ACCOUNT_JSON"]) as f:
gdrive_creds = json.load(f)
token = 'service_account'
root_file_id = os.environ["GDRIVE_ROOT_FILE_ID"]
gdfs = GoogleDriveFileSystem(token=token, root_file_id=root_file_id, creds=gdrive_creds)
gdfs.ls("", detail=True)
Example 2¶
If objects like zarr are stored in gdrive, open them and compute
In [ ]:
Copied!
mapper = gdfs.get_mapper("")
ds = xr.open_zarr(mapper, consolidated=True, decode_times=False, chunks=False)
dsl = ds.compute()
dsl.nbytes / 1e6
mapper = gdfs.get_mapper("")
ds = xr.open_zarr(mapper, consolidated=True, decode_times=False, chunks=False)
dsl = ds.compute()
dsl.nbytes / 1e6