ADLFS (Azure Data Lake FileSystem)¶
Description:
- Interact with Azure Data Lake
Resources:
- [GitHub ADLFS] https://github.com/fsspec/adlfs/blob/main/adlfs/spec.py#L69
- [GitHub] https://github.com/fsspec/adlfs
- [Docs] https://filesystem-spec.readthedocs.io/en/latest/api.html#user-functions
- [Docs] https://filesystem-spec.readthedocs.io/en/latest/api.html#built-in-implementations
- [Docs] https://filesystem-spec.readthedocs.io/en/latest/api.html#base-classes
- [Docs] https://filesystem-spec.readthedocs.io/en/latest/api.html#other-known-implementations
- [GitHub] https://github.com/fsspec/adlfs/blob/main/adlfs/spec.py#L328
Environment variables¶
This notebook reads its settings from environment variables. Set them before starting Jupyter:
ADLS_STORE_NAME: name of your Azure Data Lake storeAZURE_TENANT_ID: Azure AD tenant IDAZURE_CLIENT_ID: client ID of a service principal with access to the storeAZURE_CLIENT_SECRET: client secret of that service principal
Example 1¶
List the contents of a Azure store
In [ ]:
Copied!
import os
from fsspec import filesystem
from adlfs import AzureDatalakeFileSystem
# Note: this is the same as:
# adl = filesystem("adl")
# abfss = filesystem("abfss")
store_name = os.environ["ADLS_STORE_NAME"]
tenant_id = os.environ["AZURE_TENANT_ID"]
client_id = os.environ["AZURE_CLIENT_ID"]
client_secret = os.environ["AZURE_CLIENT_SECRET"]
adl = AzureDatalakeFileSystem(store_name=store_name, tenant_id=tenant_id, client_id=client_id, client_secret=client_secret)
adl.ls('')
import os
from fsspec import filesystem
from adlfs import AzureDatalakeFileSystem
# Note: this is the same as:
# adl = filesystem("adl")
# abfss = filesystem("abfss")
store_name = os.environ["ADLS_STORE_NAME"]
tenant_id = os.environ["AZURE_TENANT_ID"]
client_id = os.environ["AZURE_CLIENT_ID"]
client_secret = os.environ["AZURE_CLIENT_SECRET"]
adl = AzureDatalakeFileSystem(store_name=store_name, tenant_id=tenant_id, client_id=client_id, client_secret=client_secret)
adl.ls('')