Azure Storage API¶
Description:
- Work with azure blob storage
Resources:
- https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-list-python
- https://stackoverflow.com/questions/51145124/how-to-list-all-blobs-inside-of-a-specific-subdirectory-in-azure-cloud-storage-u
- https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python?tabs=managed-identity%2Croles-azure-portal%2Csign-in-azure-cli
- https://learn.microsoft.com/en-us/azure/open-datasets/dataset-catalog
- https://learn.microsoft.com/en-us/azure/open-datasets/dataset-taxi-yellow?tabs=azureml-opendatasets
- https://github.com/Azure/azure-sdk-for-python
Environment variables¶
This notebook reads these environment variables. Set them before you start Jupyter:
AZURE_STORAGE_ACCOUNT_NAME: your Azure Storage account name.AZURE_STORAGE_CONTAINER: a blob container in that storage account.AZURE_STORAGE_ACCOUNT_KEY: an access key for that storage account.
Example 1: Synchronous - Private Data¶
Synchronously list the blobs in a private container
In [ ]:
Copied!
import os
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
account_name = os.environ["AZURE_STORAGE_ACCOUNT_NAME"]
container_name = os.environ["AZURE_STORAGE_CONTAINER"]
account_key = os.environ["AZURE_STORAGE_ACCOUNT_KEY"]
account_url = f"https://{account_name}.blob.core.windows.net/{container_name}"
storage_options = {'account_name': account_name, 'account_key': account_key, 'account_url': account_url, 'container_name': container_name}
container_client = ContainerClient(**storage_options)
container_client.list_blobs()
import os
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
account_name = os.environ["AZURE_STORAGE_ACCOUNT_NAME"]
container_name = os.environ["AZURE_STORAGE_CONTAINER"]
account_key = os.environ["AZURE_STORAGE_ACCOUNT_KEY"]
account_url = f"https://{account_name}.blob.core.windows.net/{container_name}"
storage_options = {'account_name': account_name, 'account_key': account_key, 'account_url': account_url, 'container_name': container_name}
container_client = ContainerClient(**storage_options)
container_client.list_blobs()
Example 2: Async - Public Data¶
Asyncronously list the blobs in a public container
In [ ]:
Copied!
from azure.storage.blob.aio import BlobServiceClient, ContainerClient, BlobPrefix
azure_storage_account_name = "azureopendatastorage"
azure_storage_account_url = f"https://{azure_storage_account_name}.blob.core.windows.net/"
azure_storage_sas_token = r""
container_name = "nyctlc"
folder_name = "yellow"
c_client = ContainerClient(account_name=azure_storage_account_name, container_name=container_name, account_url=azure_storage_account_url)
c_client.list_blobs()
from azure.storage.blob.aio import BlobServiceClient, ContainerClient, BlobPrefix
azure_storage_account_name = "azureopendatastorage"
azure_storage_account_url = f"https://{azure_storage_account_name}.blob.core.windows.net/"
azure_storage_sas_token = r""
container_name = "nyctlc"
folder_name = "yellow"
c_client = ContainerClient(account_name=azure_storage_account_name, container_name=container_name, account_url=azure_storage_account_url)
c_client.list_blobs()