Authenticating in Google¶
There are numerous ways to authenticate with google and google cloud. Below are a few examples.
Source and license:
- Some of these examples are adapted from the google-auth-oauthlib documentation (the custom
Flowexample) (https://google-auth-oauthlib.readthedocs.io/), Copyright Google LLC, licensed under Apache-2.0. That code stays under its own license; see THIRD_PARTY_NOTICES.md.
Environment variables¶
This notebook reads these environment variables. Set them before you start Jupyter:
GOOGLE_CLOUD_PROJECT: your Google Cloud project id.GOOGLE_APPLICATION_CREDENTIALS: optional, the path of a service-account.jsonkey file. If it is not set, the first code cell writes the key you paste into it to a temporary file and sets this variable for you.GOOGLE_IMPERSONATE_SERVICE_ACCOUNT: optional, used only by the Impersonated Credentials example: the email of the service account to impersonate.
Set Service Account Default Credentials [Recommended]¶
Set default credentials to be used by Google's SDKs by
- Create a
service accountin the google cloud console - Create a new
keyfor thatservice accountand download its.jsonfile - Save the JSON key to a file on the machine running Jupyter (outside the repository)
- Set an environment variable named
GOOGLE_APPLICATION_CREDENTIALSto the path of your.jsonfile before you start Jupyter
If you cannot set the variable before starting Jupyter, paste the contents of your .json file into the cell below instead. The cell writes the key to a new file that only you can read, under /tmp/ganymede_demo/ (never inside the repository checkout), and sets GOOGLE_APPLICATION_CREDENTIALS for this session. Never commit a notebook that contains a real key.
This is the simplest and most straightforward way to get started.
import os
import json
import tempfile
from pathlib import Path
# Service account credentials: paste the contents of your key file here,
# unless GOOGLE_APPLICATION_CREDENTIALS already points at the key file
google_service_account_creds = {
"type": "service_account",
"project_id": "XXXXX",
"private_key_id": "XXXXXX",
"private_key": "XXXXXX",
"client_email": "XXXXXX",
"client_id": "XXXXXX",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "XXXXXX",
"universe_domain": "googleapis.com"
}
if "GOOGLE_APPLICATION_CREDENTIALS" in os.environ:
# Use the key file you pointed at before starting Jupyter
json_fp = os.environ["GOOGLE_APPLICATION_CREDENTIALS"]
else:
# Save the pasted key outside the repository, in a new file only you can read
tmp_dir = Path("/tmp/ganymede_demo")
tmp_dir.mkdir(parents=True, exist_ok=True)
fd, json_fp = tempfile.mkstemp(prefix="gcp_creds_", suffix=".json", dir=tmp_dir)
with os.fdopen(fd, "w") as file:
json.dump(google_service_account_creds, file)
# Set the env variable google will use for default creds
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = json_fp
# Now you can use default credentials (see below)
Application Default Credentials¶
Resources:
- https://cloud.google.com/docs/authentication/provide-credentials-adc
- https://google-auth.readthedocs.io/en/master/user-guide.html
Note: you will not be able to use this prior to setting default service account credentials above.
You must perform that step first. Then, the library can now automatically choose the right type of client and initialize credentials from the context provided in the configuration file.
import google.auth
credentials, project = google.auth.default()
Using Non-default Creds Option 1¶
- Use your service account default credentials you can create a
clientto use with various google cloud apis. - Follow the same steps above to save creds to a local file, then do the below:
# Create a `Client` from service account default credentials
from google.cloud.storage.client import Client
client = Client.from_service_account_json(json_fp)
Using Non-default Creds Option 2¶
- Use an existing or create a new service account in your project
- Create a new key in your service account
- Save the key as in the first example, then read the key file to get the serialized key data
- Pass in the serialized key data to
key_datainee.ServiceAccountCredentials
import json
import os
from pathlib import Path
import ee
# Serialized key data, read from the key file saved in the first example
serialized_creds = Path(json_fp).read_text()
# Service account email, taken from the key itself
service_account = json.loads(serialized_creds)["client_email"]
# Create SA creds
credentials = ee.ServiceAccountCredentials(service_account, key_data=serialized_creds)
# Initialize
ee.Initialize(credentials, project=os.environ["GOOGLE_CLOUD_PROJECT"])
Using Non-default Creds Option 3¶
Resources:
https://google-auth.readthedocs.io/en/stable/reference/google.oauth2.service_account.html
https://gcloud.readthedocs.io/en/latest/google-cloud-auth.html#client-provided-authentication
Create a
Credentialsobject from your service account .json file
from google.oauth2 import service_account
# Build credentials from the key file saved in the first example
sa_credentials = service_account.Credentials.from_service_account_file(
json_fp,
scopes=["https://www.googleapis.com/auth/cloud-platform"],
)
print(sa_credentials.service_account_email)
Impersonated Credentials¶
Note: you will not be able to use this prior to setting default service account credentials above.
You must perform that step first. You must also have had to grant source credentials in the "Service Account Token Creator" IAM Role.
Then, you can use impersonated credentials.
import os
from google.auth import impersonated_credentials
from google.cloud import storage
from google.oauth2 import service_account
target_scopes = ['https://www.googleapis.com/auth/devstorage.read_only']
source_credentials = service_account.Credentials.from_service_account_file(
json_fp,
scopes=['https://www.googleapis.com/auth/cloud-platform'])
target_credentials = impersonated_credentials.Credentials(
source_credentials=source_credentials,
target_principal=os.environ["GOOGLE_IMPERSONATE_SERVICE_ACCOUNT"],
target_scopes=target_scopes,
lifetime=500)
client = storage.Client(project=os.environ["GOOGLE_CLOUD_PROJECT"], credentials=target_credentials)
buckets = client.list_buckets()
for bucket in buckets:
print(bucket.name)
Create User Credentials [More advanced, not recommended]¶
Resources
- https://google-auth.readthedocs.io/en/master/user-guide.html
- https://oauthlib.readthedocs.io/en/latest/oauth2/tokens/bearer.html
- https://requests-oauthlib.readthedocs.io/en/latest/oauth2_workflow.html
- https://github.com/googleapis/google-cloud-python/tree/main/packages/google-auth-oauthlib
- https://github.com/googleapis/google-cloud-python/blob/main/packages/google-auth-oauthlib/tests/unit/test_interactive.py
- https://cloud.google.com/docs/authentication/get-id-token#python
- Using OAuthlib generate a signed JWT Token
- Using google.oauth2, create a credentials object using the token generated in step 1
This flow needs your own signing key and an oauthlib RequestValidator subclass, so it is shown as a sketch rather than a runnable cell:
from oauthlib.oauth2 import Server
from oauthlib.oauth2.rfc6749 import tokens
import google.oauth2.credentials
private_pem_key = ... # your private signing key, in PEM format
validator = ... # an instance of your oauthlib.oauth2.RequestValidator subclass
server = Server(
validator,
token_generator=tokens.signed_token_generator(private_pem_key, issuer="foobar")
)
# Once completed, the token endpoint will generate access_token in JWT form:
# access_token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJy(..)&expires_in=120&token_type=Bearer(..)
credentials = google.oauth2.credentials.Credentials(
'your_access_token')
Create a Customized Auth Flow [Advanced, Not Recommended]¶
Resources:
- https://google-auth-oauthlib.readthedocs.io/en/latest/reference/google_auth_oauthlib.html
- https://google-auth-oauthlib.readthedocs.io/en/latest/_modules/google_auth_oauthlib/interactive.html?highlight=interactive#
- Using OAuthlib generate a signed JWT Token
- Using google.oauth2, create a credentials object using the token generated in step 1
from google_auth_oauthlib import interactive
from google_auth_oauthlib.flow import Flow
# Create the flow using the client secrets file from the Google API console.
flow = Flow.from_client_secrets_file(
'path/to/client_secrets.json',
scopes=['profile', 'email'],
redirect_uri='urn:ietf:wg:oauth:2.0:oob')
# Tell the user to go to the authorization URL.
auth_url, _ = flow.authorization_url(prompt='consent')
print('Please go to this URL: {}'.format(auth_url))
# The user will get an authorization code. This code is used to get the
# access token.
code = input('Enter the authorization code: ')
flow.fetch_token(code=code)
# You can use flow.credentials, or you can just get a requests session
# using flow.authorized_session.
session = flow.authorized_session()
print(session.get('https://www.googleapis.com/userinfo/v2/me').json())
Google Earth Engine Service Account Credentials¶
import json
import os
import ee
# Read the service-account email from the key file saved in the first example
with open(json_fp) as f:
service_account = json.load(f)["client_email"]
credentials = ee.ServiceAccountCredentials(service_account, key_file=json_fp)
ee.Initialize(credentials, project=os.environ["GOOGLE_CLOUD_PROJECT"])
PyData Google Auth¶
Note: pydata_google_auth.get_user_credentials() starts a browser-based sign-in flow, so it needs a browser that can reach Google's sign-in page.
import os
import pydata_google_auth
from google.cloud import bigquery
credentials = pydata_google_auth.get_user_credentials(
['https://www.googleapis.com/auth/cloud-platform'],
)
# Use the credentials in other libraries, such as the Google BigQuery
# client library.
client = bigquery.Client(project=os.environ["GOOGLE_CLOUD_PROJECT"], credentials=credentials)