Ray Data: read a catalog table¶
A catalog node creates a DatabricksUnityCatalog client from an environment variable; the reader resolves the table through it.
When to use it. Reading governed tables by name instead of by storage path. Shown as code only: it needs a real catalog workspace and token, and a Ray version that has the catalog API.
The diagram below is the graph as Eugo's visual Ray builder exports it (a Mermaid diagram), and the cell under Generated code is the Ray code the builder produced from it. The code is plain Ray: it runs without the builder.
Experimental. The builder lists a node in this graph in its Experimental section, so the node may change in later releases of the builder.
Code only. Running this notebook needs a Databricks Unity Catalog workspace, an access token in the UNITY_CATALOG_TOKEN environment variable and Ray 2.58.0 or later. It was not run for this gallery, so it has no outputs.
Ray 2.55.1 has no ray.data.DatabricksUnityCatalog class and its read_delta takes no catalog= argument, so this code does not run there. On Ray 2.55.1 the same table can be read with ray.data.read_unity_catalog("main.sales.orders", url=..., token=os.environ["UNITY_CATALOG_TOKEN"]).
Back to the gallery · Written for Ray 2.58.0 or later
flowchart LR n_c1["Catalog"] n_n1["read_delta"] n_n2["count"] n_c1 -- "catalog" --> n_n1 n_n1 --> n_n2
The graph, as the Mermaid diagram that Eugo's visual Ray builder exports for it. A screenshot of the builder canvas is not available for this pattern.
Setup¶
Start a local Ray instance. The options keep the notebook quiet: no dashboard, and worker logs stay out of the notebook. Ray is capped at four CPUs so the example behaves the same on any machine. The generated code below only calls ray.init() when Ray is not running yet, so it reuses this instance.
import ray
ray.init(include_dashboard=False, log_to_driver=False, logging_level="ERROR", num_cpus=4)
User functions¶
This pattern needs no user functions. Set UNITY_CATALOG_TOKEN in the environment (never in the notebook) and point the catalog node at your workspace URL.
Generated code¶
This is the cell the visual builder generated from the graph above.
# Generated with Eugo's visual Ray builder
import os
import ray
from ray.data import DatabricksUnityCatalog
if not ray.is_initialized():
ray.init()
_catalog_c1 = DatabricksUnityCatalog(url="https://unity-catalog.example.com", token=os.environ["UNITY_CATALOG_TOKEN"])
ds = ray.data.read_delta("main.sales.orders", catalog=_catalog_c1)
n_rows = ds.count()
Check the result¶
The cell below checks that the generated code did what the graph describes.
assert n_rows > 0, n_rows
print("rows =", n_rows)
How the graph maps to the code¶
Each step names a node in the diagram, the lines of the generated cell it produced, and what those lines do. The node id (for example n1) is also the suffix of the variable that holds the node's result.
Setup (lines 1, 3, 5–6 and 8–9)
# Generated with Eugo's visual Ray builder
...
import os
...
import ray
from ray.data import DatabricksUnityCatalog
...
if not ray.is_initialized():
ray.init()
The first line marks the cell as generated. The builder then adds an import for every Ray API the graph uses and a guarded ray.init(), so re-running the cell reuses the running cluster.
Catalog (alpha) (node c1, line 11)
_catalog_c1 = DatabricksUnityCatalog(url="https://unity-catalog.example.com", token=os.environ["UNITY_CATALOG_TOKEN"])
The catalog client. The token is read from os.environ at run time, so it never appears in the notebook.
Source (node n1, line 12)
ds = ray.data.read_delta("main.sales.orders", catalog=_catalog_c1)
read_delta resolves the three-part table name through the catalog= argument.
Sink (node n2, line 13)
n_rows = ds.count()
count() reads the table and counts rows.
Clean up¶
Shut down Ray, so the next notebook starts fresh.
ray.shutdown()