Ray Data¶
Distributed datasets: read, transform, aggregate, write.
This section holds 12 notebooks from ray_patterns/data/ in the ganymede repository. Each page shows the notebook as committed; use the buttons at the top right of a page to download the .ipynb or open it on GitHub.
Notebooks¶
- Ray Data: range, map_batches, count: A three-step Ray Data pipeline: create a dataset, transform it in batches, and count the rows.
- Ray Data: write Parquet: Writes a dataset to a Parquet directory with SaveMode.OVERWRITE, so re-running the cell replaces the output.
- Ray Data: groupby and aggregate: Groups rows by a key and computes Count and Sum per group with Ray Data built-in aggregations.
- Ray Data: computed columns and expression filters: with_column adds a computed column from col() and lit() expressions, and filter(expr=...) keeps the matching rows, with no Python UDF.
- Ray Data: join two datasets: An inner join of two datasets on an id column. The second dataset is wired into the join node's right input.
- Ray Data: train/test split: train_test_split has two outputs; each one feeds its own count. The builder names the two datasets for you.
- Ray Data: batch inference with an actor pool: map_batches with a callable class and ActorPoolStrategy: each actor runs the expensive setup once and then processes many batches.
- Ray Data: shuffle, repartition and sort: A seeded random_shuffle, a repartition into four blocks and a descending sort, then take(5).
- Ray Data: from pandas and back: Turns a pandas DataFrame into a Ray Dataset, adds a column row by row with map, and collects the result as a DataFrame again.
- Ray Data: fit a StandardScaler preprocessor: A Ray Data preprocessor learns column statistics in one pass (fit) and applies them (transform).
- Ray Data: read a catalog table: A catalog node creates a DatabricksUnityCatalog client from an environment variable; the reader resolves the table through it.
- Ray Data: batch LLM inference: The llm transform builds a ray.data.llm processor around a vLLM engine and runs prompts through it as a dataset stage.