Dask SQL with FugueSQL¶
Description:
- Run FugueSQL (SQL extended with workflow steps such as
TAKE,PRINTandTRANSFORM) on Dask through dask-sql'sfsql_daskhelper.
Resources:
- [Docs] https://dask-sql.readthedocs.io/en/latest/fugue.html
- [GitHub] https://github.com/fugue-project/fugue
Requirements:
fsql_daskneeds Fugue's SQL extra:pip install "fugue[sql]".- The last dask-sql release (2024.5.0) does not work with dask 2025.1 or newer, so install it next to an older dask:
pip install dask-sql "fugue[sql]" "dask[dataframe]<2025.1" "distributed<2025.1".
Example 1¶
Run a FugueSQL query on a Dask dataframe and register its result back into the dask-sql context
In [ ]:
Copied!
import dask.datasets
from dask_sql import Context
from dask_sql.integrations.fugue import fsql_dask
# create a context to register tables
c = Context()
# create a table and register it in the context
# (FugueSQL needs a plain index, so the timestamp index becomes a column)
df = dask.datasets.timeseries().reset_index()
c.create_table("timeseries", df)
# FugueSQL extends SQL with workflow steps: this names the result `totals`,
# then takes the 3 rows with the largest totals and prints them
query = """
totals = SELECT name, SUM(x) AS total_x
FROM timeseries
GROUP BY name
TAKE 3 ROWS PRESORT total_x DESC
PRINT
"""
# register=True also adds every named step (here `totals`) to the context as a table
result = fsql_dask(query, c, register=True)
# the named steps come back as Dask dataframes...
print(list(result))
# ...and can be queried with plain dask-sql
c.sql("SELECT * FROM totals ORDER BY total_x DESC LIMIT 3").compute()
import dask.datasets
from dask_sql import Context
from dask_sql.integrations.fugue import fsql_dask
# create a context to register tables
c = Context()
# create a table and register it in the context
# (FugueSQL needs a plain index, so the timestamp index becomes a column)
df = dask.datasets.timeseries().reset_index()
c.create_table("timeseries", df)
# FugueSQL extends SQL with workflow steps: this names the result `totals`,
# then takes the 3 rows with the largest totals and prints them
query = """
totals = SELECT name, SUM(x) AS total_x
FROM timeseries
GROUP BY name
TAKE 3 ROWS PRESORT total_x DESC
PRINT
"""
# register=True also adds every named step (here `totals`) to the context as a table
result = fsql_dask(query, c, register=True)
# the named steps come back as Dask dataframes...
print(list(result))
# ...and can be queried with plain dask-sql
c.sql("SELECT * FROM totals ORDER BY total_x DESC LIMIT 3").compute()