Dask SQL¶
Description:
- A distributed SQL query engine in Python. It allows you to query and transform your data using a mixture of common SQL operations and Python code and also scale up the calculation easily if you need it.
Resources:
Requirements:
- The last dask-sql release (2024.5.0) does not work with dask 2025.1 or newer, so install it next to an older dask:
pip install dask-sql "dask[dataframe]<2025.1" "distributed<2025.1".
Example 1¶
Execute a SQL query on a small dataframe
In [ ]:
Copied!
import dask.datasets
from dask_sql import Context
# create a context to register tables
c = Context()
# create a table and register it in the context
df = dask.datasets.timeseries()
c.create_table("timeseries", df)
# execute a SQL query; the result is a "lazy" Dask dataframe
result = c.sql("""
SELECT
name, SUM(x)
FROM
timeseries
GROUP BY
name
""")
# actually compute the query...
result.compute()
import dask.datasets
from dask_sql import Context
# create a context to register tables
c = Context()
# create a table and register it in the context
df = dask.datasets.timeseries()
c.create_table("timeseries", df)
# execute a SQL query; the result is a "lazy" Dask dataframe
result = c.sql("""
SELECT
name, SUM(x)
FROM
timeseries
GROUP BY
name
""")
# actually compute the query...
result.compute()
Example 2¶
Execute a SQL query using a custom python function on a dataframe (about 2.6 million rows) in seconds
In [ ]:
Copied!
import numpy as np
import dask.datasets
from dask_sql import Context
# create a context to register tables
c = Context()
# register the table that the query reads
c.create_table("timeseries", dask.datasets.timeseries())
def f(x):
return x ** 2
c.register_function(f, "f", [("x", np.float64)], np.float64, replace=True)
c.sql("SELECT f(x) FROM timeseries").compute()
import numpy as np
import dask.datasets
from dask_sql import Context
# create a context to register tables
c = Context()
# register the table that the query reads
c.create_table("timeseries", dask.datasets.timeseries())
def f(x):
return x ** 2
c.register_function(f, "f", [("x", np.float64)], np.float64, replace=True)
c.sql("SELECT f(x) FROM timeseries").compute()