Ray Serve: dynamic request batching¶
The builder writes a deployment whose handler is wrapped in @serve.batch, so concurrent requests are grouped into one call.
When to use it. Models that are much faster on a batch than on single inputs, such as most neural networks on GPUs.
The picture below is the graph drawn in Eugo's visual Ray builder, and the cell under Generated code is the Ray code the builder produced from it, with one line edited by hand so that each result also reports the size of its batch (the cell's first line says so). The code is plain Ray: it runs without the builder.
Experimental. The builder lists a node in this graph in its Experimental section, so the node may change in later releases of the builder. The generated code ran on Ray 2.55.1.
Back to the gallery · Tested with Ray 2.55.1

The graph in Eugo's visual Ray builder.
Setup¶
Start a local Ray instance. The options keep the notebook quiet: no dashboard, and worker logs stay out of the notebook. Ray is capped at four CPUs so the example behaves the same on any machine. The generated code below only calls ray.init() when Ray is not running yet, so it reuses this instance.
import ray
ray.init(include_dashboard=False, log_to_driver=False, logging_level="ERROR", num_cpus=4)
RayContext(dashboard_url='', python_version='3.12.3', ray_version='2.55.1', ray_commit='237c2455ebb1ea15a32dd9e1fdeb2d617badc37f')
User functions¶
The nodes in the graph call these ordinary Python functions and classes. In the builder you write them in a normal notebook cell, and the generated code refers to them by name.
# The Batcher class is written by the builder from the node's settings.
from ray import serve
Generated code¶
This is the cell the visual builder generated from the graph above, with one line edited by hand: the handler returns (x * 2, len(items)) instead of x * 2, so each caller also learns how many requests shared its handler call, and the check below can see the batches.
# Generated with Eugo's visual Ray builder, then edited by hand: each result also carries its batch size
from ray import serve
import ray
if not ray.is_initialized():
ray.init()
@serve.deployment
class Batcher:
@serve.batch(max_batch_size=4, batch_wait_timeout_s=0.05)
async def __call__(self, items):
return [(x * 2, len(items)) for x in items]
_deploy_n1 = Batcher.bind()
serve.run(_deploy_n1)
DeploymentHandle(deployment='Batcher')
Check the result¶
The cell below checks that the generated code did what the graph describes: all eight answers are right, at least one handler call served more than one request, and no call served more than max_batch_size=4. How the eight requests split into batches varies from run to run.
A Jupyter kernel already runs an asyncio event loop, so this cell awaits the Serve responses; in a plain Python script you would call .result() on them instead.
handle = serve.get_app_handle("default")
responses = [handle.remote(i) for i in range(8)]
answers = [await r for r in responses]
assert [value for value, _ in answers] == [i * 2 for i in range(8)], answers
sizes = [size for _, size in answers] # the batch size each request was served in
assert max(sizes) > 1, sizes # some requests shared one handler call
assert max(sizes) <= 4, sizes # no batch was larger than max_batch_size
print("8 requests answered; batch size seen by each request:", sizes)
8 requests answered; batch size seen by each request: [4, 4, 4, 4, 4, 4, 4, 4]
How the graph maps to the code¶
Each step names a node in the picture, the lines of the generated cell it produced, and what those lines do. The node id (for example n1) is also the suffix of the variable that holds the node's result.
Setup (lines 1, 3–4 and 6–7)
# Generated with Eugo's visual Ray builder, then edited by hand: each result also carries its batch size
...
from ray import serve
import ray
...
if not ray.is_initialized():
ray.init()
The first line marks the cell as generated and names the edit made by hand. The builder then adds an import for every Ray API the graph uses and a guarded ray.init(), so re-running the cell reuses the running cluster.
Serve batch (experimental) (node n1, lines 9–13 and 15–16)
@serve.deployment
class Batcher:
@serve.batch(max_batch_size=4, batch_wait_timeout_s=0.05)
async def __call__(self, items):
return [(x * 2, len(items)) for x in items]
...
_deploy_n1 = Batcher.bind()
serve.run(_deploy_n1)
@serve.batch(max_batch_size=4, batch_wait_timeout_s=0.05) collects up to four requests (or waits 50 ms) and calls the handler once with a list; each caller gets its own element back. The line edited by hand pairs each result with len(items), the number of requests in that call.
Clean up¶
Shut down Serve and then Ray, so the next notebook starts fresh.
from ray import serve
serve.shutdown()
ray.shutdown()