Ray Serve: autoscaling a deployment¶
autoscaling_config lets Serve add replicas (here between 1 and 3) as the number of ongoing requests per replica grows; max_queued_requests sheds load beyond a limit.
When to use it. Traffic that varies over the day, where a fixed replica count is either wasteful or too small.
The picture below is the graph drawn in Eugo's visual Ray builder, and the cell under Generated code is the Ray code the builder produced from it. The code is plain Ray: it runs without the builder.
Back to the gallery · Tested with Ray 2.55.1

The graph in Eugo's visual Ray builder.
Setup¶
Start a local Ray instance. The options keep the notebook quiet: no dashboard, and worker logs stay out of the notebook. Ray is capped at four CPUs so the example behaves the same on any machine. The generated code below only calls ray.init() when Ray is not running yet, so it reuses this instance.
import ray
ray.init(include_dashboard=False, log_to_driver=False, logging_level="ERROR", num_cpus=4)
RayContext(dashboard_url='', python_version='3.12.3', ray_version='2.55.1', ray_commit='237c2455ebb1ea15a32dd9e1fdeb2d617badc37f')
User functions¶
The nodes in the graph call these ordinary Python functions and classes. In the builder you write them in a normal notebook cell, and the generated code refers to them by name.
from ray import serve
@serve.deployment
class Echo:
async def __call__(self, x):
return x
Generated code¶
This is the cell the visual builder generated from the graph above.
# Generated with Eugo's visual Ray builder
from ray import serve
import ray
if not ray.is_initialized():
ray.init()
_deploy_n1 = Echo.options(autoscaling_config={"min_replicas": 1, "max_replicas": 3, "target_ongoing_requests": 2}, max_queued_requests=20).bind()
serve.run(_deploy_n1)
DeploymentHandle(deployment='Echo')
Check the result¶
The cell below checks what a short run can show: the deployment answers requests, Serve applied the autoscaling settings from the graph (1 to 3 replicas, a target of 2 ongoing requests per replica, at most 20 queued requests), and with no load it runs min_replicas, one replica. It does not show Serve adding replicas: that takes sustained load for longer than the default upscale_delay_s of 30 seconds, which is too slow and too timing-dependent for this notebook. serve.status() reports replica counts but not a deployment's settings, so the check reads Serve's full status details, the same ones serve.status() summarizes, from the Serve controller through ray.serve.context._get_global_client(). That is a private Ray call and may change in later Ray releases.
A Jupyter kernel already runs an asyncio event loop, so this cell awaits the Serve responses; in a plain Python script you would call .result() on them instead.
from ray.serve.context import _get_global_client # private: see the note above
from ray.serve.schema import ServeInstanceDetails
handle = serve.get_app_handle("default")
responses = [handle.remote(i) for i in range(10)]
assert [await r for r in responses] == list(range(10))
# The settings Serve applied to the deployment, as the Serve controller reports them.
details = ServeInstanceDetails(**_get_global_client().get_serve_details())
echo = details.applications["default"].deployments["Echo"]
auto = echo.deployment_config.autoscaling_config
assert (auto["min_replicas"], auto["max_replicas"], auto["target_ongoing_requests"]) == (1, 3, 2), auto
assert echo.deployment_config.max_queued_requests == 20, echo.deployment_config.max_queued_requests
assert echo.target_num_replicas == 1, echo.target_num_replicas # no load: the autoscaler keeps min_replicas
status = serve.status().applications["default"].deployments["Echo"]
assert status.status == "HEALTHY", status.status
assert status.replica_states == {"RUNNING": 1}, status.replica_states
print("10 requests answered; deployment", status.status.value)
print(f"autoscaling: {auto['min_replicas']} to {auto['max_replicas']} replicas, "
f"target {auto['target_ongoing_requests']:g} ongoing requests per replica, "
f"at most {echo.deployment_config.max_queued_requests} queued requests")
print("replicas running with no load:", status.replica_states["RUNNING"])
10 requests answered; deployment HEALTHY autoscaling: 1 to 3 replicas, target 2 ongoing requests per replica, at most 20 queued requests replicas running with no load: 1
How the graph maps to the code¶
Each step names a node in the picture, the lines of the generated cell it produced, and what those lines do. The node id (for example n1) is also the suffix of the variable that holds the node's result.
Setup (lines 1, 3–4 and 6–7)
# Generated with Eugo's visual Ray builder
...
from ray import serve
import ray
...
if not ray.is_initialized():
ray.init()
The first line marks the cell as generated. The builder then adds an import for every Ray API the graph uses and a guarded ray.init(), so re-running the cell reuses the running cluster.
Deployment (node n1, lines 9–10)
_deploy_n1 = Echo.options(autoscaling_config={"min_replicas": 1, "max_replicas": 3, "target_ongoing_requests": 2}, max_queued_requests=20).bind()
serve.run(_deploy_n1)
autoscaling_config={"min_replicas": 1, "max_replicas": 3, "target_ongoing_requests": 2} replaces a fixed num_replicas; max_queued_requests=20 rejects requests beyond that backlog.
Clean up¶
Shut down Serve and then Ray, so the next notebook starts fresh.
from ray import serve
serve.shutdown()
ray.shutdown()