Ray Serve¶
Model serving: deployments, composition, ingress, scaling.
This section holds 9 notebooks from ray_patterns/serve/ in the ganymede repository. Each page shows the notebook as committed; use the buttons at the top right of a page to download the .ipynb or open it on GitHub.
Notebooks¶
- Ray Serve: compose two deployments: A Greeter deployment is passed into a Driver deployment with .bind(); serve.run starts both as one application.
- Ray Serve: replicas and per-replica resources: A three-deployment pipeline where the classifier runs two replicas with half a CPU each and a cap on in-flight requests.
- Ray Serve: autoscaling a deployment: autoscaling_config lets Serve add replicas (here between 1 and 3) as the number of ongoing requests per replica grows; max_queued_requests sheds load beyond a…
- Ray Serve: FastAPI ingress with a route prefix: A FastAPI app, built on each replica by a builder function, becomes the HTTP ingress of a deployment served under the /api route prefix.
- Ray Serve: dynamic request batching: The builder writes a deployment whose handler is wrapped in @serve.batch, so concurrent requests are grouped into one call.
- Ray Serve: model multiplexing: One deployment serves many models: @serve.multiplexed loads a model by id on demand and keeps up to two per replica.
- Ray Serve: HTTP proxy settings and status: An HTTP proxy config node starts Serve on 127.0.0.1:8123 before the app is deployed, and a status node reads serve.status() afterwards.
- Ray Serve: gRPC ingress: A gRPC ingress node adds a gRPC proxy on port 9000 with a servicer function; the deployment answers protobuf requests.
- Ray Serve: an OpenAI-compatible LLM endpoint: An LLM node builds an OpenAI-compatible app with ray.serve.llm: model id, model files on Hugging Face, accelerator type and autoscaling bounds.