Cloud Run GPUs for On-Demand AI Inference

The Problem#

AI inference often needs GPUs, but many applications do not have steady enough traffic to justify a permanently running GPU cluster.

Why It Matters#

Cloud Run GPU support brings the Cloud Run model to inference: managed containers, request-based scaling, and scale to zero. That is attractive for prototypes, internal tools, batch endpoints, and variable traffic.

Core Concepts#

Cloud Run attaches a GPU to a service instance. The container owns the model server. The platform manages drivers and scaling, but your app still owns image size, model loading, concurrency, and timeouts.

Implementation#

Design the service contract first:

POST /v1/infer
  input: prompt or document
  timeout: explicit
  idempotency-key: required for async work
  response: result or job id

For larger models, prefer an async job pattern so requests do not sit open during cold start or long generation.

Real Project Scenario#

A product team wants internal document summarization using a custom model. Traffic is uneven: quiet most of the day, then busy after document uploads. Cloud Run GPUs can fit this pattern because the service can scale down when idle and scale out during bursts, while the team keeps model serving inside a container.

Production Setup#

Use a job id for long inference. The request creates a job, workers process it on GPU instances, and clients poll or receive a webhook. This avoids long HTTP connections and makes retries safer when instances cold start.

Common Mistakes#

• Treating scale to zero as free latency.
• Deploying a model image so large that startup dominates every burst.
• Allowing too much concurrency per GPU and causing memory pressure.
• Forgetting regional GPU availability and quota.

Production Considerations#

Use min instances for latency-sensitive paths and scale-to-zero for batch or admin tools. Keep a canary service for runtime and driver upgrades.

Security#

Keep private models in controlled artifact storage. Limit who can deploy new images because model artifacts can change behavior as much as code.

Performance#

Measure cold start, model load, time to first token, and steady-state tokens per second separately. They improve through different changes.

Summary#

Cloud Run GPUs are a good fit for containerized inference with bursty demand. They work best when cold starts, quota, model loading, and concurrency are explicit.