Designing for Retries

The Problem#

Retries fix transient failures, but they can also multiply traffic during an outage. A failing dependency can become worse when every caller retries at the same time.

Why It Matters#

Retries are load multipliers. If one request makes three downstream calls and each call retries three times, a traffic spike can become an incident. Retry policy has to be designed with deadlines, idempotency, and downstream capacity.

Cloud SDKs, payment clients, storage clients, and queue consumers all retry by default in many configurations. Engineers need to know what their libraries are already doing before adding another retry loop.

Project Example#

A checkout API calls inventory, payments, and email. If each call retries three times with no deadline, one user request can become dozens of downstream calls. The API should use a total request budget, retry only safe operations, and add jitter.

Implementation Example#

const response = await retry(
  () => inventory.reserve(orderId, { idempotencyKey }),
  {
    retries: 2,
    minDelayMs: 100,
    maxDelayMs: 800,
    jitter: true,
    shouldRetry: (err) => err.status >= 500 || err.code === "ETIMEDOUT",
  },
);

This retries transient failures only, uses jitter, and relies on an idempotency key so the reservation is safe to repeat.

Implementation Checklist#

• Set a total deadline for the user request.
• Retry only idempotent operations or writes with idempotency keys.
• Use exponential backoff with jitter.
• Cap attempts and retry duration.
• Stop retrying when the caller deadline is nearly exhausted.
• Do not retry validation errors or permission failures.
• Budget retries across the full call chain.
• Emit metrics for attempts, final failures, and retry latency.

Production Notes#

Use Retry-After when your service rejects requests due to rate limits or temporary overload. Clients that honor it are much easier to operate than clients that retry immediately.

Common Mistakes#

• Retrying POST writes without idempotency.
• Using the same retry schedule across thousands of clients.
• Retrying after a dependency returns a clear validation error.
• Ignoring retry traffic in capacity planning.
• Nesting retries across client, service, queue, and SDK layers.

How To Validate#

Run a failure test where the dependency returns 500, times out, and returns 429. Confirm retry count, total elapsed time, final response, and downstream request volume. The policy is correct only if it behaves well during failure, not just during the happy path.

Summary#

Good retries are bounded, jittered, and deadline-aware. They should improve availability without turning partial failures into full outages.