Graceful Shutdown in Containers

The Problem#

Containers are stopped all the time: deploys, autoscaling, node drains, and failed health checks. If your app exits immediately on SIGTERM, users see failed requests and workers may lose jobs midway.

In production, this usually appears during a normal deploy. The deployment tool removes one instance from service, starts another one, and expects the old instance to finish its current work before it exits. If the application does not cooperate, the platform can only kill it.

Why It Matters#

A graceful shutdown path protects customer requests, queue jobs, logs, metrics, and database consistency. It also makes rolling deploys safer because old and new instances can overlap without dropping traffic.

Platforms such as Kubernetes, Amazon ECS, Docker, and Google Cloud Run all use the same basic idea: send a termination signal, wait for a grace period, then force the process to exit. Your application must use that grace period well.

Project Example#

In a Node.js API, the container receives SIGTERM during a rolling deploy. The app should stop accepting new connections, let in-flight requests finish, close database pools, flush logs, and exit before Kubernetes sends SIGKILL.

For example, a checkout API may be processing a payment request when deployment starts. If the process exits immediately, the payment provider may receive the charge request but the user never receives the response. If the client retries, you now need idempotency to avoid a duplicate charge. Graceful shutdown reduces the chance of that failure chain.

Implementation Example#

import http from "node:http";

const server = http.createServer(app);
let shuttingDown = false;

app.get("/health/readiness", (_req, res) => {
  res.status(shuttingDown ? 503 : 200).send(shuttingDown ? "draining" : "ready");
});

process.on("SIGTERM", async () => {
  shuttingDown = true;

  server.close(async () => {
    await db.end();
    await logger.flush();
    process.exit(0);
  });

  setTimeout(() => {
    logger.error("forced shutdown after deadline");
    process.exit(1);
  }, 25_000).unref();
});

The readiness endpoint fails first, so the load balancer stops sending new traffic. server.close() lets existing requests finish. The forced timeout ensures the container exits before the platform's grace period expires.

Implementation Checklist#

• Listen for SIGTERM.
• Mark the instance unready before closing.
• Stop accepting new requests.
• Wait for in-flight work with a deadline.
• Close database, queue, and telemetry clients.
• Exit with success when shutdown completes.
• Make background workers stop pulling new jobs before finishing current jobs.
• Set the app deadline lower than the platform grace period.
• Test shutdown under active traffic, not only during idle deploys.

Production Notes#

For Kubernetes, set terminationGracePeriodSeconds long enough for normal requests to finish, but not so long that bad pods block deploys forever. For HTTP APIs, 20 to 60 seconds is common. For queue workers, the right value depends on job length.

Do not put slow dependency checks in liveness probes. A dependency outage should usually make the pod unready, not cause Kubernetes to restart every pod at once.

How To Validate#

Run a local or staging test where requests are in flight and then send SIGTERM:

kill -TERM <pid>

Confirm that new requests stop, current requests finish, logs flush, and the process exits before the configured deadline.

Common Mistakes#

• Using liveness probes to decide deploy readiness.
• Taking longer than terminationGracePeriodSeconds.
• Forgetting background workers and cron handlers.
• Closing database pools before active requests finish.
• Calling process.exit() immediately inside the signal handler.
• Letting the load balancer continue routing traffic while the app drains.

Summary#

Graceful shutdown turns deploys from random request failures into controlled handoffs. Every production container should have a tested shutdown path.