Graceful Shutdown Why Your Container Kills In-Flight Requests for 10 Seconds on Every Deploy

You push a deploy. For the next 10 seconds, some percentage of in-flight requests get a connection reset or a 502. Then it clears. This happens on every rollout, has been happening for months, and the fix is usually three things you missed, not one.

Here’s what’s actually going on, why the naive signal handler doesn’t work, and what correct shutdown looks like in Node.js, Python, and Go.

The shutdown sequence, step by step

When Kubernetes terminates a pod — rolling deploy, scale-down, node eviction — this is the sequence:

1. Kubernetes removes the pod from the Service’s endpoint list, so new traffic stops routing to it
2. Beliebig preStop lifecycle hook runs
3. SIGTERM is sent to PID 1 inside the container
4. Kubernetes waits up to terminationGracePeriodSeconds (default: 30s)
5. If the process hasn’t exited, SIGKILL is sent — immediate, untrappable death

Steps 1 and 3 happen concurrently, not sequentially. The endpoint list update propagates through kube-proxy, iptables rules, and your load balancer — and that propagation takes a few seconds. You will get a brief window where traffic is still hitting a pod that’s already received SIGTERM. The standard mitigation is a preStop sleep:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 5"]

Five seconds gives the load balancer time to drain before your app starts shutting down. Yes, it adds 5 seconds to every pod termination. It’s worth it. One caveat: preStop time counts against terminationGracePeriodSeconds, so if your grace period is 30s, your app gets 25s for actual cleanup.

Docker has an equivalent. docker stop sends SIGTERM, waits 10 seconds by default, then sends SIGKILL. That 10-second default is where the title of this article comes from. Every deployment drops requests for exactly 10 seconds because nobody changed the default.

The PID 1 problem

Before your signal handler even runs, you need to receive the signal. This is where most container setups fail silently.

If your Dockerfile uses the shell form of CMD:

# Shell form — /bin/sh becomes PID 1
CMD node server.js

Then /bin/sh is PID 1. When SIGTERM arrives, the shell exits. It does not forward the signal to child processes. Your Node.js process either gets SIGHUP (if the shell bothers) or nothing coherent. Your shutdown handler never runs.

The exec form makes your process PID 1:

# Exec form — your process IS PID 1
CMD ["node", "server.js"]

But PID 1 has its own quirk in Linux. The kernel installs no default signal handlers for PID 1. For most processes, SIGTERM’s default action is to terminate the process. For PID 1, there’s no default — the process has to explicitly handle it, or the signal is discarded.

Node.js, Python, and Go all install their own SIGTERM handlers at the runtime level (so the process does exit), but they don’t give your application code a chance to run cleanup. The correct solution is a proper init process. Docker ships one:

# docker-compose.yml
services:
  api:
    image: myapp:latest
    init: true  # uses Docker's built-in tini

Or bake tini into your image:

FROM node:22-alpine
RUN apk add --no-cache tini
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["node", "server.js"]

Tini sits as PID 1, handles zombie reaping, and forwards signals to your app’s process group. Your process receives SIGTERM cleanly, your handler runs, you do cleanup, you exit. The way it’s supposed to work.

Signal handling in Node.js, Python, and Go

With signal delivery working, here’s what the handlers should look like.

Node.js

const http = require('http');

const server = http.createServer(app);
server.listen(3000);

const shutdown = () => {
  // Stop accepting new connections
  server.close(() => {
    console.log('All connections drained, exiting');
    process.exit(0);
  });

  // Close idle keep-alive connections immediately (Node 18.2+)
  server.closeAllConnections();

  // Hard exit if cleanup takes too long
  setTimeout(() => {
    console.error('Forced exit after timeout');
    process.exit(1);
  }, 25_000);
};

process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown); // for Ctrl+C in local dev

The 25-second timeout is intentional — if you’re running under Kubernetes with a 30-second grace period, you want your app to exit cleanly before SIGKILL arrives. Exit 0 is clean, exit 1 signals an error, SIGKILL is brutal.

Python (gunicorn / uvicorn)

Gunicorn handles SIGTERM at the master level: it sends TERM to workers and waits for them to finish in-flight requests. You get graceful shutdown mostly for free, but you need to configure the timeout:

# gunicorn.conf.py
workers = 4
timeout = 30
graceful_timeout = 25  # how long workers have to finish requests on SIGTERM
worker_class = "gthread"

If you’re running uvicorn directly (without gunicorn), use the --graceful-timeout flag (available since 0.20). For bare asyncio apps or custom ASGI runners:

import asyncio
import signal

async def main():
    server = await start_your_server()
    loop = asyncio.get_running_loop()

    stop = loop.create_future()
    loop.add_signal_handler(signal.SIGTERM, stop.set_result, None)
    loop.add_signal_handler(signal.SIGINT, stop.set_result, None)

    await stop  # wait for signal
    await server.shutdown()  # drain connections

asyncio.run(main())

Note that loop.add_signal_handler only works in the main thread. If your entry point spawns threads before the event loop starts, use signal.signal(signal.SIGTERM, handler) before the thread spawn instead.

Gehen

Go’s net/http has Shutdown(ctx) built in since Go 1.8. It stops accepting connections, closes idle keep-alives, and waits for active requests to complete — exactly what you want:

package main

import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    srv := &http.Server{Addr: ":8080", Handler: buildRouter()}

    go func() {
        if err := srv.ListenAndServe(); err != http.ErrServerClosed {
            log.Fatalf("ListenAndServe: %v", err)
        }
    }()

    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGTERM, syscall.SIGINT)
    <-quit

    ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
    defer cancel()

    if err := srv.Shutdown(ctx); err != nil {
        log.Printf("Shutdown exceeded timeout: %v", err)
        os.Exit(1)
    }
    log.Println("Clean shutdown complete")
}

Go's Shutdown correctly handles keep-alive connections: it closes idle ones immediately and waits only on connections with active requests. You don't need to track connections manually.

Warum server.close() alone isn't enough in Node.js

This is the part most tutorials skip. server.close() tells Node to stop accepting new TCP connections. It does not close existing connections. HTTP/1.1 keep-alive means a client can have a TCP connection sitting open with no active request — just waiting for the next one. server.close() leaves those connections open and won't call its completion callback until they all close on their own.

If your server's keep-alive timeout is 60 seconds, server.close()'s callback might not fire for 60 seconds after you call it. If your platform's grace period is 30 seconds, SIGKILL fires first. Your "graceful" shutdown is not graceful.

server.closeAllConnections() (Node 18.2+) solves this by closing idle keep-alive connections immediately while letting connections with active requests finish normally. On Node 16 or below, you need to track and destroy them manually:

const connections = new Set();

server.on('connection', (conn) => {
  connections.add(conn);
  conn.on('close', () => connections.delete(conn));
});

const shutdown = () => {
  server.close();
  for (const conn of connections) {
    conn.destroy(); // close idle keep-alives
  }
};

The more surgical version only destroys connections that are between requests (not actively serving one). But in practice, if you're setting a hard timeout anyway and clients have retry logic, destroying all idle connections is fine and simpler.

Setting the timing knobs

Einstellung	Plattform	Standard	Where to set it
terminationGracePeriodSeconds	Kubernetes	30s	Pod spec
stop_grace_period	Docker Compose	10s	Service definition
--time	docker stop	10s	CLI flag

Kubernetes:

spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: api
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 5"]

Docker Compose:

services:
  api:
    image: myapp:latest
    stop_grace_period: 30s
    init: true

The relationship to keep in mind: platform grace period > preStop sleep + app shutdown timeout. If K8s gives you 30s and preStop sleeps for 5s, your app has 25s. Set your hard timeout at 23–24s to leave a small buffer before SIGKILL.

Testing shutdown behavior

The only way to know if this works is to test it. The procedure: send requests that take a few seconds to complete, trigger a shutdown mid-request, and see if the response arrives or drops.

A minimal test with curl — use IO Tools' cURL Command Builder if you want to generate the right flags without reading the man page:

# Terminal 1: send a slow request (to an endpoint that sleeps or streams)
curl -v --max-time 30 http://localhost:3000/slow

# Terminal 2: stop the container while the request is in-flight
docker stop --time=30 my-container

If the slow request returns 200, graceful shutdown is working. If you see curl: (56) Recv failure: Connection reset by peer mid-stream, something in the chain is wrong. Work backwards: is the process receiving SIGTERM? Is the handler registering? Is server.close() being called? Are connections being destroyed?

For debugging the timing — comparing log timestamps against shutdown events, or checking how much of your grace period the preStop hook is consuming — the Unix-Zeitstempel-Konverter makes it straightforward to convert epoch timestamps from logs into readable times and do the arithmetic on your shutdown windows.

The full checklist

• Use exec form in Dockerfile CMD — ["node", "server.js"]sein, nicht node server.js
• Use tini or Docker's init: true so your process receives signals properly
• Handle SIGTERM (and SIGINT for local dev) explicitly in application code
• In Node.js: call server.closeAllConnections() alongside server.close()
• Set a hard timeout in your shutdown handler a few seconds under the platform grace period
• Fügen Sie eine preStop sleep (5s) to drain load balancer traffic before SIGTERM fires
• Set terminationGracePeriodSeconds oder stop_grace_period to cover preStop + your longest expected request + buffer
• Change Docker Compose's default stop_grace_period from 10s to something sensible
• Test it — actually send in-flight requests during a docker stop and verify they complete

If you're hitting exactly 10 seconds of drops, it's almost certainly the Docker Compose default you never changed. That's the low-hanging fix. The rest — tini, signal handlers, closeAllConnections, preStop sleep — layers on top of that to handle the edge cases that still bite you at scale.

Das könnte Ihnen auch gefallen