Language-Level Concurrency and Multithreading Questions

Direct answer

Start one goroutine per backend (fan-out: one request becomes many parallel calls), all sharing a child context made with context.WithCancel(parent). Each goroutine sends exactly one result into a channel buffered to len(backends); the caller then collects from that one channel (fan-in: many producers, one consumer). A channel is a typed pipe between goroutines, and a buffered channel holds up to its capacity of values, so a send only has to wait when the buffer is full. The caller reads results until the first success (then calls cancel() and returns it), until every backend has failed (then returns all errors joined), or until the parent context ends. The buffered channel is what prevents leaks: a late worker can always complete its send and exit, even though nobody is reading any more.

Complete program

A goroutine is Go's lightweight thread. A context.Context carries a cancellation signal: ctx.Done() is a channel that closes when the context is cancelled, and well-behaved code watches it. The -race flag turns on Go's data race detector (it reports unsynchronised access to shared memory at run time). select waits on several channel operations at once and runs whichever is ready first. This file runs as-is (go.mod with module hedge and go 1.23, then go run -race .).

package main

import (
	"context"
	"errors"
	"fmt"
	"runtime"
	"sort"
	"strings"
	"time"
)

type Backend func(ctx context.Context) (string, error)

type result struct {
	idx int
	val string
	err error
}

// FirstSuccess calls every backend concurrently and returns the first success.
// If all fail, it returns every error joined. Every worker exits promptly
// provided each Backend returns when its ctx is cancelled.
func FirstSuccess(parent context.Context, backends []Backend) (string, error) {
	if len(backends) == 0 {
		return "", errors.New("no backends")
	}
	ctx, cancel := context.WithCancel(parent)
	defer cancel()

	// Buffered to len(backends): a worker's send never blocks, even after
	// the receiver has stopped reading.
	results := make(chan result, len(backends))
	for i, b := range backends {
		go func(i int, b Backend) { // i and b passed as arguments: each goroutine gets its own copy
			v, err := b(ctx)
			results <- result{i, v, err}
		}(i, b)
	}

	errs := make([]error, len(backends))
	for received := 0; received < len(backends); received++ {
		select {
		case r := <-results:
			if r.err == nil {
				cancel() // tell the losers to stop now
				return r.val, nil
			}
			errs[r.idx] = fmt.Errorf("backend %d: %w", r.idx, r.err)
		case <-parent.Done():
			// Parent cancelled: workers see the same cancellation through ctx.
			return "", fmt.Errorf("parent context done: %w", context.Cause(parent))
		}
	}
	return "", errors.Join(errs...)
}

func fake(delay time.Duration, val string, fail bool) Backend {
	return func(ctx context.Context) (string, error) {
		select {
		case <-time.After(delay):
			if fail {
				return "", errors.New("boom " + val)
			}
			return val, nil
		case <-ctx.Done():
			return "", ctx.Err()
		}
	}
}

func main() {
	base := runtime.NumGoroutine()

	// 1. Fast failure, slow success, slower success: the slow success wins.
	v, err := FirstSuccess(context.Background(), []Backend{
		fake(10*time.Millisecond, "a", true),
		fake(40*time.Millisecond, "b", false),
		fake(5*time.Second, "c", false),
	})
	fmt.Println("case1:", v, err)

	// 2. All fail: every error is reported.
	v, err = FirstSuccess(context.Background(), []Backend{
		fake(5*time.Millisecond, "x", true),
		fake(10*time.Millisecond, "y", true),
	})
	lines := strings.Split(err.Error(), "\n")
	sort.Strings(lines)
	fmt.Printf("case2: %q %d errors: %s\n", v, len(lines), strings.Join(lines, " | "))

	// 3. Parent deadline fires before any backend answers.
	pctx, pcancel := context.WithTimeout(context.Background(), 30*time.Millisecond)
	defer pcancel()
	v, err = FirstSuccess(pctx, []Backend{fake(5*time.Second, "slow", false)})
	fmt.Println("case3:", v == "", err)

	time.Sleep(50 * time.Millisecond) // let exited goroutines be reaped
	fmt.Println("goroutines leaked:", runtime.NumGoroutine()-base)
}

Timeline of case 1 (the three fake backends): at t=0 all three start. At t=10 ms backend a fails; the error is stored in errs[0] and the loop keeps waiting. At t=40 ms backend b succeeds, so FirstSuccess calls cancel() and returns "b". Backend c was waiting on time.After(5s) or ctx.Done(); the cancel closes ctx.Done(), so c returns ctx.Err() at once, its send lands in the buffer, and its goroutine exits. Nothing waits five seconds.

Output of three consecutive runs with go run -race . in golang:1.23 (identical each time; the timings leave a wide margin between the cases):

case1: b <nil>
case2: "" 2 errors: backend 0: boom x | backend 1: boom y
case3: true parent context done: context deadline exceeded
goroutines leaked: 0

Why each piece is there

1. Buffered results channel (capacity len(backends)). With an unbuffered channel, the winner returns and nobody receives from the losers any more, so each loser blocks forever on its send. Changing only that one line to make(chan result) and rerunning the same program prints goroutines leaked: 2: one slow loser in case 1 and the one worker still running in case 3, each stuck sending to a channel nobody reads. Case 2 stays clean because the function reads every result.
2. defer cancel() plus an explicit cancel() on success. The explicit call stops the losers immediately; the deferred one guarantees the context's resources are released on every return path, including all-failed.
3. Collect errors by index, join at the end. fmt.Errorf with the %w verb wraps an error so errors.Is and errors.As can still see the original cause. errors.Join (Go 1.20 and later) returns an error whose Error() text is the messages separated by newlines, and errors.Is still works on each wrapped cause. Join drops nil entries, and on the all-failed path every slot holds an error.
4. A cancelled loser reports context.Canceled, and that is ignored. After the first success the function has already returned, so those errors land in the buffer and are garbage collected. They never get mixed into the all-failed error, because the all-failed path only runs when no success ever arrived.
5. parent.Done() in the select. If the caller's deadline fires, the function returns at once with the parent's cause. The workers see the same cancellation through the child context.

What the no-leak guarantee depends on

The guarantee is conditional: every Backend must return when its ctx is cancelled. A backend that blocks in code with no context support (a legacy client, a bare conn.Read) keeps its goroutine alive after FirstSuccess has returned. The caller is still released on time, but the goroutine lives until that blocked call returns. The fix belongs inside the backend: pass ctx into the HTTP request (http.NewRequestWithContext), or set SetReadDeadline on a connection (a time after which a blocked read returns an error) and close it from a goroutine that watches ctx.Done(). These last techniques are for backends that cannot take a context directly.

Trade-offs and pitfalls

• Cost. Sending every request to every backend multiplies load by the backend count. If that is too costly, hedge (a hedged request: start the first call, and start the next only after a delay with no answer, so you pay for the second call only when the first is slow). The cancellation and buffering structure above stays the same.
• errgroup is the wrong fit here. errgroup.WithContext cancels the group on the first error, which is the opposite of "an early failure must not stop a slower success".
• Do not close the results channel from a worker. Several senders and a closer is a classic panic ("send on closed channel"). This design never closes it; the buffer makes closing unnecessary.
• Verify with the race detector and a leak count. -race finds data races, not leaks, so the program also compares runtime.NumGoroutine() before and after. Production code usually uses a leak-detection test helper (a library such as goleak that fails a test if goroutines are still running when it ends) for the same job.

Direct answer

Use the standard library's atomic integer. atomic.Int64 (Go 1.19 and later) turns n++ into one indivisible read-modify-write (on x86-64 a single LOCK XADD instruction; other CPUs use an equivalent atomic-add instruction or a short retry loop), so no lock is needed:

package main

import (
	"fmt"
	"sync"
	"sync/atomic"
)

// Counter is safe for concurrent use and contains no mutex.
type Counter struct{ n atomic.Int64 }

func (c *Counter) Inc()         { c.n.Add(1) }
func (c *Counter) Value() int64 { return c.n.Load() }

// PlainCounter is the broken baseline.
type PlainCounter struct{ n int64 }

func (c *PlainCounter) Inc()         { c.n++ }
func (c *PlainCounter) Value() int64 { return c.n }

type inc interface {
	Inc()
	Value() int64
}

func run(c inc) int64 {
	var wg sync.WaitGroup
	for g := 0; g < 8; g++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for i := 0; i < 100000; i++ {
				c.Inc()
			}
		}()
	}
	wg.Wait()
	return c.Value()
}

func main() {
	fmt.Println("atomic:", run(&Counter{}), "want 800000")
	fmt.Println("plain: ", run(&PlainCounter{}), "want 800000 (often less)")
}

Why the plain version is wrong

c.n++ is three steps: load the value, add one, store it back. Two goroutines can both load 41 and both store 42, so one increment is lost. This is a data race (two goroutines touching one variable, at least one writing, with no synchronization ordering them). Run in a golang:1.23 container with go run -race and several CPUs available, the plain counter produced WARNING: DATA RACE reports and a total below 800000, and the atomic counter printed 800000. The atomic counter prints 800000 every time, with or without -race. The plain counter's total changes from run to run and depends on how many CPUs the process may use: with 4 or more CPUs it usually prints a total below 800,000, though a run can still land exactly on 800000, with 2 CPUs the total is often exactly 800000 even though -race still reports the race, and restricted to a single CPU (for example docker run --cpuset-cpus=0) it printed 800000 and -race reported nothing, because each goroutine tends to finish its loop before the next one gets the CPU. So a correct total from the plain version proves nothing, and a clean -race run on a one-CPU machine is not proof of safety; the data race is in the code either way.

Why the atomic version is correct, and what it guarantees

• Indivisible update: Add is an atomic read-modify-write, so no increment can be lost.
• Visibility and order: the Go memory model says all sync/atomic operations behave as if executed in one sequentially consistent order, with the same semantics as C++ sequentially consistent atomics and Java volatile. "Sequentially consistent" means every goroutine agrees on a single global order of the atomic operations, and that order respects the order in each goroutine's own code, so no goroutine can see a later atomic step before an earlier one. In practice: Value() sees every Inc() that finished before it started.
• Mutex-free is not contention-free. ("Lock-free" is a technical term for algorithms where some thread always makes progress without waiting on a lock; it says nothing about speed.) On a machine with many cores, every Add on the same counter needs exclusive ownership of the same cache line (the unit of memory the CPU caches move between cores), so cores take turns. Picture two cores each wanting to write the same line: the line travels from one core's cache to the other's and back on every increment, a "cache line ping-pong", and that transfer, not the add itself, is the cost. Atomics are cheaper than a mutex but still serialize on one hot location (a location many goroutines update at once).

Mutex version for comparison

type MutexCounter struct {
	mu sync.Mutex
	n  int64
}
func (c *MutexCounter) Inc()         { c.mu.Lock(); c.n++; c.mu.Unlock() }
func (c *MutexCounter) Value() int64 { c.mu.Lock(); defer c.mu.Unlock(); return c.n }

The benchmark is a counter_test.go file in the same package as the first listing, with MutexCounter above saved beside it (a mutex.go file with package main and import "sync"). It only calls Inc:

package main

import "testing"

func BenchmarkAtomic(b *testing.B) {
	var c Counter
	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			c.Inc()
		}
	})
}

func BenchmarkMutex(b *testing.B) {
	var c MutexCounter
	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			c.Inc()
		}
	})
}

go test -bench . -run XXX -cpu 1,8 in a golang:1.23 container on a 14-core aarch64 host gave one sample run:

The absolute figures depend on the CPU and on having at least 8 cores free for the -cpu 8 column, and the -cpu 8 values move noticeably from run to run; the ordering (atomic faster than mutex) held in every run here. Two readings: the atomic is faster in both columns, and both get much slower per operation at 8 goroutines than at 1, because of contention on one cache line. Contention, not the lock type, dominates at high core counts: the line ping-pong above is why 8 goroutines pay far more per increment than 1.

Trade-offs in a production server

Recommend the atomic for a standalone counter. Switch to a mutex the moment the counter must stay consistent with other state (for example "increment, and if over the limit, reject and record why"), because atomics cannot span two variables. Sharding is rarely needed: reach for it only when a profile shows the counter itself as a hotspot on a hot path (code that runs for nearly every request). The idea: keep, say, 16 counters instead of one, have each caller increment one chosen by a cheap spreading rule (a per-worker index, or a hash or random number taken on each call, since Go gives a goroutine no id to use), and make Value() add all 16. Each counter is padded (extra unused bytes) so it sits alone on its own cache line; otherwise neighbouring counters share a line and the ping-pong returns. Writes spread across 16 lines, and reads get slower because they touch all 16.

A goroutine-owned version

If the interviewer rules out atomics as well, one goroutine can own the number:

type OwnedCounter struct {
	incs  chan struct{}
	reads chan chan int64
	stop  chan struct{}
}

func NewOwnedCounter() *OwnedCounter {
	c := &OwnedCounter{
		incs:  make(chan struct{}, 1024), // buffered so Inc rarely blocks
		reads: make(chan chan int64),
		stop:  make(chan struct{}),
	}
	go func() {
		var n int64 // touched only by this goroutine
		for {
			select {
			case <-c.incs:
				n++
			case r := <-c.reads:
				for len(c.incs) > 0 { // count increments already queued before answering
					<-c.incs
					n++
				}
				r <- n
			case <-c.stop:
				return
			}
		}
	}()
	return c
}

func (c *OwnedCounter) Inc() { c.incs <- struct{}{} }
func (c *OwnedCounter) Value() int64 {
	r := make(chan int64, 1)
	c.reads <- r
	return <-r
}
func (c *OwnedCounter) Close() { close(c.stop) }

Driven by run(NewOwnedCounter()) from the first listing (8 goroutines each calling Inc 100,000 times, then Value() after they all returned), it prints 800000 under -race, with one CPU and with several. The drain loop in the read case matters for correctness: Inc returns as soon as its token is in the buffered channel, so some increments may still be queued when a read arrives, and the select could pick the read first. Without draining, Value() could miss increments that completed before it was called. It is correct but slower than both alternatives (each Inc is a channel send) and adds a goroutine to start and stop, so it is a fallback for when the state is richer than one number, not a better counter. It also has a lifecycle to respect: after Close, the owner goroutine is gone, so Value() blocks forever and Inc blocks once the 1024-slot buffer fills; callers must stop using the counter before closing it.

Direct answer

A goroutine blocked in conn.Read is not watching any context, so cancelling a context does nothing for it. Only closing the connection (or an expired read deadline) wakes it. Make shutdown deterministic (the same ordered sequence every time, with a bounded wait) with four steps in a fixed order (the numbered list further down adds the exit-path cleanup those steps rely on and an idle timeout for normal operation): stop accepting new connections (a listener is the socket that waits for incoming connections, and Accept is the call that takes the next one), close every tracked connection so each blocked read returns an error, wait for all handler goroutines with a sync.WaitGroup, and bound that wait with a deadline. To confirm nothing leaks in production, graph the goroutine count against load and, when it drifts, take goroutine profiles (pprof, Go's built-in profiler, can list every goroutine with its stack) and compare them to see which stack is accumulating. Start with the graph; the other tools are for locating the leak once the graph shows one.

Reproduction and fix

A goroutine is Go's lightweight thread. In this program the "naive" server closes only its listener on shutdown, which is what many first versions do; the "fixed" server also tracks each connection and closes it. Five idle clients keep five handler goroutines blocked in Read. Run with go run -race . (go.mod: module shutdowndemo, go 1.23).

package main

import (
	"bytes"
	"context"
	"fmt"
	"net"
	"runtime"
	"runtime/pprof"
	"strings"
	"sync"
	"time"
)

type Server struct {
	ln     net.Listener
	wg     sync.WaitGroup
	mu     sync.Mutex
	conns  map[net.Conn]struct{}
	track  bool // true: the fixed server tracks and closes connections
	closed bool // set by Shutdown under mu: no new connection may be registered after the sweep
}

func Start(track bool) *Server {
	ln, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		panic(err)
	}
	s := &Server{ln: ln, conns: map[net.Conn]struct{}{}, track: track}
	s.wg.Add(1)
	go s.acceptLoop()
	return s
}

func (s *Server) Addr() string { return s.ln.Addr().String() }

func (s *Server) acceptLoop() {
	defer s.wg.Done()
	for {
		c, err := s.ln.Accept()
		if err != nil {
			return // listener closed: stop accepting
		}
		s.mu.Lock()
		if s.closed { // Accept returned just before Shutdown swept the registry
			s.mu.Unlock()
			c.Close()
			return
		}
		s.conns[c] = struct{}{}
		s.wg.Add(1)
		s.mu.Unlock()
		go s.handle(c)
	}
}

func (s *Server) handle(c net.Conn) {
	defer s.wg.Done()
	defer func() {
		c.Close()
		s.mu.Lock()
		delete(s.conns, c)
		s.mu.Unlock()
	}()
	buf := make([]byte, 512)
	for {
		if _, err := c.Read(buf); err != nil { // blocks until data, close or error
			return
		}
	}
}

// Shutdown stops accepting, unblocks every handler, and waits for them,
// bounded by ctx. It returns how many goroutines were still running.
func (s *Server) Shutdown(ctx context.Context) error {
	s.ln.Close()
	if s.track {
		s.mu.Lock()
		s.closed = true // from here on, a late accept closes its own connection
		for c := range s.conns {
			c.Close() // makes the blocked Read return an error
		}
		s.mu.Unlock()
	}
	done := make(chan struct{})
	go func() { s.wg.Wait(); close(done) }()
	select {
	case <-done:
		return nil
	case <-ctx.Done():
		return ctx.Err()
	}
}

func goroutineProfileTop() string {
	var b bytes.Buffer
	pprof.Lookup("goroutine").WriteTo(&b, 1)
	for _, line := range strings.Split(b.String(), "\n") {
		if strings.Contains(line, "main.(*Server).handle") {
			f := strings.Fields(line)
			return f[len(f)-1][strings.LastIndex(f[len(f)-1], "/")+1:]
		}
	}
	return "none"
}

func run(name string, track bool) {
	base := runtime.NumGoroutine()
	s := Start(track)
	var clients []net.Conn
	for i := 0; i < 5; i++ {
		c, err := net.Dial("tcp", s.Addr())
		if err != nil {
			panic(err)
		}
		clients = append(clients, c) // idle clients: the server's Read blocks
	}
	time.Sleep(100 * time.Millisecond)
	fmt.Printf("%s: running, extra goroutines = %d\n", name, runtime.NumGoroutine()-base)

	ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
	err := s.Shutdown(ctx)
	cancel()
	time.Sleep(50 * time.Millisecond)
	fmt.Printf("%s: shutdown err = %v, extra goroutines = %d\n", name, err, runtime.NumGoroutine()-base)
	if err != nil {
		fmt.Printf("%s: stuck at %s\n", name, goroutineProfileTop())
	}
	for _, c := range clients {
		c.Close() // test cleanup, so the next run starts clean
	}
	time.Sleep(100 * time.Millisecond)
}

func main() {
	run("naive", false)
	run("fixed", true)
}

Output, identical across six consecutive runs in golang:1.23:

naive: running, extra goroutines = 6
naive: shutdown err = context deadline exceeded, extra goroutines = 6
naive: stuck at main.go:67
fixed: running, extra goroutines = 6
fixed: shutdown err = <nil>, extra goroutines = 0

The six extra goroutines while running are the accept loop plus five handlers. After the naive shutdown the accept loop has exited, so the six are five handlers still parked (asleep, waiting for an event) on line 67 (the c.Read call) plus the helper goroutine inside Shutdown that is itself stuck in wg.Wait(). That helper is a second, smaller leak: when the context expires, Shutdown returns but the waiter goroutine lives on until the handlers finish. In a real process that exits right after the timeout this does not matter; in a test it shows up in the count.

For reference, goroutineProfileTop asks the runtime for the goroutine profile and returns the file and line of the first stack that contains main.(*Server).handle; that is how the output names line 67.

What a goroutine profile looks like. The debug=1 text for the naive server, trimmed to the interesting stack (program-counter addresses, function offsets and the standard-library path prefix removed), reads:

goroutine profile: total 7
5 @ ...
#	internal/poll.runtime_pollWait
#	internal/poll.(*FD).Read
#	net.(*conn).Read
#	main.(*Server).handle	main.go:67
1 @ ...
#	sync.(*WaitGroup).Wait
#	main.(*Server).Shutdown.func1	main.go:86

Read it as: each block is one distinct stack, the number at the left is how many goroutines share it, and the frames below list the call chain from the blocking call (runtime_pollWait, the network wait) up to your own code. Five goroutines parked in net.(*conn).Read called from main.(*Server).handle at line 67 is the leak signature; the single WaitGroup.Wait goroutine is the shutdown helper described above. The profile also contains the goroutine that is writing the profile itself (the remaining one of the total of 7), which is always present. In production the same shape appears with a count like 5000 @ instead of 5, growing between two snapshots.

Making shutdown deterministic

1. Stop accepting first. Closing the listener makes Accept return an error, so the accept loop exits. After the listener is closed no new connection can be accepted, but one that Accept has already returned may not be registered yet. Registration and the sweep therefore share a lock and a closed flag (as in the code above): a connection that arrives late is closed by the accept loop itself, which is what lets the next steps be exhaustive. Without the flag, a burst of connects during shutdown can leave a handler blocked in Read that the sweep never saw, and Shutdown then times out. A test that dials in a loop from several goroutines while Shutdown runs, and requires it to return nil, exercises this window.
2. Wake every blocked reader. Close each tracked connection (as above) or call conn.SetReadDeadline(time.Now()), which makes a pending read fail immediately with a timeout error. Closing is the blunt tool and is correct when the server is stopping.
3. Make every handler's exit path run its cleanup. defer the close, the wg.Done() and any release of pooled resources, so a read error ends the loop and the handler leaves cleanly.
4. Wait with a bound. wg.Wait() has no timeout of its own, so run it in a goroutine and select on its completion versus ctx.Done(). On expiry, log how many connections remain and exit anyway; a hung handler must not block a deploy forever.
5. Add an idle timeout in normal operation. Setting a read deadline before each Read (for example 60 seconds) means a client that vanishes without a FIN (the TCP packet a client sends to say it is closing the connection; a phone that drops off the network never sends it), such as a phone leaving coverage, cannot hold a goroutine for hours.

If the service uses net/http, Server.Shutdown(ctx) does steps 1 and 4 for ordinary requests: it closes the listeners, closes idle connections, and waits for active ones to become idle, returning the context's error if the deadline passes first. The Go documentation also states that it does not close or wait for hijacked connections (connections a handler has taken over from the HTTP server, as WebSocket upgrades do). Those long-lived connections need your own registry, notified from a function registered with Server.RegisterOnShutdown.

Confirming in production that nothing leaks

• A metric (do this first). Export runtime.NumGoroutine() (the Prometheus Go client's go_goroutines gauge does this). Healthy services show a count that tracks load and returns to a baseline. A leak shows as a staircase that does not come back down after traffic falls.
• A snapshot that names the culprit. With net/http/pprof registered on an internal-only port, GET /debug/pprof/goroutine?debug=1 groups goroutines by identical stack with counts, and ?debug=2 prints each goroutine with how long it has been blocked (for example [IO wait, 47 minutes]). A leak is a stack with a large and growing count parked on the same line. IO wait in that header means the goroutine is parked waiting for the network.
• A comparison. Save two profiles an hour apart and run go tool pprof -base first.pb.gz second.pb.gz (profile from /debug/pprof/goroutine) to see only what grew.
• A regression test. In tests, go.uber.org/goleak fails a test if extra goroutines remain: defer goleak.VerifyNone(t) per test, or goleak.VerifyTestMain(m) for the whole package (the second form is the one that works with parallel tests).
• The shutdown itself. Log the number of handlers still alive when the shutdown deadline expires. A line that says 0 on every deploy is the production proof of deterministic shutdown, and a nonzero count carries its own stack dump.

Trade-offs and pitfalls

• Closing a connection mid-request drops in-flight work. For request/response protocols, drain first (stop accepting, let active requests finish up to the deadline), then close what remains. Idle connections can be closed immediately.
• context.Context alone is not enough. The standard Read API takes no context. Libraries that do (http.NewRequestWithContext, database/sql's QueryContext) can be cancelled; raw sockets need a deadline or a close.
• The leak is usually one missing exit path, not many. Every goroutine launched with go needs a stated answer to "what makes this return", written next to the go statement.

Direct answer

On SIGTERM, flip a "shutting down" flag from the signal handler and then run five steps in a fixed order: (1) close the listening socket so no new connections arrive, (2) wait for in-flight requests up to a configurable DRAIN_TIMEOUT, (3) cancel whatever is still running after that deadline and wait for it to unwind, (4) cancel the long-running background tasks, (5) flush metrics last, when nothing can change them any more, then return from asyncio.run so the process exits with code 0. The order matters: background tasks and metrics are used by requests, so they go after the requests, and the signal handler must only set a flag, not do the work.

Terms used here: SIGTERM is the signal an orchestrator (Kubernetes, systemd) sends to ask a process to stop, followed later by an unstoppable SIGKILL; draining means finishing work already accepted while refusing new work; the event loop is asyncio's single-threaded scheduler; a task is a coroutine the loop is running; cancellation raises CancelledError inside a task at its next await.

The server

This is a self-contained asyncio server (standard library only, so nothing to install) that answers GET /fast after 0.2 s and GET /slow after 3 s. It speaks HTTP by hand so that nothing hides the mechanics: an HTTP/1.1 request is a request line (GET /slow HTTP/1.1) followed by header lines and a blank line, so handle reads the first line, then reads and discards lines until the blank one. A response is a status line (HTTP/1.1 200 OK), headers (Content-Length tells the client how many body bytes follow, Connection: close says the connection ends afterwards), a blank line and the body. A framework does exactly this for you:

import asyncio, os, signal, sys

DRAIN_TIMEOUT = float(os.environ.get("DRAIN_TIMEOUT", "1.0"))   # seconds allowed for in-flight requests

class App:
    def __init__(self):
        self.shutting_down = asyncio.Event()
        self.inflight = set()                  # handler tasks currently serving a connection
        self.background = []                   # long-running tasks (metrics ticker, etc.)
        self.metrics = {"served": 0, "aborted": 0}

    async def handle(self, reader, writer):
        task = asyncio.current_task()
        self.inflight.add(task)
        try:
            if self.shutting_down.is_set():                            # accepted just before the listener closed
                writer.write(b"HTTP/1.1 503 Service Unavailable\r\nContent-Length: 0\r\nConnection: close\r\n\r\n")
                await writer.drain()
                return
            request_line = (await reader.readline()).decode().strip()   # e.g. "GET /slow HTTP/1.1"
            while (await reader.readline()) not in (b"\r\n", b"\n", b""):
                pass                                                    # skip headers
            delay = 3.0 if "/slow" in request_line else 0.2
            await asyncio.sleep(delay)                                  # pretend work
            body = b"done\n"
            hdr = b"HTTP/1.1 200 OK\r\nContent-Length: %d\r\nConnection: close\r\n\r\n" % len(body)
            writer.write(hdr + body)
            await writer.drain()
            self.metrics["served"] += 1
        except asyncio.CancelledError:
            self.metrics["aborted"] += 1                                # record, then let cancellation continue
            raise
        finally:
            self.inflight.discard(task)
            writer.close()

    async def ticker(self):
        try:
            while True:
                await asyncio.sleep(0.5)                                # periodic background job
        except asyncio.CancelledError:
            print("ticker: cancelled", flush=True)
            raise

    def request_shutdown(self, signame):
        if self.shutting_down.is_set():
            print(f"{signame} again: already shutting down", flush=True)   # signals can arrive twice
            return
        print(f"{signame} received: starting shutdown", flush=True)
        self.shutting_down.set()

    async def run(self, port):
        loop = asyncio.get_running_loop()
        for sig in (signal.SIGTERM, signal.SIGINT):
            loop.add_signal_handler(sig, self.request_shutdown, sig.name)

        server = await asyncio.start_server(self.handle, "127.0.0.1", port)
        self.background.append(asyncio.create_task(self.ticker()))
        print("listening", flush=True)

        await self.shutting_down.wait()

        # 1. stop accepting: close the listening socket. Existing connections stay open.
        server.close()
        print("step 1: listener closed", flush=True)

        # 2. drain: wait for in-flight handlers, but only up to DRAIN_TIMEOUT.
        deadline = loop.time() + DRAIN_TIMEOUT
        while self.inflight and loop.time() < deadline:
            await asyncio.wait(set(self.inflight), timeout=deadline - loop.time())
        still = set(self.inflight)
        print(f"step 2: drain over, still running {len(still)}", flush=True)
        # 3. force: cancel what is left and wait for it to unwind.
        for t in still:
            t.cancel()
        await asyncio.gather(*still, return_exceptions=True)

        # 4. stop background tasks AFTER requests, since requests may still use them.
        for t in self.background:
            t.cancel()
        await asyncio.gather(*self.background, return_exceptions=True)

        # 5. flush metrics last, once nothing can change them.
        print("step 5: flush metrics", self.metrics, flush=True)
        await server.wait_closed()

if __name__ == "__main__":
    asyncio.run(App().run(int(sys.argv[1])))

Signal handling is loop.add_signal_handler, not signal.signal: the loop calls the handler on its own thread, between tasks, so the handler can safely touch asyncio objects. It only sets an Event (a flag that other tasks can wait on); the real work happens in run() after await self.shutting_down.wait() returns.

Proving it, with a driver

import asyncio, os, signal, subprocess, sys, time

async def get(port, path):
    try:
        r, w = await asyncio.open_connection("127.0.0.1", port)
    except OSError as e:
        return f"{path}: connection refused"
    w.write(f"GET {path} HTTP/1.1\r\nHost: x\r\n\r\n".encode())
    await w.drain()
    data = await r.read()
    w.close()
    status = data.split(b"\r\n")[0].decode() if data else "closed with no response"
    return f"{path}: {status}"

async def main():
    port = 8765
    p = subprocess.Popen([sys.executable, "server.py", str(port)], stdout=subprocess.PIPE, text=True,
                         env={**os.environ, "DRAIN_TIMEOUT": os.environ.get("DRAIN_TIMEOUT", "1.0")})
    assert p.stdout.readline().strip() == "listening"
    fast = asyncio.create_task(get(port, "/fast"))
    slow = asyncio.create_task(get(port, "/slow"))
    await asyncio.sleep(0.1)                  # both requests are now in flight
    p.send_signal(signal.SIGTERM)
    time.sleep(0.05); p.send_signal(signal.SIGTERM)    # a second SIGTERM
    await asyncio.sleep(0.2)
    late = await get(port, "/late")
    print("client:", await fast)
    print("client:", await slow)
    print("client:", late)
    out, _ = p.communicate(timeout=10)
    print("--- server log ---"); print(out.strip())
    print("exit code:", p.returncode)

asyncio.run(main())

The driver (save it next to the server file, which must be named server.py because the driver launches it by that name) starts the server, opens /fast and /slow requests, sends SIGTERM twice (a second signal is a common real event), then tries a new connection. Run in a python:3.12-slim container (Python 3.12.15) with the default 1 second drain timeout, it printed (output order of client lines is deterministic; the server log is the child's own stdout):

client: /fast: HTTP/1.1 200 OK
client: /slow: closed with no response
client: /late: connection refused
--- server log ---
SIGTERM received: starting shutdown
step 1: listener closed
SIGTERM again: already shutting down
step 2: drain over, still running 1
ticker: cancelled
step 5: flush metrics {'served': 1, 'aborted': 1}
exit code: 0

Repeated five times, the metrics line 'served': 1, 'aborted': 1 appeared every time. With DRAIN_TIMEOUT=5 (enough for the 3-second request) the same driver printed /slow: HTTP/1.1 200 OK, still running 0 and {'served': 2, 'aborted': 0}. So the fast request finished, the late connection was refused, the slow request was cut off when the timeout expired and was counted, the background ticker was cancelled, metrics reflect the final state, and the exit code was 0.

The timeline of that run, with times measured from the first SIGTERM: at 0 s two requests are in flight, /fast (due at 0.2 s) and /slow (due at 3 s); the listener closes at once, so the later /late connection is refused. The deadline is 0 s + 1.0 s = 1.0 s. /fast finishes at 0.2 s and leaves inflight. The drain loop keeps waiting for /slow until the deadline, finds it still running, cancels it (its except asyncio.CancelledError counts it as aborted), and the process moves on to steps 4 and 5.

Choosing the timeout

Set DRAIN_TIMEOUT below the platform's kill deadline. Kubernetes, for example, sends SIGKILL after terminationGracePeriodSeconds (the pod setting for how long it waits between SIGTERM and SIGKILL; 30 s by default), so a 25 s drain leaves time for steps 3 to 5. If requests must be allowed longer than that, raise the grace period in the deployment rather than the app timeout alone.

Race conditions to watch for

1. Signal arrives twice. Because the handler only sets an Event, a second SIGTERM is harmless: setting an already-set event changes nothing, and the shutdown sequence in run() executes exactly once. The is_set() check only keeps the log from announcing a second shutdown. It would start to matter if the handler itself launched work, for example by creating a shutdown task, so keep the handler a flag-setter.
2. Connection accepted just before the listener closed. The loop may have accepted it and scheduled its handler task, but the handler has not run yet, so it is not in inflight. Timeline: at 0 s the listener closes; a client's connection was accepted a moment earlier and its handler task exists but has not yet run, so inflight is empty; the drain condition while self.inflight and ... is false at once, so step 2 waits for nothing and the sequence runs on to the metrics flush before that handler starts, perhaps 0.001 s later, and begins a request. The drain loop's re-read of self.inflight does not close this gap: it picks up handlers that register while the loop is already waiting, not one that registers after the first empty check. What protects the late handler is the first branch of handle: once the flag is set it answers 503 immediately, so it does no real work and cannot outlive the drain deadline. With that branch removed, a late /slow request is served in full after the metrics line has been printed, and the served counter misses it. On Python 3.12, server.wait_closed() waits for active connections to finish (the documentation says it waits "until the close() method completes and all active connections have finished"), which is why the 503 is written before the process exits. asyncio.wait(tasks, timeout=...) returns when all the given tasks are done or the timeout passes (it does not raise for task errors), which is why the loop calls it again with a fresh set(self.inflight) each time round: that picks up any handler that registered during the wait.
3. Cancelling a handler mid-write. CancelledError can arrive between a database commit and the response. Make writes idempotent (running the same request twice has the same effect as running it once, for example an INSERT keyed by a client-supplied request id, so a retry after a cut-off is harmless) or finish them under asyncio.shield (which lets the inner commit run to completion even if the awaiting handler is cancelled, for example await asyncio.shield(commit_order(order))). Shield alone is not enough at shutdown: once the handler is cancelled nobody awaits the inner task any more, and asyncio.run cancels every task still pending when it closes the loop, so a shielded commit that is still running at exit is cancelled too. Keep the shielded tasks in a set and await them, with their own timeout, in step 3 before moving on.
4. Swallowed cancellation. A handler that catches Exception is fine (CancelledError derives from BaseException, not Exception, since Python 3.8, so except Exception does not catch it and cancellation passes through), but a bare except: or a try/finally that awaits forever blocks gather in step 3. The server above records the abort and re-raises.
5. Ordering of steps 4 and 5. Cancelling the background tasks before the requests finish can break requests that call them; flushing metrics before the cancelled handlers update their counters loses data. The sequence in run() avoids both.
6. Load balancer lag. Closing the listener stops new connections at this process, but a load balancer may keep routing for a few seconds. Fail the readiness probe first (a readiness probe is the health check the load balancer or Kubernetes calls to decide whether to send this instance traffic; flip a readiness flag in the same signal handler so it starts answering "not ready") and, if needed, sleep a short interval before step 1.
7. Dependencies closed too early. Close database pools and HTTP clients after step 3, never before.

Verify a change with the driver: it fails visibly if the late connection is accepted, if the slow request is not counted, or if the exit code is not 0.

Direct answer

The C++11 memory model says what values a thread may see when other threads write the same memory. Every std::atomic operation takes a memory_order argument that chooses how much ordering it imposes on the other reads and writes around it. The default, memory_order_seq_cst (sequentially consistent), behaves as if all such operations ran in one global order that every thread agrees on. The weaker orders buy speed on weakly ordered CPUs (ARM, POWER: processors whose hardware is allowed to make one core's writes visible to others in a different order than the program wrote them, unlike x86, which reorders far less) by promising less. A metrics counter is safe with relaxed, because only the counter's own value matters. A "ready" flag that guards other data is wrong with relaxed, because the data is not ordered against the flag.

Terms used here: an atomic is a variable whose reads and writes cannot be torn or interleaved mid-operation; a data race is two threads touching the same non-atomic memory with at least one write and no ordering between them, which is undefined behaviour (the compiler and CPU may do anything); reordering means the compiler or CPU makes memory operations take effect in a different order than the source lists them; a read-modify-write (RMW) operation such as fetch_add reads and writes in one indivisible step.

What each order promises

The release-acquire pair is what the standard calls synchronizes-with: the acquire load that reads the value stored by the release store creates a happens-before edge (a guarantee that everything before the store is visible after the load). Without that edge, a plain variable written by one thread and read by another is a data race. "Synchronizes-with" is just the name for that pairing: a specific release store and the specific acquire load that read its value.

Where relaxed is safe: a service metrics counter

#include <atomic>
#include <cstdio>
#include <thread>
#include <vector>

std::atomic<long> requests_total{0};     // statistics only: nobody branches on it to read other data

int main() {
    std::vector<std::thread> workers;
    for (int t = 0; t < 8; ++t)
        workers.emplace_back([] {
            for (int i = 0; i < 100000; ++i)
                requests_total.fetch_add(1, std::memory_order_relaxed);
        });
    for (auto& w : workers) w.join();
    std::printf("requests_total=%ld\n", requests_total.load(std::memory_order_relaxed));
}

Compiled in a gcc:14 container (GCC 14.4, aarch64) with g++ -O1 -g -std=c++20 -fsanitize=thread metrics_counter.cpp, it prints requests_total=800000 and ThreadSanitizer (TSan, GCC's data-race detector) reports nothing. The increments never lose an update because each fetch_add is atomic; their relative order does not matter because addition commutes (1+2+3 and 3+2+1 give the same total, so it does not matter which thread's increment lands first) and the total is read after join() (which itself synchronizes). relaxed is also fine for a sampled gauge that a dashboard thread reads occasionally, because a slightly stale value is acceptable.

Where relaxed is wrong: a flag that publishes other data

#include <atomic>
#include <cstdio>
#include <thread>

struct Config { int timeout_ms; int retries; };

Config cfg;                               // plain data, not atomic
std::atomic<bool> ready{false};

#ifdef FIXED                    // building with -DFIXED picks release/acquire; otherwise relaxed
constexpr auto kPublish = std::memory_order_release;
constexpr auto kObserve = std::memory_order_acquire;
#else
constexpr auto kPublish = std::memory_order_relaxed;   // WRONG: no ordering for cfg
constexpr auto kObserve = std::memory_order_relaxed;
#endif

int main() {
    std::thread loader([] {
        cfg = Config{250, 3};             // ordinary writes
        ready.store(true, kPublish);
    });
    std::thread server([] {
        while (!ready.load(kObserve)) {}
        std::printf("timeout=%d retries=%d\n", cfg.timeout_ms, cfg.retries);   // ordinary reads
    });
    loader.join(); server.join();
}

Saved as config_publish.cpp and built without -DFIXED and run under -fsanitize=thread, TSan reports (first lines of the report, trimmed; the line numbers are lines of the listing above):

WARNING: ThreadSanitizer: data race (pid=30)
  Read of size 4 at 0x0000004200d0 by thread T2:
    #0 operator() /w/config_publish.cpp:25
  Previous write of size 4 at 0x0000004200d0 by thread T1:
    #0 operator() /w/config_publish.cpp:20

Reading the report: thread T1 is the loader thread and T2 is the server thread (TSan numbers threads in creation order). "Read of size 4 ... by thread T2 ... :25" is the printf line reading cfg.timeout_ms (an int is 4 bytes). "Previous write of size 4 ... by thread T1 ... :20" is the line cfg = Config{250, 3}. TSan is saying: the same address was written by T1 and read by T2 with no happens-before edge between them, which is the definition of a data race. operator() is how a lambda's body appears in the stack.

The relaxed store of ready promises nothing about the earlier write to cfg, so the server may see ready == true and still read stale cfg fields (or, formally, trigger undefined behaviour). Built with -DFIXED (release store, acquire load) and run ten times under TSan, every run printed timeout=250 retries=3 with no report. The acquire load that sees true is the one that synchronizes with the release store, so cfg is fully visible.

How to choose

1. Default to seq_cst (just omit the argument). It is the easiest to reason about and is usually cheap enough; weaken it only where profiling shows it matters.
2. Use relaxed only when the atomic is the only shared thing the operation depends on: counters, statistics, unique-ID generators, a "stop requested" flag when the stopping thread does not need to read anything the other thread wrote.
3. Use release on the write and acquire on the read whenever an atomic announces that other data is ready (a flag, a pointer to a freshly built object, a queue index). Use acq_rel for an RMW that both consumes and publishes, such as the decrement in a reference count that may run the destructor. In outline:

// each owner, when it is done with the object:
if (refs.fetch_sub(1, std::memory_order_acq_rel) == 1) {
    delete obj;          // we were the last owner
}

The release half publishes everything this owner did to the object before it let go; the acquire half lets the thread that sees the count reach zero see everything every other owner did to the object, so the destructor never runs against stale data.
4. Reach for seq_cst again when two threads each write one variable and read the other and the code needs a single agreed order across them. Release and acquire alone do not give that on the C++ abstract machine. The classic case is store buffering. Thread 1 runs x = 1; r1 = y; and thread 2 runs y = 1; r2 = x; (all atomics, initially 0). Under seq_cst there is one global order of the four operations, and in any such order at least one store comes before the other thread's load, so r1 == 0 && r2 == 0 is impossible. With release stores and acquire loads there is no such single order, so both threads may read 0: each thread's store can still be sitting in its core's store buffer when the other thread's load runs.

The common mistake is reasoning about the C++ model by what x86 does. A relaxed-flag bug can look correct for years on x86 and fail the first day on an ARM server or a phone; TSan checks the C++ rules, not the CPU you happen to run on.