Concurrency, Synchronization & Deadlock Questions

Direct answer

counter += 1 is not one operation. The hardware does a load, an add and a store, and a second thread can run its own load between your load and your store. Both threads then compute from the same old value and one increment is overwritten (a lost update). Fix it by making the read-modify-write indivisible: protect it with a mutex, or use an atomic increment instruction. Choose an atomic for a single independent word; choose a lock when more than one variable or a check-then-act must change together.

Exactly how the interleaving loses an update

Here is what the compiler produced for void inc(void) { counter += 1; } with gcc:14 on aarch64 (gcc -O0 -S, then -O2 -S). The -O0 listing is adrp/add/ldr/add/adrp/add/str; at -O2 it is shorter but is still one load, one add and one store:

inc:                                  (gcc -O2, aarch64)
	adrp	x1, .LANCHOR0
	ldr	x0, [x1, #:lo12:.LANCHOR0]    ; load counter
	add	x0, x0, 1                      ; add 1 in a register
	str	x0, [x1, #:lo12:.LANCHOR0]    ; store back
	ret

Reading the listing: adrp loads the address of the 4 KB memory page that holds counter into register x1 (a register is one of the CPU's few fast storage slots); :lo12:.LANCHOR0 is the low 12 bits of the address, the offset inside that page, so [x1, #:lo12:.LANCHOR0] means "the memory at the counter's address". x0 is the register that does the arithmetic. The comments mark the three separate steps: load, add, store.

Starting from counter = 5:

Two increments ran, the value moved from 5 to 6. The window is only a few instructions wide, so any single increment is rarely hurt, but a million of them make it near-certain that some are.

Why it is intermittent

The loss needs the scheduler to switch threads, or another core to touch the cache line (the small block of memory, commonly 64 bytes, that cores hold in their private caches and hand to each other; only one core at a time may write it), inside that tiny window. Different runs interleave differently, and anything that changes timing (a print, a debugger, a different compiler flag) changes how often it happens. Below, five runs of the same program lost different amounts.

The fixes, compared in one harness

#include <pthread.h>
#include <stdatomic.h>
#include <stdio.h>
#define N 1000000
static long plain = 0;
static long locked = 0;
static atomic_long atom = 0;
static pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;

static void *work(void *arg) {
    (void)arg;
    for (int i = 0; i < N; i++) {
        plain += 1;                                   /* data race */
        pthread_mutex_lock(&m); locked += 1; pthread_mutex_unlock(&m);
        atomic_fetch_add_explicit(&atom, 1, memory_order_relaxed);
    }
    return NULL;
}
int main(void) {
    pthread_t t[2];
    for (int i = 0; i < 2; i++) pthread_create(&t[i], NULL, work, NULL);
    for (int i = 0; i < 2; i++) pthread_join(t[i], NULL);
    printf("expected=%d plain=%ld locked=%ld atomic=%ld\n", 2 * N, plain, locked, atom);
    return 0;
}

gcc -O0 -pthread counter.c -o c && ./c, five runs in a gcc:14 container (aarch64):

expected=2000000 plain=1975751 locked=2000000 atomic=2000000
expected=2000000 plain=1986956 locked=2000000 atomic=2000000
expected=2000000 plain=1998485 locked=2000000 atomic=2000000
expected=2000000 plain=1954567 locked=2000000 atomic=2000000
expected=2000000 plain=1982365 locked=2000000 atomic=2000000

Under -fsanitize=thread the plain line is reported as a data race at counter.c:13 and the locked and atomic lines print 2000000 with no report. (I used memory_order_relaxed for the atomic. A memory order says how much an atomic operation also orders the surrounding reads and writes: relaxed promises only that the operation itself is indivisible; acquire/release is the pairing used to hand other data from one thread to another; sequentially consistent, the default, makes all threads agree on one order for all such operations. The counter publishes no other data, so only the atomicity of the increment matters here.)

Which hardware instruction backs the atomic

The atomic increment compiles to a single indivisible read-modify-write, and which one depends on the target. From gcc:14: on x86-64 (--platform linux/amd64, -O2) it is lock addq $1, counter(%rip); on aarch64 with -march=armv8-a -mno-outline-atomics it is a load-exclusive / store-exclusive loop (ldxr, add, stxr, cbnz back if the store failed); with -march=armv8.1-a it is one ldadd. Reading these: lock addq is an ordinary add to memory with a lock prefix that makes the whole read-modify-write indivisible. ldxr loads the value and marks the location as exclusively watched by this core, stxr stores only if nobody wrote the location since and writes a success flag, and cbnz (compare and branch if nonzero) jumps back to retry when that flag says the store failed; so the exclusive pair retries if another core wrote the location in between. ldadd does the whole atomic add in one instruction. These instruction names are context for how hardware provides the guarantee; what matters is that the hardware offers one indivisible read-modify-write.

Lock or atomic

• Atomic: one word, one independent operation (counter, flag, sequence number, pointer swap). No sleeping, no lock to forget to release, safe from signal handlers (functions the OS runs asynchronously, interrupting your thread, when a signal arrives) and, for lock-free types (atomic types implemented with real atomic instructions rather than a hidden lock; is_lock_free() tells you), interrupt handlers.
• Mutex: more than one variable must change together, or you have check-then-act ("increment only if below the cap"), or the critical section calls other code. Two separate atomic variables are never one atomic update: AtomicInteger used and AtomicInteger limit read one after the other still allow another thread's change in between (the same holds for AtomicInteger on Android: each call is atomic, a sequence of calls is not).
• Under contention (many threads using the same variable at the same time) both serialise on the same cache line. A contended mutex can additionally put a waiting thread to sleep in the kernel and wake it later, which costs context switches; an atomic add just waits for the cache line. When one counter is hot, the usual real fix is neither: keep a counter per thread (or per shard, one of several independent slices of the data) and sum when read, so threads stop sharing the line. I did not time these variants here, so the comparison above is a reasoning claim, not a measurement.
• Shared cache (a map plus eviction order): the invariant spans several fields, so use a lock (sharded by key to reduce contention). A lock-free repair is realistic for one counter or one pointer, not for an invariant over a whole structure.

Pitfalls

• volatile does not make counter += 1 atomic; it only stops the compiler from caching the value.
• On an embedded target the same loss occurs between a task and an interrupt service routine; the fix there is a brief interrupt mask or a hardware atomic, not a mutex that the ISR cannot take.
• Reading the final value after pthread_join is safe because join orders the writes before the read.

Direct answer

A data race is a memory-level fact: two threads access the same memory location at the same time with no ordering between them, at least one access is a write, and the accesses are not both atomic (an atomic access is one the language guarantees happens as a single indivisible step that other threads see either completely or not at all, such as a C11 atomic_int operation; a plain int access is not). In C and C++ the language definition makes the whole program's behaviour undefined if one occurs. A race condition is a logic-level fact: the program's correctness depends on the timing or interleaving of operations, so some interleavings produce a wrong result. A program can have either without the other: you can lock every single access (no data race) and still check a balance in one critical section and withdraw in another (race condition).

The precise definitions

In plain words for the next paragraph: an evaluation is one execution of an expression, such as a read or a write of a variable; a memory location is a variable (or a distinct piece of one) that has its own address; and undefined behaviour means the language places no limit on what the program does. cppreference's statement of the C++ memory model says two evaluations conflict if one modifies a memory location and the other reads or modifies the same location, and a program with two conflicting evaluations has a data race unless both run on the same thread (or in the same signal handler), both are atomic operations, or one happens-before the other. It also says: "If a data race occurs, the behavior of the program is undefined." (Happens-before is the ordering a mutex unlock/lock pair, a thread join, or an atomic release/acquire pair gives you: everything before the unlock is visible after the matching lock.)

Worked example: both bugs in one program, run in a Linux container

One file, two parts. Part 1 is a data race (two threads increment a plain int). Part 2 is a race condition with no data race: every access to balance is under a mutex, but the check and the withdrawal are separate critical sections. A barrier forces both threads to finish their check before either acts, so the bad interleaving happens on every run instead of rarely.

#include <pthread.h>
#include <stdio.h>

/* Part 1: a data race. Two threads write the same int with no synchronization. */
static int hits = 0;
static void *bump(void *arg) { (void)arg; for (int i = 0; i < 100000; i++) hits++; return NULL; }

/* Part 2: a race CONDITION with no data race. Every access is under the mutex,
   but check and act are in separate critical sections. */
static pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;
static pthread_barrier_t both_checked;
static int balance = 100;
static int withdraw(int amt) {
    pthread_mutex_lock(&m);
    int ok = balance >= amt;          /* check */
    pthread_mutex_unlock(&m);
    pthread_barrier_wait(&both_checked); /* force the bad interleaving for the demo */
    if (!ok) return 0;
    pthread_mutex_lock(&m);
    balance -= amt;                   /* act on a stale check */
    pthread_mutex_unlock(&m);
    return 1;
}
static void *w(void *arg) { (void)arg; withdraw(80); return NULL; }

int main(void) {
    pthread_t a, b;
    pthread_create(&a, NULL, bump, NULL); pthread_create(&b, NULL, bump, NULL);
    pthread_join(a, NULL); pthread_join(b, NULL);
    printf("hits = %d (expected 200000)\n", hits);

    pthread_barrier_init(&both_checked, NULL, 2);
    pthread_create(&a, NULL, w, NULL); pthread_create(&b, NULL, w, NULL);
    pthread_join(a, NULL); pthread_join(b, NULL);
    printf("balance = %d (should never be negative)\n", balance);
    return 0;
}

Compiled and run with gcc:14 (GCC 14.4, aarch64 Linux container), five runs of gcc -O1 -pthread race.c -o r && ./r (each run prints both lines; the hits line is timing-dependent, so your five runs will differ):

hits = 100000 (expected 200000)
balance = -60 (should never be negative)
hits = 100000 (expected 200000)
balance = -60 (should never be negative)
hits = 100000 (expected 200000)
balance = -60 (should never be negative)
hits = 200000 (expected 200000)
balance = -60 (should never be negative)
hits = 100000 (expected 200000)
balance = -60 (should never be negative)

Note the fourth run: the plain int race printed the "right" answer once. That is the point of the next paragraph: the program is wrong on every run, and only the symptom varies.

Why 100000 in four of five runs at -O1? A register is a storage cell inside the CPU that is much faster than memory. The compiler is allowed to assume no data race, so it kept the counter in a register (it hoisted the load and store out of the loop, meaning it moved them before and after the loop instead of repeating them every iteration): the generated bump loads hits once, adds 100000, and stores once (I read the gcc -O1 -S output: one ldr, one str, a register loop between them). Walk through it with illustrative timing: thread A loads hits (0) into a register and adds 1 a hundred thousand times in the register, so the register holds 100000, then stores 100000 to memory; thread B did the same, starting from its own load of 0, and stores 100000 too. If the two threads overlap, both loaded 0, so whichever store lands last writes the same 100000 and one thread's whole contribution is overwritten. If one thread happens to finish completely before the other starts (thread start-up is slow compared with a 100000-iteration register loop, but it can happen, as in the fourth run), the second loads 100000 and stores 200000 and the symptom disappears for that run. So the result is 100000 or 200000 depending on timing, never a reliable total. In the ldr/str reading, ldr is the Arm instruction that loads from memory into a register and str stores a register back, so seeing only one of each around the loop is the evidence that the counter lived in the register. At -O0 the same program loses a variable amount: three runs printed 118833, 135721 and 125027. Both are "correct" outcomes of undefined behaviour, which is the practical meaning of UB: you cannot predict the symptom.

Under TSan (gcc -O1 -g -fsanitize=thread -pthread race.c -o rt && ./rt) the first part is reported as a data race at race.c:6 in bump (a read and a previous write by the two threads) and the count prints 200000, because instrumentation changes the code the compiler generates. A TSan data-race report names the kind of problem, then gives two blocks, one for the access that triggered it (read or write, its thread, its function and file:line) and one for the earlier conflicting access in the other thread, plus the location of the variable; two accesses to one address from two threads with no ordering between them is the whole diagnosis. The summary line is ThreadSanitizer: reported 1 warnings. Part 2 produced balance = -60 in that run too, and TSan said nothing about it: the accesses are all ordered by the mutex, so there is no data race to report. TSan's documentation describes the tool as one that "detects data races", so a clean TSan run says nothing about check-then-act bugs.

Why -60: both withdrawals checked 100 >= 80 before either subtracted, then 100 - 80 - 80 = -60. The fix is to make check and update one critical section (or a compare-and-swap loop on the balance):

lock; if (balance >= amt) balance -= amt, ok = 1; unlock

Why code can be free of data races and still be wrong

Synchronizing each variable protects the memory, not the rule that relates several operations. Typical shapes: check-then-act (above), read-modify-write split across two lock regions, "if absent then insert" on a locked map, two separately atomic variables that must change together, and iterating a collection whose size was read earlier.

The same race between a task and an interrupt service routine (ISR)

A counter updated by both a task and an ISR is the same defect. volatile stops the compiler from caching the variable, but cppreference notes that volatile access is suitable for communication with a signal handler, "but not with another thread of execution". count++ is still a load, an add and a store, and an ISR that fires between the load and the store loses an update; the fix is a short critical section (a stretch of code only one thread or interrupt handler may be inside at a time) with interrupts masked, or a hardware atomic read-modify-write where the core has one.

Trade-offs and pitfalls

• "Benign" data races do not exist in C/C++: even a racy flag can be hoisted out of a loop, as the register-resident counter above shows.
• Removing a data race with atomics does not fix a race condition; ask what invariant ties the operations together.
• Do not rely on a test passing: with a two-instruction window, a failure may need millions of iterations; use TSan for data races and invariant checks plus forced interleavings (as the barrier does) for logic races.

Design

Two requirements pull in different directions. Events can be posted from any thread, but handlers must run on one designated thread (for example a user-interface thread; on Android that is the main thread with its message queue, driven through Handler.post, and on iOS the main dispatch queue, DispatchQueue.main.async; both are a queue drained by one thread, which is what post and run build here in C++). And handlers may call subscribe, unsubscribe and post themselves, which is where naive designs deadlock. The structure that satisfies both:

1. A queue between posters and the delivery thread. post locks only the queue's mutex, pushes, signals a condition variable and returns. It never calls a handler, so a poster can never be blocked by someone else's callback.
2. A subscriber list that is replaced, never edited in place (copy-on-write: to change shared data, copy it, modify the copy, then switch everyone to the copy). subscribe and unsubscribe build a new vector under the list's mutex and swap in a shared_ptr (a reference-counted pointer that keeps the list alive while anyone still holds it) to it. The delivery thread takes a snapshot (a private reference to the list as it is right now) of the current pointer under the same mutex, releases the mutex, and then iterates the snapshot while calling handlers. No lock is held during any callback, so a handler that calls subscribe, unsubscribe or post cannot deadlock on a lock the dispatcher holds, and the vector being iterated is never modified.
3. A per-subscription active flag. A snapshot can still contain a subscription that was removed after the snapshot was taken. The dispatcher checks active immediately before each call, and unsubscribe clears it before removing the entry.

In the code, std::function is a holder for any callable (a handler), and std::atomic<bool> is a flag that can be read and written safely from several threads.

#include <atomic>
#include <condition_variable>
#include <deque>
#include <functional>
#include <iostream>
#include <memory>
#include <mutex>
#include <thread>
#include <vector>

struct Event { int type; int payload; };

class EventBus {
public:
    struct Subscription {
        std::function<void(const Event&)> handler;
        std::atomic<bool> active{true};
    };
    using Token = std::shared_ptr<Subscription>;

    Token subscribe(std::function<void(const Event&)> h) {
        auto s = std::make_shared<Subscription>();
        s->handler = std::move(h);
        std::lock_guard<std::mutex> lk(sub_m);
        auto next = std::make_shared<std::vector<Token>>(*subs);   // copy-on-write
        next->push_back(s);
        subs = next;
        return s;
    }
    void unsubscribe(const Token& t) {
        t->active.store(false);                  // takes effect for every later call, any thread
        std::lock_guard<std::mutex> lk(sub_m);
        auto next = std::make_shared<std::vector<Token>>();
        for (auto& s : *subs) if (s != t) next->push_back(s);
        subs = next;
    }
    void post(Event e) {                         // any thread; never calls a handler
        { std::lock_guard<std::mutex> lk(q_m); q.push_back(e); }
        q_cv.notify_one();
    }
    void stop() { post({-1, 0}); }
    void run() {                                 // the designated delivery thread
        for (;;) {
            Event e;
            {
                std::unique_lock<std::mutex> lk(q_m);
                q_cv.wait(lk, [&] { return !q.empty(); });
                e = q.front(); q.pop_front();
            }                                    // queue lock released before any callback
            if (e.type < 0) return;
            std::shared_ptr<std::vector<Token>> snap;
            { std::lock_guard<std::mutex> lk(sub_m); snap = subs; }   // lock released before callbacks
            for (auto& s : *snap)
                if (s->active.load()) s->handler(e);
        }
    }
private:
    std::mutex sub_m, q_m;
    std::condition_variable q_cv;
    std::deque<Event> q;
    std::shared_ptr<std::vector<Token>> subs = std::make_shared<std::vector<Token>>();
};

int main() {
    EventBus bus;
    std::atomic<bool> a_handled{false};          // set once A's handler has finished its re-entrant calls
    int a_calls = 0, b_calls = 0, c_calls = 0;   // touched only on the delivery thread
    EventBus::Token ta, tb;
    ta = bus.subscribe([&](const Event& e) {
        ++a_calls;
        if (e.payload == 1) {                    // re-entrant calls from inside a handler
            bus.unsubscribe(ta);                 // unsubscribe self
            bus.unsubscribe(tb);                 // and B, which comes later in this same dispatch
            bus.subscribe([&](const Event&) { ++c_calls; });   // C sees only later events
            bus.post({1, 1000});                 // posting from a handler cannot deadlock
            a_handled.store(true);
        }
    });
    tb = bus.subscribe([&](const Event&) { ++b_calls; });

    std::thread loop([&] { bus.run(); });
    bus.post({1, 1});                            // first event in the queue: triggers the above
    while (!a_handled.load()) std::this_thread::yield();   // the rest start only after A has finished, so the counts below do not depend on timing
    std::vector<std::thread> posters;
    for (int t = 0; t < 3; ++t)
        posters.emplace_back([&] { for (int i = 0; i < 1000; ++i) bus.post({1, 2}); });
    for (auto& p : posters) p.join();
    bus.stop();
    loop.join();
    // 3000 events from posters + 1 posted by the handler = 3001 events after the first one.
    std::cout << "a_calls=" << a_calls << " b_calls=" << b_calls << " c_calls=" << c_calls << "\n";
}

Built with g++ -std=c++20 -O1 -g -fsanitize=thread -pthread in a gcc:14 container and run 20 times, TSan printed no report and every run printed the line below. The main thread waits for A's handler to finish before the posters start; without that wait, a delivery thread that is slow to run A's handler could see the stop marker before A's own posted event and print c_calls=3000, so the count would depend on timing:

a_calls=1 b_calls=0 c_calls=3001

What the test shows: handler A, while handling the first event, unsubscribes itself, unsubscribes B (which comes later in the same dispatch), subscribes a new handler C and posts one more event. Afterwards A was called once, B never (the active check skipped it even though B was in the snapshot), and C received exactly the 3000 events posted by the three posting threads plus the one posted by A's handler, 3001 in total, with no event missed.

The delivery guarantees, stated

• No missed events for a new subscriber: the delivery thread takes its snapshot immediately before dispatching each event. A subscriber added while event E is being dispatched does not get E, and gets every event dispatched afterwards. A subscriber added after an event was posted but before it is dequeued does receive that event.
• Unsubscribe from the delivery thread (a handler removing itself or another handler): takes effect before the next handler call, as the example shows.
• Unsubscribe from another thread: a call that has already passed the active check may still start or be running when unsubscribe returns, and later events skip the handler. If you need "no callback is running when unsubscribe returns", additionally wait for the in-flight dispatch (for example by posting a marker event and waiting for it), and never do that wait from the delivery thread itself, which would wait for itself.
• Ordering: events are delivered in queue order, one at a time.

Failure modes this avoids

• Holding the subscriber lock while calling handlers: a handler that subscribes would block on the same lock (self-deadlock) and a handler that blocks would stall every subscriber change.
• Mutating a vector while iterating it, which invalidates iterators (undefined behaviour in C++: the language gives the program no defined meaning, so anything can happen).
• Handlers that post events in an unbounded loop would fill the queue; production code needs a size limit or back-pressure (making posters slow down or drop events when the queue is full).

Cost: each subscribe/unsubscribe copies the list (cheap when subscribers are few and dispatch is frequent, which is the usual shape); each dispatch costs one lock/unlock to snapshot.

Direct answer

If only one thread ever touches a piece of data, nothing can race on it, so it needs no lock. Thread confinement means giving each piece of mutable state exactly one owning thread (or one serial queue). Message passing means every other thread asks the owner to act by putting a message on a queue, instead of reaching into the data. The queue is then the only place threads meet, and it carries its own synchronization. You still need real synchronization at that queue, for data that is genuinely shared, and for anything that escapes the owner by pointer.

How it removes the lock

1. Pick the owner: a dedicated thread, a goroutine (Go's lightweight thread, started with the go keyword), or a serial dispatch queue (a queue, as in Apple's Grand Central Dispatch, that runs its tasks strictly one after another).
2. The owner is the only code that holds a reference to the state.
3. Other threads send small command or event messages.
4. The owner processes messages one at a time, so every operation on the state is already atomic with respect to every other operation.

The queue itself is synchronized by the language runtime. In Go a queue between goroutines is a channel, and a send on a channel is synchronized before the corresponding receive completes (Go memory model, go.dev/ref/mem), which means everything the sender wrote before sending is guaranteed visible to the receiver. Java documents the same kind of guarantee, called happens-before (an action that happens-before another is guaranteed to be visible to it), for concurrent collections: actions before placing an object into one happen-before actions after taking it out in another thread.

Two familiar examples. The Android UI toolkit is not thread-safe, and its developer documentation tells you not to access it from outside the UI thread: the main thread owns every view, and worker threads post results to it. A server can give one goroutine ownership of a counter map and let request handlers send events to it.

Worked example, run with the Go race detector

The Go syntax to know: <-chan event is a channel the function may only receive from and chan<- map[string]int one it may only send on; for e := range in receives until the channel is closed; defer wg.Done() runs when the function returns, telling the WaitGroup (a counter the main goroutine waits on) that one worker has finished. The owner goroutine below is the only code that touches counts. Four goroutines send 1,000 events each. When the owner finishes it sends the whole map to the receiver and never touches it again, which is a clean hand-over of ownership. The second half shows the hole in the pattern: a message that carries a pointer.

package main

import (
	"fmt"
	"os"
	"sync"
)

// Confinement: only the owner goroutine ever touches counts. Everyone else sends a message.
type event struct{ word string }

func owner(in <-chan event, done chan<- map[string]int) {
	counts := map[string]int{} // confined: no other goroutine has a reference
	for e := range in {
		counts[e.word]++
	}
	done <- counts // ownership of the map transfers to the receiver; owner never touches it again
}

// The hole: a message that carries a pointer lets sender and receiver share memory.
type buf struct{ data []int }

func main() {
	in := make(chan event, 16)
	done := make(chan map[string]int)
	go owner(in, done)
	var wg sync.WaitGroup
	for w := 0; w < 4; w++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for i := 0; i < 1000; i++ {
				in <- event{"hit"}
			}
		}()
	}
	wg.Wait()
	close(in)
	fmt.Println("confined counter:", (<-done)["hit"])

	// Escaped pointer: sender keeps writing to a buffer it already sent.
	if len(os.Args) > 1 && os.Args[1] == "leak" {
		ch := make(chan *buf, 1)
		b := &buf{data: make([]int, 4)}
		ch <- b
		var wg2 sync.WaitGroup
		wg2.Add(1)
		go func() { defer wg2.Done(); r := <-ch; r.data[0] = 1 }()
		b.data[0] = 2 // sender still writes after sending: shared, unsynchronized
		wg2.Wait()
		fmt.Println("leaky version finished")
	} else {
		ch := make(chan *buf, 1)
		b := &buf{data: make([]int, 4)}
		b.data[0] = 2
		ch <- b
		b = nil // the sender gives up its reference after sending
		var wg2 sync.WaitGroup
		wg2.Add(1)
		go func() { defer wg2.Done(); r := <-ch; r.data[0] = 1; fmt.Println("receiver wrote", r.data[0]) }()
		wg2.Wait()
	}
}

Run in a golang:1.23 container with go run -race confine.go:

confined counter: 4000
receiver wrote 1

Running go run -race confine.go leak takes the other branch, where the sender keeps writing to a buffer after sending its pointer. The race detector reports a data race between two writes, one on the receiver goroutine (line 48) and one on the main goroutine (line 49). The report appeared on each of a dozen re-runs; the address, the goroutine numbering and which of the two writes is listed as the earlier one can differ between runs, because that depends on which goroutine gets there first. The report is written to standard error the moment the detector sees the second unordered access, so it appears before the program's own last print, leaky version finished:

confined counter: 4000
==================
WARNING: DATA RACE
Write at 0x00c000142000 by goroutine 12:
  main.main.func2()
      /w/confine.go:48 +0xac

Previous write at 0x00c000142000 by main goroutine:
  main.main()
      /w/confine.go:49 +0x600

Goroutine 12 (running) created at:
  main.main()
      /w/confine.go:48 +0x5dc
==================
leaky version finished
Found 1 data race(s)
exit status 66

Read it as: the two stacks name the two unordered writes to the same address (r.data[0] = 1 in the receiver, b.data[0] = 2 in main), and exit status 66 is how go run reports that the race detector fired. Here the file is named confine.go and /w is the container's working directory. The channel gave an ordering for what happened before the send, but the sender's later write has no ordering with the receiver's write.

What stays shared even when every buffer is per-thread

The first two items cause wrong results in everyday code, so check them first; the last three are narrower, mostly performance or special contexts.

• Escaped pointers. Sending a pointer, slice or reference shares the memory. Either send a copy, or treat the send as a transfer and drop your own reference, as the non-leaking branch does.
• Static and global state. A function-local variable is confined; a global, a singleton (the one shared instance of a class) or a static is not, no matter which thread's code touches it.
• The allocator. The allocator is the runtime component behind malloc or new that hands out memory. Each thread's buffer comes from a shared heap. Allocators synchronize internally (often with per-thread caches), so this is correct, but heavy allocation from many threads still contends, and freeing memory from a thread other than the one that allocated it can be slower.
• False sharing. The cache line is the block of memory (typically 64 bytes) that cores copy between their caches, and a core must own the whole line to write any byte of it. Two threads' private counters that sit in the same cache line make the cores fight over that line. Results stay correct but throughput collapses. Pad or align per-thread data to separate lines.
• Signal handlers. A signal interrupts a thread and runs on top of it. Confinement does not protect that thread's data from its own handler, and only a small set of async-signal-safe functions (the short list of functions the system documents as safe to call from inside a handler, because each is either reentrant or cannot be interrupted by a signal handler; functions like malloc and printf are not on it, since a handler could interrupt them while they hold an internal lock) may be called inside one.

When you still need synchronization

• At the queue: its enqueue and dequeue must be thread-safe (the runtime's channels and concurrent queues are).
• For read-mostly data many threads share (a configuration object): publish an immutable copy through an atomic reference rather than confining it.
• When one operation spans two owners, such as moving money between two accounts owned by different threads: either have one owner coordinate with a two-step message exchange, or take locks in a fixed order.
• When owners wait for each other: if owner A sends a request to owner B and blocks for the reply while B does the same to A, you have a deadlock built from queues instead of locks.

Trade-offs

Confinement trades locks for latency and for a bottleneck: all work on that state is serial, and an owner that blocks on slow I/O stalls everyone waiting on it. Keep owner handlers short, bound the queue so a flood applies backpressure (senders are forced to wait or are refused when the queue is full) instead of filling memory, and move slow work to other threads that report back by message.

Direct answer

A concurrency bug that vanishes when you add logging or appears only in a release build on one device is almost always a data race (two threads touch the same variable, at least one writes, and nothing orders them), and the symptom depends on things that the observation changes. Three mechanisms: (1) the compiler optimizer may assume a plain variable is not changed by another thread and keep it in a register or hoist a read out of a loop (release builds optimize, debug builds do not); (2) the CPU may reorder memory operations, and different CPU architectures and core counts reorder differently; (3) timing: logging, a debugger or a breakpoint adds locks, system calls and delay, which shifts which interleaving occurs. To reproduce deterministically: use a tool that flags the race without needing the bad timing (ThreadSanitizer, TSan, a compiler-based tool that reports data races at run time), reproduce under the same optimization level and CPU architecture as the failure, force the interleaving with test hooks, and record the failing run if you can.

Why logging makes it disappear (and why the debugger does too)

Logging an error calls into a library that typically takes a lock. Locking and unlocking act as barriers (a barrier forbids the compiler and CPU from moving memory operations across it and makes earlier writes visible to the other threads that take the same lock). So the logging call accidentally supplies the synchronization the program lacks. It also slows one thread by microseconds, which can be enough to avoid the narrow window. A debugger stops threads and runs them one at a time, which removes parallelism.

Compiler hoisting: the flag that never changes

#include <atomic>
#include <chrono>
#include <cstdio>
#include <thread>

#if defined(USE_ATOMIC)
std::atomic<bool> stop{false};
#else
bool stop = false;   // plain flag: a data race, so the compiler may assume nobody else writes it
#endif

long spin_until_stop() {
    long n = 0;
    while (!stop) {
        n++;
#ifdef OPAQUE_CALL
        std::fputc('.', stderr);   // opaque library call: the compiler must assume it might change `stop`
#endif
    }
    return n;
}

int main() {
    long n = 0;
    std::thread worker([&] { n = spin_until_stop(); });
    std::this_thread::sleep_for(std::chrono::milliseconds(200));
    stop = true;
    worker.join();
    std::puts("worker saw the flag and exited");
    (void)n;
}

Compiled with GCC 14 (aarch64 container); each run has a 3 second limit from timeout 3, and the commands are g++ -std=c++20 -O0 -pthread hoist.cpp -o h0 (and -O2, -O2 -DOPAQUE_CALL, -O2 -DUSE_ATOMIC, and -O1 -g -fsanitize=thread for the TSan run):

-O0, plain bool:                 worker saw the flag and exited      (exit status 0)
-O2, plain bool:                 (no output, killed by timeout)      (exit status 124)
-O2, plain bool + fputc in loop: worker saw the flag and exited      (exit status 0)
-O2, std::atomic<bool>:          worker saw the flag and exited      (exit status 0)
TSan, plain bool:                WARNING: ThreadSanitizer: data race

At -O2 the optimizer is allowed to assume no other thread writes stop, so it reads the flag once and compiles the loop into an infinite jump. GCC 14 for aarch64 at -O2 emits this for the worker function (label numbers differ between compiler versions and runs):

spin_until_stop():
    adrp  x0, .LANCHOR0
    ldrb  w0, [x0, #:lo12:.LANCHOR0]   // read `stop` once
    tbnz  x0, 0, .L10                   // if bit 0 is set, go to the exit
.L11:
    b     .L11                          // branch to itself forever
.L10:
    mov   x0, 0
    ret

Reading it: adrp puts the address of the 4 KB page that holds stop in register x0 (a register is one of the CPU's few fast storage slots), and :lo12: is the offset of stop inside that page. ldrb loads one byte (the bool) into w0. tbnz tests bit 0 of the value and jumps to .L10, the exit, only if it is set. If the flag was clear, execution falls into .L11: b .L11, a branch to its own address: no further load, so no way to see the flag change. That is the hoisting: the compiler moved the read out of the loop so it happens once. Adding a call the compiler cannot see through (fputc) forces it to reload the flag, so the "debug print" version works; that is the "disappears when I add logging" effect in its purest form. -O0 keeps the variable in memory because it does no optimization. (-O0, -O1 and -O2 are increasing optimization levels; a release build typically uses a high one, a debug build -O0.)

The same bug exists on phones in Java and Kotlin. A plain field such as boolean stopped read in a loop on another thread may be read once by the JIT compiler and never again, and a release build or a different runtime may hoist where a debug build did not. The fix is a volatile field (@Volatile in Kotlin) or an AtomicBoolean; in Swift, a plain Bool shared across threads is the same kind of data race, and you use a lock, an atomic or an actor.

volatile versus atomic: in C and C++, volatile makes the compiler perform every access, which would fix this one symptom, but it does not make the access atomic and gives no ordering between threads, so it is still a data race (undefined behaviour) and you should use std::atomic. (In Java and Kotlin, volatile does give visibility and ordering through the Java memory model, so do not carry the C++ rule across languages.)

Why one device only: CPU architecture and ordering

A tiny example: one thread runs data = 1; flag = 1; and another runs if (flag == 1) use(data);. x86-64 keeps stores in order so the reader never sees flag == 1 with old data; an Arm core may make flag = 1 visible first, so the reader sees the flag and reads stale data. x86-64 CPUs have a comparatively strong memory model; most ARM CPUs, including phones, are allowed to reorder loads and stores (a weak model), so a missing acquire/release pair that works on a developer's laptop fails on an ARM device. Core count, big.LITTLE (phone chips that combine fast and slow cores) and clock speed change the timing, so one device may reach the bad interleaving and another may not. A release build turns the optimizer on, and the optimizer is what hoists and reorders code.

How to reproduce it deterministically

1. Detect the race instead of waiting for the symptom: build the exact release configuration (-O2 -g) with ThreadSanitizer. TSan reports the conflicting accesses even on a run that does not misbehave, as it did on the flag above.
2. Bisect the variable: first by optimization level (-O0, -O1, -O2; the lowest level that fails points to compiler reordering), then by file (compile suspect files unoptimized one at a time), then by CPU (run the same binary on an ARM device or emulator versus x86), then by commit with git bisect driven by a stress loop that runs the test many times.
3. Force the interleaving: add a test-only hook at the suspected gap that waits on a latch (a one-shot gate that blocks threads until a test opens it) so the test controls which thread goes first. A race that happens once in a million runs then happens every run.
4. Record: on Linux, rr records a failing run and replays it identically, so you debug the real failure instead of a vanished one.
5. Constrain the machine: pin threads to few cores (taskset), or raise thread counts beyond core count, to push toward the production timing.

Pitfalls

Do not conclude the bug is fixed because a run with logging passes. Do not "fix" it with a sleep. Do not add volatile to a C++ shared variable and call it done.