Python Programming Questions

Approach summary
Use an atomic filesystem operation to create a lockfile and store owner PID. "Atomic" here means the operating system guarantees that checking whether the file already exists and creating it happen as a single, indivisible step: the kernel resolves any race internally, so it is impossible for two processes to both be told "it didn't exist, you just created it" for the same file. Concretely, os.open(path, O_CREAT|O_EXCL) either creates the file and returns a valid file descriptor (this process is now the owner), or, if the file already exists, raises OSError with errno.EEXIST and creates nothing, there is no window in between where a second process could sneak in and also succeed. On Unix use os.open with O_EXCL|O_CREAT; on Windows use msvcrt.locking or CreateFile with exclusive flags. Implement acquire(timeout) with retries and stale-lock detection via PID and age.

Implementation (Unix-first, cross-platform fallback)

import os, time, errno
from pathlib import Path

class FileLock:
    def __init__(self, path):
        self.path = Path(path)
    def acquire(self, timeout=10):
        end = time.time()+timeout
        while time.time()<end:
            try:
                fd = os.open(self.path, os.O_CREAT|os.O_EXCL|os.O_WRONLY)
                os.write(fd, str(os.getpid()).encode())
                os.close(fd)
                return True
            except OSError as e:
                if e.errno!=errno.EEXIST: raise
            time.sleep(0.1)
        return False
    def release(self):
        try: self.path.unlink()
        except FileNotFoundError: pass

Worked trace: two processes racing to acquire the same lock, simulated in one process for illustration (the real guarantee comes from the OS's atomic O_CREAT|O_EXCL, not from anything special in this simulation, but the sequence of return values is exactly what would happen with two real, separate processes):

lock_a = FileLock("/tmp/demo.lock")
lock_b = FileLock("/tmp/demo.lock")

print(lock_a.acquire(timeout=1))   # True: lock_a's os.open call wins the race, file now exists
print(lock_b.acquire(timeout=0.3)) # False: lock_b's os.open calls all hit EEXIST and it
                                    # gives up once the 0.3s timeout elapses
lock_a.release()
print(lock_b.acquire(timeout=1))   # True: now that lock_a released (deleted the file),
                                    # lock_b's next os.open call succeeds

The first acquire call's os.open(..., O_CREAT|O_EXCL) either wins outright (file did not exist, now it does, True) or loses outright (EEXIST, False after retries time out); there is no third outcome where both processes believe they created the file, which is exactly the race the atomic system call rules out by construction.

Platform notes

• Unix: O_EXCL is atomic across processes. Use fcntl.flock for advisory locks (a lock that only stops other processes if they also choose to check for it before touching the file; unlike a mandatory, OS-enforced lock, the OS itself does not block a process that simply ignores the lock and opens the file directly) when needed.
• Windows: O_EXCL behaves differently; prefer msvcrt.locking or pywin32 CreateFile for exclusive access.

Stale locks
Check PID from file and whether process exists; remove stale if safe. For production, add jitter, robust error handling, and optional directory-level locking for network filesystems.

Direct answer

Give the decorator function an optional first positional parameter (conventionally func=None) alongside keyword-only configuration parameters. When Python evaluates @retry with no parentheses, it calls retry(the_function) directly, so func arrives as the function itself. When Python evaluates @retry(max_attempts=5), it first calls retry(max_attempts=5) with func left at its default of None, and that call must return the actual decorator, which Python then applies to the function on the next line. The whole trick is: check whether func is None and either apply the wrapping immediately or hand back a configured decorator to be applied a moment later.

Structured elaboration

What makes this genuinely ambiguous is that @retry and @retry(max_attempts=5) invoke the same name in two structurally different ways, and Python gives you no explicit signal about which form was used; the only observable difference is what got passed as the first argument:

• No-parens form (@retry): Python evaluates retry as an expression, then immediately calls it with the decorated function as the sole argument: retry(some_function). Here, func is bound to a real callable.
• Parens form (@retry(max_attempts=5)): Python first calls retry(max_attempts=5) to produce a value, then applies that value as the decorator to the function on the next line, calling that value with the function. Here, func is not supplied at all (or would only be supplied as None by default), and the whole first call must return something else that is itself callable and ready to receive the function.

So the detection is: is the value bound to func a callable function (no-parens case), or is it the default None (parens case, meaning we're still waiting for both the config and the function)? Concretely:

• If func is None: this is the parenthesized call. Return the inner decorator function, which Python will apply to the actual function next.
• If func is not None: this is the no-parens call, func already is the function to wrap. Apply decorator(func) immediately and return the wrapped result.

functools.wraps is orthogonal to the ambiguity itself but is required either way, without it, the wrapped function's __name__, __doc__, and other metadata would silently become the wrapper's instead of the original function's, which breaks introspection, debugging, and tools that key off __name__.

Worked example

import functools
from typing import Callable, Optional

def retry(func: Optional[Callable] = None, *, max_attempts: int = 3):
    def decorator(fn: Callable):
        @functools.wraps(fn)
        def wrapped(*args, **kwargs):
            attempts = 0
            while True:
                try:
                    return fn(*args, **kwargs)
                except Exception:
                    attempts += 1
                    if attempts >= max_attempts:
                        raise
        return wrapped

    if func is None:
        return decorator      # parens form: still need the function
    return decorator(func)    # no-parens form: func is already the function

calls = {"a": 0, "b": 0}

@retry
def flaky_a():
    calls["a"] += 1
    if calls["a"] < 2:
        raise ValueError("boom")
    return "a-ok"

@retry(max_attempts=5)
def flaky_b():
    calls["b"] += 1
    if calls["b"] < 4:
        raise ValueError("boom")
    return "b-ok"

print(flaky_a(), calls["a"])   # a-ok 2
print(flaky_b(), calls["b"])   # b-ok 4
print(flaky_a.__name__)        # flaky_a  (functools.wraps preserved the name)

flaky_a succeeds on its second internal call (default max_attempts=3 is enough); flaky_b needed 4 internal calls, only reachable because max_attempts=5 was configured through the parens form.

Trade-offs & pitfalls

• The keyword-only * before max_attempts is load-bearing, not stylistic. Without it, retry(5) (a bare positional call) would ambiguously try to bind 5 to func, which is nonsensical; keyword-only parameters force all configuration to be named, so the only way func gets a non-None value is through the actual no-parens decoration path.
• This pattern does not extend to async def functions for free. The wrapped function here calls fn(*args, **kwargs) synchronously; wrapping an async def with this decorator would return a coroutine object without awaiting it, silently swallowing the retry logic. An async-aware version needs an async def wrapped that awaits fn(...), and ideally detection of which kind of function it's wrapping (inspect.iscoroutinefunction).
• Retrying on bare Exception is intentionally broad here for brevity but is a common wrong turn in production code: it will retry on bugs (a TypeError from a coding mistake) exactly as eagerly as on transient failures (a network timeout), masking real errors behind a few extra silent attempts. A production version should accept an explicit tuple of exception types to retry on.
• No backoff between attempts means a fast-failing dependency gets hammered immediately on every retry; production retry decorators almost always add a delay (ideally with jitter) between attempts.

Direct answer

with expr as name: calls expr.__enter__() to acquire a resource and binds its return value to name, then guarantees expr.__exit__(exc_type, exc, tb) runs when the block ends, whether it ended normally or via an exception. That guarantee, not the syntax, is why context managers matter: cleanup code (closing a file, releasing a lock, rolling back a transaction) runs even when the code between acquisition and release raises, without the caller having to remember a try/finally.

Structured elaboration

What __enter__ and __exit__ are responsible for

• __enter__(self) acquires the resource and returns whatever the as clause should bind (often self, sometimes the underlying handle).
• __exit__(self, exc_type, exc, tb) always runs on the way out of the block. If the block raised, exc_type/exc/tb describe that exception; if it exited normally, all three are None.
• The return value of __exit__ controls exception suppression: returning a truthy value swallows the exception (it will not propagate further); returning False or None (the default if you don't return anything) lets it propagate. Suppressing exceptions silently is rarely what you want, so __exit__ should return False unless there is a specific, documented reason to swallow a particular exception type.

Why this beats manual try/finally

• A bare try:/finally: works too, but the context manager packages "how to acquire" and "how to release" into one reusable object, so callers cannot forget the finally and the acquire/release pairing is enforced by the protocol rather than by convention.

Worked example

A minimal custom context manager for a resource that must always be released, even if the block raises:

class ManagedResource:
    '''Stand-in for any resource that must be released: a file, socket,
    lock, or connection. __enter__ acquires it, __exit__ always releases it.
    '''
    def __init__(self, name):
        self.name = name
        self.closed = False

    def __enter__(self):
        print(f"acquired {self.name}")
        return self

    def __exit__(self, exc_type, exc, tb):
        self.closed = True
        print(f"released {self.name} (exc_type={exc_type})")
        return False  # never suppress the exception

r = ManagedResource("conn-1")
try:
    with r as res:
        print("using", res.name)
        raise ValueError("boom")
except ValueError as e:
    print("caught:", e)
print("closed?", r.closed)

Running this prints, in order:

acquired conn-1
using conn-1
released conn-1 (exc_type=<class 'ValueError'>)
caught: boom
closed? True

__exit__ runs and sets closed = True before the exception is allowed to propagate out of the with block; the caller's try/except then catches it exactly as if the with statement were not there at all, because __exit__ returned False.

Trade-offs & pitfalls

• Returning a truthy value from __exit__ to suppress an exception is occasionally the right call (the standard library's contextlib.suppress does exactly this), but doing it without saying so in the class's own name or docstring is a common source of silently-swallowed bugs: an exception vanishes with no trace of why.
• For a simple case like this, contextlib.contextmanager (a generator wrapped with a try/finally) is usually less boilerplate than a full class with __enter__/__exit__; reach for the class form when the object needs to expose state or methods beyond the with block itself (as ManagedResource does here by keeping closed inspectable afterward).
• A context manager only guarantees cleanup for the resource it explicitly manages; if __enter__ itself partially acquires state before raising, __exit__ is never called (the with statement only calls __exit__ once __enter__ has returned successfully), so __enter__ needs to clean up after itself on its own failure paths.

Direct answer

The Global Interpreter Lock (GIL, the mutex that lets only one thread execute Python bytecode at a time in CPython) makes a handful of single, self-contained C-level operations effectively atomic: list.append(x), list.pop() (no index), and dict.__setitem__/dict.__delitem__ (plain d[k] = v or del d[k]) each compile to one operation that runs to completion without another thread's bytecode interleaving in the middle. That guarantee stops the instant an operation is compound, meaning it involves more than one read-or-write step under the hood: d[k] = d[k] + 1, d.setdefault-style check-then-act, list.extend from an arbitrary Python iterable, and any "read the current value, then write something derived from it" sequence can interleave between threads and silently lose updates, even though the source line looks like a single statement.

Structured elaboration

Why single ops are safe: the GIL serializes bytecode dispatch, not "one line of Python source." A single bytecode instruction like STORE_SUBSCR (which implements d[k] = v) runs as one atomic unit from the interpreter's point of view: no other thread's bytecode executes in the middle of servicing it. list.append is likewise one C-level call. Because the whole operation is one indivisible step from the interpreter's perspective, no other thread can observe or interfere with it partway through.

Why compound ops are not: counts["hits"] = counts["hits"] + 1 compiles to a short sequence: load the dict, load the key, do a subscript lookup, add one, then store back. The GIL can hand control to a different thread at the boundary between any of those bytecode instructions. Two threads can each load the old value before either writes the incremented result back, and one increment is lost, not because either op was individually unsafe, but because the whole read-modify-write is not one step.

Which common ops fall on which side (CPython-specific, an implementation detail, not a language guarantee):

• Atomic: list.append(x), list.pop() (no args), d[k] = v, del d[k].
• Not atomic: d[k] = d[k] + 1 or any get-then-set, list.extend(iterable) when the iterable is an arbitrary Python object whose iteration can itself run Python bytecode (a generator, a custom __iter__) rather than another built-in list/tuple, list += other, check-then-act patterns like a lazy-initialization if x is None: x = build(), and iterating a container while another thread mutates it.

Worked example

Verified on CPython 3.12. list.append under contention from four threads reaches the exact expected count, confirming atomicity in practice, not just in theory:

import threading

shared_list = []

def appender(n):
    for _ in range(n):
        shared_list.append(1)

threads = [threading.Thread(target=appender, args=(50_000,)) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(len(shared_list))

This prints 200000 (exactly 4 * 50_000) on every run.

The compound dict update loses updates once the read and the write-back are pulled apart by a forced thread switch (time.sleep(0) between them makes the interleaving reliable instead of leaving it to chance, since in a tight loop with no forced yield, thread switches are timer-driven and infrequent enough that the race rarely shows up in a short demo):

import threading
import time

counts = {"hits": 0}

def bump(n):
    for _ in range(n):
        temp = counts["hits"]       # read
        time.sleep(0)               # force an interleave between read and write
        counts["hits"] = temp + 1   # write

threads = [threading.Thread(target=bump, args=(300,)) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(counts["hits"], "expected", 1200)

Three separate runs of this produced 303, 327, and 331 against an expected 1200: reliably and substantially below the expected total every time, though the exact number is not reproducible run to run (it depends on the OS thread scheduler); the lost-update pattern itself is the reproducible claim, not a specific count.

The bytecode confirms the shape of the problem directly:

import dis
def demo(d):
    d["hits"] = d["hits"] + 1
dis.dis(demo)

On CPython 3.12 this shows LOAD_FAST, LOAD_CONST, BINARY_SUBSCR, LOAD_CONST, BINARY_OP, then STORE_SUBSCR, six separate steps, any pair of which can have another thread's bytecode run in between.

Trade-offs & pitfalls

• The safe fix for compound updates is a lock around the whole read-modify-write, not a smarter atomic-looking one-liner: with lock: counts["hits"] += 1 is correct because the lock, not the syntax, defines the atomic region.
• list.extend is a nuance worth getting right rather than blanket-labeling "unsafe": extending from a concrete list or tuple runs as a tight C-level copy loop with no arbitrary Python callback per element, and behaves atomically in practice; extending from a generator or a custom iterable calls back into Python bytecode on every next(), which can be interleaved. The safe assumption is to treat extend as unsafe unless you know the source is a plain built-in sequence, since relying on an implementation detail that varies by argument type is fragile.
• The check-then-act shape shows up far beyond counters: if key not in cache: cache[key] = build() (lazy initialization) races exactly like the dict-increment example, and is one of the most common places this class of bug actually ships in production, since it is rarely exercised under real contention until traffic spikes.
• None of this reasoning is language-level: it is a CPython implementation detail tied to how the GIL schedules bytecode dispatch. Relying on it for correctness is fragile even on CPython (a future bytecode change could alter which single ops stay atomic) and outright wrong for other implementations (CPython is the standard, C-implemented Python interpreter most people mean by "Python"; alternative implementations like PyPy or Jython exist and are free to schedule threads and manage the GIL differently, so the specific atomicity guarantees described here do not carry over to them); write explicit locks for anything you actually depend on being atomic, and treat the "these ops happen to be safe" list as an optimization fact, not an API contract.
• For CPU-bound work where lock contention on a shared counter becomes the bottleneck, prefer multiprocessing (which sidesteps the GIL and shared-state races entirely by giving each worker its own memory) or have each thread accumulate a private total and merge once at the end, rather than acquiring a lock on every single increment.

Direct answer

The Global Interpreter Lock (GIL) guarantees that only one thread executes Python bytecode at a time, but it does not make count += 1 a single, uninterruptible step. That statement compiles to several bytecode operations (load the current value, add one, store the new value), and the GIL can hand control to another thread in between any two of them. Two threads can both read the same old value before either writes back the incremented one, so an increment gets lost. The fix is to protect the whole read-modify-write with a lock (or have each thread accumulate locally and merge once), not to assume the GIL alone provides atomicity.

Structured elaboration

Why the GIL doesn't save you: the GIL prevents two threads from running Python bytecode simultaneously, but threads still interleave, control passes back and forth between them constantly. count += 1 is not one bytecode instruction; it is a small sequence, and the interpreter can switch threads between any of those instructions. If thread A reads count as 5, then control passes to thread B, which also reads 5, increments to 6, and writes back, then control returns to thread A, which still has 5 in hand, increments to 6, and writes 6 again, one increment from B is silently lost.

What's actually safe to share without a lock, and what isn't:

• Safe: a single, atomic operation on an immutable value, for example rebinding a name to point at a brand-new object (x = new_value) is effectively atomic at the bytecode level because CPython's reference-count updates (CPython keeps a running count, on every object, of how many names or containers currently point to it, and updates that count whenever a name is bound or rebound; that single count update is what happens atomically here) for the old and new objects happen under the GIL without another thread's bytecode interleaving inside that single STORE step.
• Not safe: any compound operation, no matter how short it looks in source: count += 1, d[k] = d.get(k, 0) + 1, list.append followed later by reading len(list) to decide something, if not cache: cache = build() (check-then-act). All of these are read-then-write (or read-then-branch-then-write) and can interleave.
• The rule of thumb: if the line reads a shared value and then writes back something derived from what it read, it needs a lock, regardless of how atomic-looking the syntax is.

This is the same defect wearing three different costumes, and the fix is identical each time (protect the read-modify-write, or the check-then-act, with a lock):

• Shared counter (this question): count += 1 from multiple threads.
• Shared dict: d[k] = d.get(k, 0) + 1 is exactly as compound as the counter case; two threads can race on the same key and lose an update the same way.
• Lazy singleton: if instance is None: instance = build() is a check-then-act; two threads can both see None, both call build(), and end up with two different instances (or worse, two objects doing conflicting side effects during construction) unless the check and the assignment are both inside the same lock.

Worked example

The race, made reliably observable (verified on CPython 3.12). A natural count += 1 loop rarely shows lost updates in a short demo, because the read and the write-back usually land close enough together that no thread switch happens to fall between them, forcing a time.sleep(0) between the read and the write makes the interleaving deterministic instead of leaving it to chance:

import threading, time

count = 0

def worker(n):
    global count
    for _ in range(n):
        temp = count       # read
        time.sleep(0)      # forces a thread switch between read and write-back
        count = temp + 1   # write

threads = [threading.Thread(target=worker, args=(200,)) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(count, "expected", 800)

Three separate runs of this produced 225, 270, and 238 against an expected 800: the exact number is not reproducible run to run (it depends on the OS thread scheduler), but it is reliably and substantially below 800 every time, which is the point being demonstrated: lost updates, not a specific count.

Corrected, efficient version: each thread accumulates a local total (no shared state, no lock needed for that part) and merges into the shared counter once at the end, so lock contention happens once per thread instead of once per increment:

import threading

count = 0
count_lock = threading.Lock()

def worker(n):
    global count
    local = 0
    for _ in range(n):
        local += 1              # purely local, no synchronization needed
    with count_lock:
        count += local          # one synchronized update per thread

threads = [threading.Thread(target=worker, args=(200_000,)) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(count)   # 800000, exactly, on every run

This produced exactly 800000 across repeated runs, matching 4 * 200_000 precisely, because there is now only one lock-protected read-modify-write per thread instead of one per increment.

The same lesson applied to a shared dict and a lazy singleton, both verified on CPython 3.12:

import threading

# Shared dict: same compound read-modify-write as the counter.
counts_lock = threading.Lock()
counts = {}

def bump(key, n):
    for _ in range(n):
        with counts_lock:
            counts[key] = counts.get(key, 0) + 1

threads = [threading.Thread(target=bump, args=("hits", 50_000)) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(counts["hits"])   # 200000, exactly

# Lazy singleton: same check-then-act race, fixed with double-checked locking.
class Config:
    _instance = None
    _lock = threading.Lock()

    @classmethod
    def get_instance(cls):
        if cls._instance is None:            # fast path, no lock in the common case
            with cls._lock:
                if cls._instance is None:    # re-check: another thread may have
                    cls._instance = cls()    # already built it while we waited
        return cls._instance

Trade-offs & pitfalls

• Complexity: the corrected counter does $O(n)$ work per thread with $O(1)$ lock acquisitions per thread (not per increment), versus $O(n)$ lock acquisitions in a naive "lock every increment" fix; batching the merge is what keeps contention low.
• A naive fix that locks every single increment is correct but pays full lock overhead $n$ times per thread; that overhead, not correctness, is the usual reason it gets flagged as inefficient.
• The single-check if instance is None: instance = build() singleton pattern looks obviously wrong once you frame it as check-then-act, but it is one of the most common places this bug actually ships, because it is rarely exercised under real contention until traffic spikes.
• Common wrong turn: reaching for collections.Counter or a plain dict and assuming built-in types are internally synchronized against interleaved compound updates from multiple threads; they are not, their individual C-level operations may be atomic, but get-then-set across two separate operations is not.
• Edge case: the demo above uses an inserted time.sleep(0) to force the race to show up reliably; without it, whether a given run of an unprotected count += 1 loop actually shows a wrong answer depends on thread count, iteration count, and the OS scheduler, so an unprotected version passing a quick manual test is not evidence it is safe.