Python Programming Questions

Direct answer

A bounded queue (fixed maxsize) applies backpressure automatically: once it is full, put() blocks (threading) or suspends (await put, asyncio) until a consumer frees a slot, so a fast producer is throttled to the consumers' pace without any extra code. queue.Queue gives you this with an internal threading.Condition; asyncio.Queue gives you the same shape with await instead of blocking waits. The wake-up semantics differ by mechanism: a raw condition variable needs you to choose notify() (wake exactly one waiter) versus notify_all() (wake everyone, most of whom will just recheck and go back to waiting), while asyncio.Queue hides that choice entirely, await put/await get suspend and resume the right coroutine without you managing wake-ups by hand.

Structured elaboration

Threading version: queue.Queue(maxsize=N) is backed by a Lock plus two Conditions internally (not-full, not-empty). put() blocks on the not-full condition when the queue is at capacity; get() blocks on not-empty when it is empty. You do not touch the condition variables yourself, Queue already calls notify() correctly.

asyncio version: a coroutine is a function defined with async def that can pause mid-execution, at an await, and let other coroutines run, then resume later exactly where it left off; the event loop is the scheduler that decides which paused coroutine gets to run next. asyncio.Queue(maxsize=N) has the same contract as the threading version, but await q.put(item) suspends the coroutine (yielding control to the event loop, not blocking an OS thread) when full, and await q.get() suspends when empty. Concurrency comes from running many producer/consumer coroutines as tasks on one event loop instead of many OS threads.

Graceful shutdown, both versions: signal producers to stop (an Event), let running producers finish their current item, then drain the queue (q.join(), which waits until every put item has had a matching task_done()), then push one sentinel value per consumer so each one exits its loop cleanly. q.join() hanging is almost always a sign that a consumer path skipped task_done(), most often via an exception on the processing line before it reached the task_done() call.

The harder variant, without queue.Queue/asyncio.Queue: you build the bounded buffer yourself from a Lock plus two Condition objects sharing that lock, one for "not full", one for "not empty." This is exactly what queue.Queue does internally, made explicit:

import threading, collections

class BoundedBuffer:
    def __init__(self, maxsize):
        self.maxsize = maxsize
        self.items = collections.deque()
        self.lock = threading.Lock()
        self.not_full = threading.Condition(self.lock)
        self.not_empty = threading.Condition(self.lock)

    def put(self, item):
        with self.not_full:
            while len(self.items) >= self.maxsize:
                self.not_full.wait()      # releases the lock, sleeps until notified
            self.items.append(item)
            self.not_empty.notify()       # wake exactly one waiting consumer

    def get(self):
        with self.not_empty:
            while not self.items:
                self.not_empty.wait()
            item = self.items.popleft()
            self.not_full.notify()        # wake exactly one waiting producer
            return item

notify() (one waiter) is correct here because exactly one slot opened up, waking everyone with notify_all() would just cause the other waiters to recheck their while condition and go back to sleep, which is harmless but wasteful (a "thundering herd" of wasted wake-ups). notify_all() earns its keep when a single event can satisfy many different waiters at once, which is not the case for a single-slot state change like this.

Worked example

asyncio.Queue-based pipeline with graceful shutdown, verified on CPython 3.12 (produced and consumed counts always match, since q.join() guarantees every item was drained before sentinels are sent; the exact counts vary run to run because they depend on real-time scheduling):

import asyncio, random

SENTINEL = object()

async def producer(pid, q, stop_event, rng):
    i = 0
    while not stop_event.is_set():
        await q.put((pid, i))          # blocks here once the queue is full
        i += 1
        await asyncio.sleep(rng.random() * 0.02)

async def consumer(cid, q, processed):
    while True:
        item = await q.get()
        if item is SENTINEL:
            q.task_done()
            break
        processed.append((cid, item))
        q.task_done()

async def run_pipeline(n_producers=2, n_consumers=3, maxsize=5, run_time=0.3, seed=7):
    rng = random.Random(seed)
    q = asyncio.Queue(maxsize=maxsize)
    stop_event = asyncio.Event()
    processed = []

    producers = [asyncio.create_task(producer(i, q, stop_event, rng)) for i in range(n_producers)]
    consumers = [asyncio.create_task(consumer(i, q, processed)) for i in range(n_consumers)]

    await asyncio.sleep(run_time)
    stop_event.set()
    await asyncio.gather(*producers)
    await q.join()                          # wait until every produced item is processed
    for _ in consumers:
        await q.put(SENTINEL)
    await asyncio.gather(*consumers)
    assert len(processed) == sum(1 for _ in processed)  # every produced item made it through

asyncio.run(run_pipeline())

A representative run reported "total produced: 62, total consumed: 62"; the invariant produced == consumed holds on every run because q.join() will not return until all items are drained, the specific count itself is not reproducible (it depends on wall-clock scheduling) and is not the claim being made here.

Trade-offs & pitfalls

• Complexity: $O(1)$ per item for queue operations; space is bounded by maxsize for the queue itself, plus $O(k)$ for the $k$ in-flight producer/consumer tasks.
• Edge case: if a consumer raises an exception before calling task_done(), q.join() hangs forever; wrap the processing body in try/finally so task_done() always runs.
• Edge case: sending fewer sentinels than there are consumers leaves some consumers blocked on get() forever; one sentinel per consumer is required, not one total.
• Choosing notify() when the change actually affects multiple waiters (rare for a single bounded queue, more common in custom condition-based coordination) silently starves the other waiters; if you are not certain only one waiter's condition changed, notify_all() is the safer default even though it wakes more coroutines/threads than strictly necessary.
• The asyncio version scales to many more concurrent producers/consumers than the threading version for the same memory budget, since coroutines are far cheaper than OS threads, but it only helps if the "work" inside producer/consumer is itself non-blocking; a CPU-bound consumer inside an asyncio pipeline will stall the whole event loop exactly like any other CPU-bound coroutine.

Direct answer

Define a custom exception when a caller needs to programmatically distinguish and handle a specific failure mode; reuse a built-in (ValueError, TypeError, KeyError) when the failure is a generic, well-understood violation with no library-specific handling to offer. Keep except clauses as narrow as the exception you can actually recover from, never bare except:. Use raise NewError(...) from original whenever you translate a low-level exception into a library-level one, so the original traceback and type are preserved for debugging instead of discarded.

Structured elaboration

When to define a custom exception type

• Define one when a caller might reasonably want to catch this specific failure and do something different for it than for other failures (retry, fall back, surface a specific user-facing message). If every caller would handle it the same way as a generic ValueError, a custom type adds ceremony without adding value.
• Root the hierarchy at a single package-level base (for example MyLibError(Exception)), with specific failures subclassing it (ModelLoadError(MyLibError), InvalidDatasetError(MyLibError)). This lets a caller catch the single base class to mean "anything this library can go wrong in" without having to enumerate every subclass, while still allowing narrower catches where useful.
• Name exceptions for what failed semantically, not for the internal mechanism that detected it; a caller should be able to catch ModelLoadError without knowing or caring whether the implementation currently reads checkpoints from disk, S3, or a database.

How narrow an except clause should be

• Catch the most specific exception type you can actually do something about. A library function should generally let exceptions it cannot meaningfully handle propagate (or wrap them in its own type), rather than catching broadly and hiding the failure.
• Bare except: (or except Exception: used as a catch-all) at the library level is almost always wrong: it catches things like KeyboardInterrupt-adjacent control-flow signals in the case of bare except:, and in the case of except Exception: it hides programming errors (a typo causing an AttributeError) behind the same handling path as an expected, recoverable failure.
• Application-level code (the outermost layer, close to a user or an operator) is where broader catches are more defensible, specifically to provide a fallback, a user-facing error message, or a metric increment, since at that point there is nowhere further up to propagate to.

Exception chaining with raise ... from ...

• raise ModelLoadError(...) from original_exc sets original_exc as the new exception's __cause__, so the traceback shown to a developer includes both: "the following exception occurred while handling this one," preserving the original type, message, and traceback instead of losing them.
• Omitting from (raise ModelLoadError(...) inside an except block) still implicitly chains the original as __context__, shown as "during handling of the above exception, another exception occurred"; explicit from is preferred when the translation is intentional, since it documents the relationship as deliberate rather than incidental. raise ... from None suppresses the chain entirely, which is appropriate only when the original exception is genuinely irrelevant noise (rare in a library boundary).

Worked example

class LibraryError(Exception):
    '''Base class for all errors raised by this library.'''

class ModelLoadError(LibraryError):
    '''Raised when a model checkpoint cannot be loaded.'''

def load_checkpoint(path):
    raise FileNotFoundError(path)

try:
    load_checkpoint("/tmp/does-not-exist.pt")
except FileNotFoundError as e:
    raise ModelLoadError(f"could not load checkpoint at {e}") from e

This raises ModelLoadError: could not load checkpoint at /tmp/does-not-exist.pt, and the traceback CPython prints includes both exceptions, joined by the line "The above exception was the direct cause of the following exception:", with the original FileNotFoundError and its own traceback shown first. A caller who only knows about this library's API can catch ModelLoadError (or the LibraryError base) without needing to know the failure originated from a missing file rather than, say, a corrupted checkpoint format; a developer debugging the failure still sees the full original traceback via the chained __cause__.

Trade-offs & pitfalls

• A hierarchy that is too deep (many single-use subclasses that no caller ever catches individually) adds API surface for no behavioral benefit; a hierarchy that is too flat (one exception type for every failure) forces every caller to parse the message string to distinguish cases, which is fragile and not something the standard library's own conventions encourage.
• Catching broadly "to be safe" inside a library function is the single most common way debugging information gets lost: an unrelated bug (a typo, an off-by-one) gets silently reclassified as the same expected failure the except clause was written for, and the real bug ships unnoticed.
• Changing which exception type a public function raises (or removing a subclass from the hierarchy) is a breaking API change for any caller who catches it specifically; treat the exception hierarchy itself as part of the library's versioned public contract, not as an implementation detail.

Direct answer

The Global Interpreter Lock (GIL, the mutex that lets only one thread execute Python bytecode at a time in CPython) makes a handful of single, self-contained C-level operations effectively atomic: list.append(x), list.pop() (no index), and dict.__setitem__/dict.__delitem__ (plain d[k] = v or del d[k]) each compile to one operation that runs to completion without another thread's bytecode interleaving in the middle. That guarantee stops the instant an operation is compound, meaning it involves more than one read-or-write step under the hood: d[k] = d[k] + 1, d.setdefault-style check-then-act, list.extend from an arbitrary Python iterable, and any "read the current value, then write something derived from it" sequence can interleave between threads and silently lose updates, even though the source line looks like a single statement.

Structured elaboration

Why single ops are safe: the GIL serializes bytecode dispatch, not "one line of Python source." A single bytecode instruction like STORE_SUBSCR (which implements d[k] = v) runs as one atomic unit from the interpreter's point of view: no other thread's bytecode executes in the middle of servicing it. list.append is likewise one C-level call. Because the whole operation is one indivisible step from the interpreter's perspective, no other thread can observe or interfere with it partway through.

Why compound ops are not: counts["hits"] = counts["hits"] + 1 compiles to a short sequence: load the dict, load the key, do a subscript lookup, add one, then store back. The GIL can hand control to a different thread at the boundary between any of those bytecode instructions. Two threads can each load the old value before either writes the incremented result back, and one increment is lost, not because either op was individually unsafe, but because the whole read-modify-write is not one step.

Which common ops fall on which side (CPython-specific, an implementation detail, not a language guarantee):

• Atomic: list.append(x), list.pop() (no args), d[k] = v, del d[k].
• Not atomic: d[k] = d[k] + 1 or any get-then-set, list.extend(iterable) when the iterable is an arbitrary Python object whose iteration can itself run Python bytecode (a generator, a custom __iter__) rather than another built-in list/tuple, list += other, check-then-act patterns like a lazy-initialization if x is None: x = build(), and iterating a container while another thread mutates it.

Worked example

Verified on CPython 3.12. list.append under contention from four threads reaches the exact expected count, confirming atomicity in practice, not just in theory:

import threading

shared_list = []

def appender(n):
    for _ in range(n):
        shared_list.append(1)

threads = [threading.Thread(target=appender, args=(50_000,)) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(len(shared_list))

This prints 200000 (exactly 4 * 50_000) on every run.

The compound dict update loses updates once the read and the write-back are pulled apart by a forced thread switch (time.sleep(0) between them makes the interleaving reliable instead of leaving it to chance, since in a tight loop with no forced yield, thread switches are timer-driven and infrequent enough that the race rarely shows up in a short demo):

import threading
import time

counts = {"hits": 0}

def bump(n):
    for _ in range(n):
        temp = counts["hits"]       # read
        time.sleep(0)               # force an interleave between read and write
        counts["hits"] = temp + 1   # write

threads = [threading.Thread(target=bump, args=(300,)) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(counts["hits"], "expected", 1200)

Three separate runs of this produced 303, 327, and 331 against an expected 1200: reliably and substantially below the expected total every time, though the exact number is not reproducible run to run (it depends on the OS thread scheduler); the lost-update pattern itself is the reproducible claim, not a specific count.

The bytecode confirms the shape of the problem directly:

import dis
def demo(d):
    d["hits"] = d["hits"] + 1
dis.dis(demo)

On CPython 3.12 this shows LOAD_FAST, LOAD_CONST, BINARY_SUBSCR, LOAD_CONST, BINARY_OP, then STORE_SUBSCR, six separate steps, any pair of which can have another thread's bytecode run in between.

Trade-offs & pitfalls

• The safe fix for compound updates is a lock around the whole read-modify-write, not a smarter atomic-looking one-liner: with lock: counts["hits"] += 1 is correct because the lock, not the syntax, defines the atomic region.
• list.extend is a nuance worth getting right rather than blanket-labeling "unsafe": extending from a concrete list or tuple runs as a tight C-level copy loop with no arbitrary Python callback per element, and behaves atomically in practice; extending from a generator or a custom iterable calls back into Python bytecode on every next(), which can be interleaved. The safe assumption is to treat extend as unsafe unless you know the source is a plain built-in sequence, since relying on an implementation detail that varies by argument type is fragile.
• The check-then-act shape shows up far beyond counters: if key not in cache: cache[key] = build() (lazy initialization) races exactly like the dict-increment example, and is one of the most common places this class of bug actually ships in production, since it is rarely exercised under real contention until traffic spikes.
• None of this reasoning is language-level: it is a CPython implementation detail tied to how the GIL schedules bytecode dispatch. Relying on it for correctness is fragile even on CPython (a future bytecode change could alter which single ops stay atomic) and outright wrong for other implementations (CPython is the standard, C-implemented Python interpreter most people mean by "Python"; alternative implementations like PyPy or Jython exist and are free to schedule threads and manage the GIL differently, so the specific atomicity guarantees described here do not carry over to them); write explicit locks for anything you actually depend on being atomic, and treat the "these ops happen to be safe" list as an optimization fact, not an API contract.
• For CPU-bound work where lock contention on a shared counter becomes the bottleneck, prefer multiprocessing (which sidesteps the GIL and shared-state races entirely by giving each worker its own memory) or have each thread accumulate a private total and merge once at the end, rather than acquiring a lock on every single increment.

Direct answer

Python has four argument-passing shapes: positional (matched by order), keyword (matched by name), default values (make a parameter optional), and the catch-alls *args (extra positionals, collected as a tuple) and **kwargs (extra keywords, collected as a dict). They combine in one fixed signature order: positional-or-keyword params, then *args, then keyword-only params (which may have defaults), then **kwargs.

Structured elaboration

Signature order and what each piece means:

def process_batch(data, transform, *args, normalize=True, batch_size=100, **kwargs):
    ...

• data, transform: positional-or-keyword, required. Callers usually pass these positionally because order is natural (the data, then what to do to it).
• *args: soaks up any extra positional arguments beyond data and transform, exposed inside the function as a tuple. Anything named after a bare *args (or a bare *) becomes keyword-only, meaning it cannot be passed positionally.
• normalize, batch_size: keyword-only parameters with defaults, so they are optional and callers must name them.
• **kwargs: soaks up any remaining keyword arguments not already named in the signature, exposed as a dict.

When each pattern is idiomatic versus overkill:

• Positional: idiomatic for the 1-2 arguments every call needs, in an order a reader can memorize. Overkill once a function takes more than 3-4 positional arguments; callers start passing things in the wrong order silently.
• Keyword + default: idiomatic for tunable, optional behavior (batch_size, normalize) where the default covers the common case and the name documents intent at the call site.
• *args: idiomatic for genuinely variadic operations (print(*values), a sum_all(*numbers)) or for forwarding positional arguments through a wrapper. Overkill as a substitute for a real, named parameter list; it hides the function's actual contract from anyone reading a call site.
• **kwargs: idiomatic for forwarding options to an underlying API you do not want to re-declare (backend-specific keys like shuffle or device), or for a plugin-style function whose exact options vary by caller. Overkill as a way to avoid deciding what a function's parameters actually are; unvalidated **kwargs swallows typos (nomralize=True) silently instead of raising TypeError.

Worked example

def process_batch(data, transform, *args, normalize=True, batch_size=100, **kwargs):
    results = []
    for item in data:
        value = transform(item, *args)
        if normalize:
            value = value / batch_size
        results.append(value)
    return results, kwargs

out, extra = process_batch(
    [10, 20, 30],
    lambda x: x * 2,
    normalize=True,
    batch_size=10,
    shuffle=True,
    device="cpu",
)
print(out)    # [2.0, 4.0, 6.0]
print(extra)  # {'shuffle': True, 'device': 'cpu'}

Here data and transform are passed positionally because every call needs them and the order is obvious; normalize and batch_size are named because they are optional tuning knobs; shuffle and device are not declared parameters at all, they arrive through **kwargs and the function forwards or ignores them as extra.

Trade-offs & pitfalls

• A bare * with no name (def f(a, *, b):) is legal and forces everything after it to be keyword-only without collecting extra positionals; use it to make a signature self-documenting even when you do not need *args.
• **kwargs that is never validated is a common production bug: transform_data(dat=my_df) (a typo of data=) silently lands in kwargs instead of raising TypeError: missing required argument. Pop and check known keys explicitly (kwargs.pop("shuffle", False)) if you accept **kwargs, or avoid it and declare the real parameter list.
• Order matters and is enforced by the interpreter, not just style: positional-or-keyword parameters must come before *args, which must come before keyword-only parameters, which must come before **kwargs; writing them out of order is a SyntaxError at definition time.
• Mutable defaults are the sharpest edge here (def f(items=[]):): a default value is evaluated once, when the def statement runs, not on each call, so a mutable default is shared and can accumulate state across unrelated calls.

Direct answer

F-strings (Python 3.6+) are the best default: they are the most readable because the expression sits inline at the point of use, and the fastest because the interpreter compiles the expression directly into the code rather than parsing a template string at run time. %-formatting is the oldest and leanest for simple positional substitution. str.format() is the most flexible for templates that are built or reused separately from the values (and the most dangerous if that template can come from untrusted input), but pays a parsing and lookup cost that makes it the slowest of the three in most cases.

Structured elaboration

Readability

• F-strings: the expression and its formatting spec live together, e.g. f"loss={loss:.4f}"; nothing to visually match up against a separate argument list.
• str.format(): clear for reusable templates ("{name} scored {score}".format(...)) but requires matching placeholders to arguments, which gets harder to scan as the template grows.
• %-formatting: compact for one or two values, but easy to get positional order wrong and the %s/%d type markers are an extra thing to keep in sync with the actual argument types.

Performance
F-strings are generally fastest because the expressions inside {} are compiled straight into bytecode at compile time, no template string is parsed at call time. %-formatting is close behind for simple cases since it is a single lightweight C-level operation. str.format() tends to be the slowest of the three because it parses the template string and does attribute/index lookups on every call. This ordering is well documented CPython behavior; treat it as a rule of thumb, not a promise for every input shape; measure with timeit on your actual code if a specific format call is on a hot path, rather than assuming the ranking transfers exactly.

Security: format-string injection
The risk is specifically about who controls the template string, not the values being substituted:

• With %-formatting and str.format(), if the template itself (not just the values) comes from untrusted input, the attacker controls what gets read. str.format() is the sharper edge here because its placeholder syntax can traverse attributes and indexes on any object you pass in: feeding the template "leaked={0.secret}" through .format(some_object) reads some_object.secret, an attribute that was never meant to be exposed by the calling code. %-formatting has no attribute/index traversal, so its blast radius is smaller but not zero (an attacker-controlled format spec can still cause exceptions or unexpected coercions).
• F-strings do not have this specific injection shape, because the expression is fixed in the source code by whoever wrote the code, not supplied as a separate runtime string. The residual risk with f-strings is indirect: if you ever build an f-string-like template dynamically and eval() it, you have reintroduced the exact same problem by hand.
• The universal fix: never format a template that comes from outside your code's control, regardless of which of the three you use, and prefer structured, escaping-aware serializers (json.dumps, logging's own %-style deferred formatting) for anything that touches untrusted data or gets machine-parsed downstream.

Worked example

class Config:
    def __init__(self):
        self.secret = "top-secret-value"

cfg = Config()

print("value=%s" % (cfg.secret,))          # value=top-secret-value

print("value={}".format(cfg.secret))       # value=top-secret-value (safe: fixed template)

untrusted_template = "leaked={0.secret}"
print(untrusted_template.format(cfg))      # leaked=top-secret-value (unsafe: template is data)

name = "world"
print(f"hello {name}")                     # hello world

The third line is the injection risk made concrete: if untrusted_template came from a config file, a URL parameter, or any place an attacker can reach, .format(cfg) reads cfg.secret even though nothing in the calling code explicitly asked for it.

Trade-offs & pitfalls

• Deferred formatting for logging. For log calls, prefer logger.debug("loss=%.4f step=%d", loss, step) (parameterized, %-style) over an f-string built eagerly, logger.debug(f"loss={loss:.4f}"): the f-string always pays the formatting cost even when the debug level is disabled, while the parameterized call only formats if the message is actually going to be emitted.
• Locale and precision control matter for % and str.format() numeric specs the same way they do for f-strings (:.4f works identically in an f-string and in str.format()); this is a formatting-spec detail shared across all three, not a differentiator between them.
• Common wrong turn: treating "f-strings are fastest and most readable" as "always safe to use directly on untrusted data." F-strings sidestep the template-injection shape described above, but they do nothing to sanitize the values you interpolate; a value containing control characters or extremely long content can still cause downstream problems (e.g. log injection, oversized output) regardless of which formatting style produced it.