Python Programming Questions

Direct answer

Five standard-library modules that come up constantly in data-oriented Python work: csv for reading and writing delimited files without loading everything into memory at once, gzip for transparent compression on data that moves through a pipeline, itertools for building memory-efficient iteration patterns (chunking, windowing), collections for purpose-built containers (Counter, defaultdict, deque) that replace hand-rolled counting and buffering logic, and datetime for parsing and normalizing timestamps consistently.

Structured elaboration

• csv: streams rows from a delimited file one at a time via an iterator, so a multi-gigabyte export never has to sit fully in memory just to be transformed and reloaded elsewhere.
• gzip: reads and writes .gz-compressed data directly, useful for ingesting compressed log archives or writing compact intermediate files without shelling out to an external tool.
• itertools: composes lazy, memory-efficient iteration pipelines; islice is the workhorse for turning any iterable into fixed-size chunks without building an intermediate list of everything first.
• collections: Counter replaces a hand-rolled dict frequency table, defaultdict removes repeated if key not in d checks, and a bounded deque gives an O(1) fixed-size rolling buffer instead of manually trimming a list.
• datetime: normalizes timestamps to a consistent, timezone-aware representation, which matters the moment data arrives from more than one source with different (or missing) timezone conventions.

Worked example

Each, verified on CPython 3.12:

import csv, io
buf = io.StringIO("id,value\n1,10\n2,20\n")
rows = list(csv.DictReader(buf))
print(rows)   # [{'id': '1', 'value': '10'}, {'id': '2', 'value': '20'}]

import gzip
payload = b"repeat this line\n" * 100
compressed = gzip.compress(payload)
assert gzip.decompress(compressed) == payload
print(len(payload), len(compressed))   # 1700 52

import itertools
def chunked(iterable, n):
    it = iter(iterable)
    while chunk := list(itertools.islice(it, n)):
        yield chunk
print(list(chunked(range(10), 3)))   # [[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]

from collections import Counter, defaultdict, deque
print(Counter("mississippi"))          # Counter({'i': 4, 's': 4, 'p': 2, 'm': 1})
d = defaultdict(list); d["a"].append(1)
print(dict(d))                          # {'a': [1]}
dq = deque(maxlen=3)
for x in range(5): dq.append(x)
print(list(dq))                         # [2, 3, 4]

from datetime import datetime, timezone
dt = datetime(2026, 7, 21, 12, 0, 0, tzinfo=timezone.utc)
print(dt.isoformat())   # 2026-07-21T12:00:00+00:00

Trade-offs & pitfalls

• csv.DictReader is convenient but re-creates a dict per row; for very large files where the per-row dict overhead itself matters, csv.reader (yielding plain lists) is cheaper, at the cost of accessing fields by position instead of name.
• gzip.compress/decompress operate on the whole payload in memory; for files too large to hold fully in RAM, use gzip.open and read/write in a streaming loop instead of the one-shot compress/decompress functions shown above.
• itertools functions are lazy by design, which is exactly the point for memory efficiency, but that also means a generator built from them is single-pass: iterating it a second time yields nothing, which surprises anyone expecting list-like reuse.
• defaultdict's automatic key creation is a footgun the moment you need to distinguish "key was never touched" from "key exists with the default value"; a plain dict.setdefault or an explicit in check is safer when that distinction matters.
• datetime objects without a tzinfo are "naive" and silently compare or arithmetic-combine with other naive datetimes without any timezone reconciliation; mixing naive and timezone-aware datetimes in the same pipeline raises TypeError at comparison time, which is usually the first sign a source is inconsistent about timezone handling.

Direct answer

The Global Interpreter Lock (GIL) guarantees that only one thread executes Python bytecode at a time, but it does not make count += 1 a single, uninterruptible step. That statement compiles to several bytecode operations (load the current value, add one, store the new value), and the GIL can hand control to another thread in between any two of them. Two threads can both read the same old value before either writes back the incremented one, so an increment gets lost. The fix is to protect the whole read-modify-write with a lock (or have each thread accumulate locally and merge once), not to assume the GIL alone provides atomicity.

Structured elaboration

Why the GIL doesn't save you: the GIL prevents two threads from running Python bytecode simultaneously, but threads still interleave, control passes back and forth between them constantly. count += 1 is not one bytecode instruction; it is a small sequence, and the interpreter can switch threads between any of those instructions. If thread A reads count as 5, then control passes to thread B, which also reads 5, increments to 6, and writes back, then control returns to thread A, which still has 5 in hand, increments to 6, and writes 6 again, one increment from B is silently lost.

What's actually safe to share without a lock, and what isn't:

• Safe: a single, atomic operation on an immutable value, for example rebinding a name to point at a brand-new object (x = new_value) is effectively atomic at the bytecode level because CPython's reference-count updates (CPython keeps a running count, on every object, of how many names or containers currently point to it, and updates that count whenever a name is bound or rebound; that single count update is what happens atomically here) for the old and new objects happen under the GIL without another thread's bytecode interleaving inside that single STORE step.
• Not safe: any compound operation, no matter how short it looks in source: count += 1, d[k] = d.get(k, 0) + 1, list.append followed later by reading len(list) to decide something, if not cache: cache = build() (check-then-act). All of these are read-then-write (or read-then-branch-then-write) and can interleave.
• The rule of thumb: if the line reads a shared value and then writes back something derived from what it read, it needs a lock, regardless of how atomic-looking the syntax is.

This is the same defect wearing three different costumes, and the fix is identical each time (protect the read-modify-write, or the check-then-act, with a lock):

• Shared counter (this question): count += 1 from multiple threads.
• Shared dict: d[k] = d.get(k, 0) + 1 is exactly as compound as the counter case; two threads can race on the same key and lose an update the same way.
• Lazy singleton: if instance is None: instance = build() is a check-then-act; two threads can both see None, both call build(), and end up with two different instances (or worse, two objects doing conflicting side effects during construction) unless the check and the assignment are both inside the same lock.

Worked example

The race, made reliably observable (verified on CPython 3.12). A natural count += 1 loop rarely shows lost updates in a short demo, because the read and the write-back usually land close enough together that no thread switch happens to fall between them, forcing a time.sleep(0) between the read and the write makes the interleaving deterministic instead of leaving it to chance:

import threading, time

count = 0

def worker(n):
    global count
    for _ in range(n):
        temp = count       # read
        time.sleep(0)      # forces a thread switch between read and write-back
        count = temp + 1   # write

threads = [threading.Thread(target=worker, args=(200,)) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(count, "expected", 800)

Three separate runs of this produced 225, 270, and 238 against an expected 800: the exact number is not reproducible run to run (it depends on the OS thread scheduler), but it is reliably and substantially below 800 every time, which is the point being demonstrated: lost updates, not a specific count.

Corrected, efficient version: each thread accumulates a local total (no shared state, no lock needed for that part) and merges into the shared counter once at the end, so lock contention happens once per thread instead of once per increment:

import threading

count = 0
count_lock = threading.Lock()

def worker(n):
    global count
    local = 0
    for _ in range(n):
        local += 1              # purely local, no synchronization needed
    with count_lock:
        count += local          # one synchronized update per thread

threads = [threading.Thread(target=worker, args=(200_000,)) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(count)   # 800000, exactly, on every run

This produced exactly 800000 across repeated runs, matching 4 * 200_000 precisely, because there is now only one lock-protected read-modify-write per thread instead of one per increment.

The same lesson applied to a shared dict and a lazy singleton, both verified on CPython 3.12:

import threading

# Shared dict: same compound read-modify-write as the counter.
counts_lock = threading.Lock()
counts = {}

def bump(key, n):
    for _ in range(n):
        with counts_lock:
            counts[key] = counts.get(key, 0) + 1

threads = [threading.Thread(target=bump, args=("hits", 50_000)) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(counts["hits"])   # 200000, exactly

# Lazy singleton: same check-then-act race, fixed with double-checked locking.
class Config:
    _instance = None
    _lock = threading.Lock()

    @classmethod
    def get_instance(cls):
        if cls._instance is None:            # fast path, no lock in the common case
            with cls._lock:
                if cls._instance is None:    # re-check: another thread may have
                    cls._instance = cls()    # already built it while we waited
        return cls._instance

Trade-offs & pitfalls

• Complexity: the corrected counter does $O(n)$ work per thread with $O(1)$ lock acquisitions per thread (not per increment), versus $O(n)$ lock acquisitions in a naive "lock every increment" fix; batching the merge is what keeps contention low.
• A naive fix that locks every single increment is correct but pays full lock overhead $n$ times per thread; that overhead, not correctness, is the usual reason it gets flagged as inefficient.
• The single-check if instance is None: instance = build() singleton pattern looks obviously wrong once you frame it as check-then-act, but it is one of the most common places this bug actually ships, because it is rarely exercised under real contention until traffic spikes.
• Common wrong turn: reaching for collections.Counter or a plain dict and assuming built-in types are internally synchronized against interleaved compound updates from multiple threads; they are not, their individual C-level operations may be atomic, but get-then-set across two separate operations is not.
• Edge case: the demo above uses an inserted time.sleep(0) to force the race to show up reliably; without it, whether a given run of an unprotected count += 1 loop actually shows a wrong answer depends on thread count, iteration count, and the OS scheduler, so an unprotected version passing a quick manual test is not evidence it is safe.

Direct answer

A list of dicts (row-oriented) stores each record as its own independent Python object, scattered across the heap and reached through pointers; a dict of lists or tuple-of-arrays (column-oriented) stores each field as one contiguous run of values. Row-oriented layout is convenient when you read or write whole records at once and the schema varies row to row; column-oriented layout is dramatically faster and more memory-compact for operations that touch one field across many rows, because a contiguous, uniformly-typed buffer lets tools like numpy operate on it in tight, cache-friendly C loops instead of chasing a pointer through a fresh Python object for every row.

Structured elaboration

Memory layout: in a list of dicts, each row is a full dict object (with its own internal hash table for a handful of keys) plus a separate small object for each field value; the list itself only holds pointers to these scattered dict objects. In a dict of lists, each column is one Python list whose elements still are individual boxed objects (still scattered, just grouped by field instead of by row), while a numpy array goes further and stores raw, unboxed values contiguously, with no per-element Python object at all.

CPU cache behavior: processing "one field across every row" in a list of dicts means following a different pointer for every row (poor spatial locality: each dict lives at an unrelated heap address, so the CPU's cache line brought in for one row's dict is unlikely to help with the next row) plus a hash lookup per field access. A numpy column is one contiguous buffer, so scanning it sequentially reads mostly-adjacent memory, which is exactly the access pattern CPU caches are built to reward, and it enables SIMD-style, C-level loops (SIMD: Single Instruction, Multiple Data, a CPU feature that applies one instruction to several values in a single step instead of one at a time; numpy's vectorized operations) instead of a Python-level loop with interpreter overhead on every element.

Which operations get faster or slower:

Worked example

Same aggregation (sum(price * qty)), computed three ways on 50,000 rows, verified on CPython 3.12 to confirm all three agree exactly before comparing their memory profile:

import sys
import numpy as np

N = 50_000
rows = [{"price": float(i), "qty": i % 7} for i in range(N)]                       # row-oriented
cols_list = {"price": [float(i) for i in range(N)], "qty": [i % 7 for i in range(N)]}  # column-oriented, plain lists
cols_np = {"price": np.arange(N, dtype="f8"), "qty": (np.arange(N) % 7)}            # column-oriented, numpy

total_row = sum(r["price"] * r["qty"] for r in rows)
total_col_list = sum(p * q for p, q in zip(cols_list["price"], cols_list["qty"]))
total_col_np = float(np.sum(cols_np["price"] * cols_np["qty"]))

assert total_row == total_col_list
assert abs(total_row - total_col_np) < 1e-6

All three report 3749975004.0. The memory picture behind that identical result is not identical at all:

print(sys.getsizeof(rows[0]))                 # 184: per-row dict shell alone, excludes key/value objects
print(cols_np["price"].nbytes / N)             # 8.0 bytes/element, contiguous
print(cols_np["qty"].nbytes / N)               # 8.0 bytes/element, contiguous

Every row's dict shell alone costs 184 bytes before counting a single stored value, while the entire price or qty column packs into exactly 8 bytes per element, matching a raw C double/int64 with no extra per-value overhead at all.

Trade-offs & pitfalls

• Row-oriented storage is the right default for heterogeneous, semi-structured records (each row can have a different set of fields) or workloads that mostly touch whole records at a time (an OLTP-style, online-transaction-processing pattern of single-record reads and writes); forcing such data into a rigid columnar shape adds friction for no analytical benefit.
• Column-oriented storage assumes a fixed, uniform schema; a sparse or highly variable-shape dataset needs either a hybrid design (a dense columnar core plus a fallback dict for exceptions) or accepts padding with a sentinel/NaN value for missing fields.
• Converting a large list-of-dicts into a columnar representation is itself an $O(n)$ pass with real allocation cost; doing this conversion repeatedly inside a hot loop (rather than once, at the pipeline's ingestion boundary) throws away the benefit it is meant to provide.
• A plain "dict of lists" is columnar in shape but not in memory: unlike a numpy array, its elements are still individually-boxed Python objects scattered on the heap, so it gets the API convenience of column-wise iteration without the cache-locality or memory-compactness benefit; the win from true contiguous storage only shows up once you move to numpy (or a similar typed, packed buffer).
• For large datasets, exporting the columnar in-memory representation to a columnar on-disk format (Parquet, a compressed, column-oriented file format for tabular data, storing each column's values contiguously on disk the same way this section's in-memory columnar layout does) at pipeline boundaries compounds the benefit further (compression exploits column-wise value repetition much better than row-wise), but that is a persistence decision, separate from the in-memory layout choice this question is about.

Direct answer

Python resolves a name by searching four scopes in order: Local (the current function body), Enclosing (any outer function that defines this one, for nested functions), Global (the module), Built-in (names like len and range). Lookup stops at the first scope that has the name. The catch is assignment, not lookup: if a name is assigned anywhere inside a function body, Python treats it as local to that function for the whole body, even before the assignment line runs, unless you declare it nonlocal (binds to the nearest enclosing function scope) or global (binds to module scope).

Structured elaboration

The four scopes, outside in:

• Local: parameters and names assigned inside the current function.
• Enclosing: names in any outer function that lexically contains this one (relevant only for nested functions); this is what makes closures possible.
• Global: names bound at module level.
• Built-in: the builtins module (len, range, print, and so on), checked last.

Why nonlocal exists: Python decides, at compile time, whether a name inside a function is local by scanning the function body for any assignment to that name. Reading an enclosing variable works fine without any keyword. The moment you assign to a name that lives in an enclosing scope, Python's default is to create a brand-new local variable with that name instead of reaching outward, which shadows the outer one and usually breaks the intended logic. nonlocal tells the compiler: this name is not local, resolve it in the nearest enclosing function scope (skipping straight past to global is a compile error if no enclosing binding exists). This differs from global, which always targets the module scope regardless of nesting depth.

The closure "late binding" gotcha: a nested function does not capture a variable's value at definition time, it captures the variable itself and reads it whenever the nested function actually runs. If several closures share a loop variable, they all read whatever value that variable holds when they are finally called, which is usually the loop's last value.

Worked example

Where nonlocal is required, verified on CPython 3.12:

def make_accumulator():
    total = 0
    def add(x):
        nonlocal total   # without this, `total += x` raises UnboundLocalError
        total += x
        return total
    return add

acc = make_accumulator()
print(acc(10))   # 10
print(acc(5))    # 15
print(acc(1))    # 16

What happens if you drop nonlocal (exact CPython 3.12 message):

def make_broken_accumulator():
    total = 0
    def add(x):
        total += x   # `total` is assigned here, so LEGB makes it local to add()
        return total
    return add

broken = make_broken_accumulator()
try:
    broken(10)
except UnboundLocalError as e:
    # expected: total is assigned inside add(), so LEGB makes it local for the
    # whole function body, and the read on the += happens before any assignment runs
    print(f'raises as expected: UnboundLocalError: {e}')

The late-binding closure gotcha, same mechanism (enclosing-scope lookup happens at call time, not definition time):

funcs = []
for i in range(4):
    funcs.append(lambda: i)
print([f() for f in funcs])   # [3, 3, 3, 3] -- all read the final value of i

# fix: bind the current value as a default argument, evaluated at definition time
funcs2 = []
for i in range(4):
    funcs2.append(lambda i=i: i)
print([f() for f in funcs2])  # [0, 1, 2, 3]

Trade-offs & pitfalls

• Reaching for nonlocal on every enclosing read is a common overcorrection: it is only needed when you assign to the name, never for reading it.
• nonlocal cannot create a new binding; if no enclosing function scope already has that name, Python raises a SyntaxError at compile time, it will not silently fall through to global.
• The late-binding gotcha bites hardest with lambda inside loops (event handlers, callbacks passed to a scheduler, list comprehensions of functions); the default-argument fix works because default values are evaluated once, at function-definition time, not at call time.
• Mutable enclosing state (a list or dict) sidesteps nonlocal entirely, since mutating the object's contents is not the same as rebinding the name, but that trades an explicit rebind for a shared mutable object, which is its own source of bugs in concurrent code.

Approach

Never build a shell command by string interpolation; pass the executable and its arguments as a list, which bypasses the shell entirely so a user-supplied value can never be reinterpreted as shell syntax. Resolve the executable with shutil.which so exactly which binary runs is known and explicit. Start the child in its own new session (start_new_session=True on POSIX) so it becomes the leader of its own process group, then wait on it with a timeout; if it times out, kill the whole process group, not just the one child process, so anything that child itself spawned is killed too instead of being orphaned and left running. (A process group is a set of processes the operating system lets you signal all at once, as a single unit, instead of one at a time. An orphan is a still-running child process left behind after its parent exits without cleaning it up: it does not disappear, it just gets reparented to the operating system's init process and keeps running unsupervised. A zombie is the opposite kind of loose end: a process that has already finished running, but whose exit status the parent has not yet collected via wait/communicate, so the OS keeps a small record of it around until that collection happens; failing to collect it is how zombies accumulate.)

Code (Python 3.12)

import os
import signal
import subprocess
import shutil

def run_cmd_safe(cmd, timeout=10):
    '''Run cmd (a list of args, never a shell string) and return
    (returncode, stdout, stderr). Kills the whole process group on
    timeout, not just the direct child, so any process the command
    itself spawned does not become an orphan.
    '''
    if not cmd or not isinstance(cmd, list):
        raise ValueError("cmd must be a non-empty list of arguments")
    exe = shutil.which(cmd[0])
    if exe is None:
        raise FileNotFoundError(f"{cmd[0]!r} not found on PATH")
    cmd = [exe, *cmd[1:]]

    proc = subprocess.Popen(
        cmd,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        text=True,
        start_new_session=True,  # proc.pid becomes a new process-group leader
    )
    try:
        stdout, stderr = proc.communicate(timeout=timeout)
    except subprocess.TimeoutExpired:
        os.killpg(proc.pid, signal.SIGKILL)  # kill the whole group, not just proc
        stdout, stderr = proc.communicate()  # reap the process, avoid a zombie
        raise TimeoutError(f"command timed out after {timeout}s: {cmd}") from None
    return proc.returncode, stdout, stderr

rc, out, err = run_cmd_safe(["echo", "ok"], timeout=5)
print(rc, out.strip())
# 0 ok

try:
    run_cmd_safe(["python3", "-c", "import time; time.sleep(5)"], timeout=0.3)
except TimeoutError as e:
    print("caught:", e)
# caught: command timed out after 0.3s: [...]

Why the process-group kill matters (a correction to a common pattern)

A common but incomplete pattern catches subprocess.TimeoutExpired and calls e.process.kill(). Verified on Python 3.12: TimeoutExpired does not carry a process attribute at all (dir() on a caught instance shows args, cmd, output, stderr, stdout, timeout, nothing else), so that line raises AttributeError and never kills anything. Even keeping the process handle separately and calling proc.kill() only sends the kill signal to the direct child's process ID, not to anything that child itself spawned. If the command forks or execs a further subprocess (a shell script backgrounding a helper process, for example), killing only the direct child leaves that grandchild running as an orphan, exactly the failure mode the question asks to avoid.

This follows directly from standard POSIX process semantics (see man 2 kill, man 2 setsid): a signal sent to one process ID never propagates to that process's own children, they simply get reparented to the operating system's init process and keep running on their own schedule. start_new_session=True is what makes the child (and anything it spawns) share one process group under proc.pid, so os.killpg(proc.pid, signal.SIGKILL) (SIGKILL is the unconditional, uncatchable kill signal) reaches all of them in one call instead of only the direct child. Note that subprocess.run() itself already calls process.kill() and waits on it internally on timeout, so it never leaves a zombie for the direct child; the gap it does not close is exactly this grandchild-orphan case, which is why the version above uses Popen with start_new_session and killpg directly rather than subprocess.run.

Key points

• Passing arguments as a list, never shell=True with a formatted string, is what actually removes the shell-injection risk; shlex.quote can make a shell string safer, but a list removes the entire injection class rather than mitigating it.
• start_new_session=True makes proc.pid double as the process group ID, which is why os.killpg(proc.pid, ...) needs no separate lookup.
• communicate(timeout=...) is used instead of wait(timeout=...) because it also drains the stdout/stderr pipes as it waits; waiting without draining a pipe can deadlock once the child fills the OS pipe buffer and blocks on writing to it.
• After killing on timeout, proc.communicate() (no timeout argument) is called once more to reap the process, avoiding a zombie entry, and to collect whatever partial output the child had already produced.

Complexity

Not meaningful in the big-O sense here; the cost is dominated by the external process's own runtime and by OS-level process management calls, not by anything in the Python wrapper. The wrapper's own overhead, setting up a couple of pipes, is negligible next to spawning an entire process.

Edge cases

• Executable not found: shutil.which returns None, so FileNotFoundError is raised before Popen is ever called, an immediate, clear failure instead of a confusing error surfaced from exec.
• Timeout with output already produced: communicate() after the kill still returns whatever the child had already written to its pipes before being killed.
• Non-POSIX (Windows): start_new_session and os.killpg are POSIX-only; the Windows equivalent is creating the process with CREATE_NEW_PROCESS_GROUP and sending CTRL_BREAK_EVENT, or killing the process tree by process ID (PID) with taskkill /T /F, since Windows has no direct killpg equivalent.
• Command exits normally with a non-zero return code: this is not an exception at all here (check is not set), the caller inspects returncode; only a timeout raises.