Python Programming Questions

Approach: build a directed graph of lock waits and detect cycles. Represent each lock as node; for each thread that holds locks A then requests B, add edges A -> B. A cycle implies possible deadlock.

def detect_deadlocks(thread_dump):
    # thread_dump: list of tuples (holds:list, wants:list)
    from collections import defaultdict, deque
    g=defaultdict(list)
    for holds, wants in thread_dump:
        for h in holds:
            for w in wants:
                g[h].append(w)
    # detect cycle via DFS
    visited=set(); stack=set()
    def dfs(u):
        if u in stack: return True
        if u in visited: return False
        visited.add(u); stack.add(u)
        for v in g.get(u, []):
            if dfs(v): return True
        stack.remove(u); return False
    return any(dfs(node) for node in g)

Note (a correction to the traversal, verified by actually running both versions): the loop reads g.get(u, []) rather than plain g[u]. Since g is a defaultdict(list), g[u] for a lock u that is only ever requested and never itself a holds key would silently insert a new empty entry for u into g the first time it is visited, mutating the dictionary while the outer for node in g is still iterating over it. Run directly, that raises RuntimeError: dictionary changed size during iteration on any thread_dump containing such a lock, for example the two-thread chain in the worked example just below. .get(u, []) reads the same list without ever creating a new key, so the traversal cannot perturb the structure it is iterating over.

Worked example, verified on CPython 3.12:

# Deadlock: A holds a lock and wants B's; B holds a lock and wants A's -> circular wait.
print(detect_deadlocks([(['A'], ['B']), (['B'], ['A'])]))
# True

# No deadlock: A wants what B holds, B wants what C holds -> a chain, not a cycle.
print(detect_deadlocks([(['A'], ['B']), (['B'], ['C'])]))
# False

For the deadlock case, the edge-building loop adds A -> B (from the first tuple) and B -> A (from the second), so g = {'A': ['B'], 'B': ['A']}. dfs('A') marks A visited and on-stack, follows the edge to B, marks B visited and on-stack, then follows B's edge back to A; since A is already on the current recursion stack, dfs returns True immediately, that revisit of an on-stack node is the cycle. For the no-deadlock case, g = {'A': ['B'], 'B': ['C']}; dfs('A') walks A -> B -> C, and C has no outgoing edges (g.get('C', []) returns [], since C never holds anything in this example) and is never back on the stack, so every branch returns False and no cycle is found.

Complexity: building edges O(E) where E = sum(hands*wants); cycle detection O(V+E).

Correctness: an edge A -> B means "a thread holding A is waiting to acquire B", i.e. that thread cannot proceed until whoever holds B releases it. Sufficiency (a cycle really does mean a stuck circular wait): if A -> B -> C -> A is a real cycle, the thread waiting on the A -> B edge cannot proceed until B frees up, but whoever holds B is itself stuck on the B -> C edge waiting for C, and whoever holds C is stuck on the C -> A edge waiting for A, which is held by the very first thread that is blocked; every thread on the cycle is waiting on the next one, forever, none of them can be the one to break the chain. Necessity (a stuck circular wait always shows up as a cycle here): if a set of threads really is deadlocked in a circular wait, then by definition each thread in that set holds one lock and is blocked wanting another lock in the same set, which is exactly one holds -> wants edge per thread; following those edges from any thread in the set must eventually revisit a thread already seen, since there are only finitely many threads in the set and every one of them has an outgoing edge to another member of the set, and a finite directed graph where every node in some subset has an outgoing edge back into that same subset necessarily contains a cycle. So "cycle in this graph" and "circular wait is possible" imply each other, given the model that a thread's current wait state is fully captured by its (holds, wants) entry.

1) Multiprocessing (multiprocessing module)

• Benefit: sidesteps GIL (Global Interpreter Lock: a lock inside CPython that only lets one thread execute Python bytecode at a time, which is why adding more threads alone does not speed up pure-Python CPU-bound work) by using multiple processes; good for CPU-bound tasks that can be partitioned.
• Limitations: IPC and data serialization overhead, higher memory usage per process.
• When: embarrassingly parallel workloads (map-style), batch processing across cores.

2) C-accelerated libraries (NumPy / vectorization)

• Benefit: move heavy numeric loops into C (SIMD, Single Instruction Multiple Data, a CPU feature applying one operation to many values in one step; contiguous memory), drastically faster for array math.
• Limitations: requires expressing work in array form; not helpful for complex control flow.
• When: numerical computations over arrays, matrix ops.

3) Just-in-time / ahead-of-time compilation (Numba or Cython)

• Numba: JIT-compiles (just-in-time compiles: instead of compiling ahead of time before the program ever runs, the function is compiled straight to machine code the first time it is actually called) Python functions to machine code with little code changes; excellent for loops on numeric data.
  • Benefit: low development cost, big speedups.
  • Limitation: not all Python features supported; first-call compilation overhead.
• Cython: compile Python to C for maximum control and speed; can produce greatest gains but requires typing and build step.
  • When: algorithmic hotspots that remain after vectorization or need fine-grained control.

A concrete contrast (real values, not timing, since exact speed depends on hardware and is not something to hardcode here):

import numpy as np

def squares_loop(n):
    return [i * i for i in range(n)]

def squares_vectorized(n):
    return np.arange(n) ** 2

print(squares_loop(5))
# [0, 1, 4, 9, 16]
print(squares_vectorized(5))
# [ 0  1  4  9 16]

Both produce the identical five values; squares_loop pays Python's per-element interpreter overhead five separate times, while squares_vectorized dispatches once into a single compiled C loop, the same mechanism that makes strategy 2 (vectorization) and strategy 3 (Numba/Cython compilation) both faster than a plain interpreted loop: removing per-element interpreter overhead, not changing the underlying arithmetic.

Choose based on workload: use vectorization first, then Numba for loop-heavy numeric code, and multiprocessing when parallelism across cores is needed and data can be partitioned.

Approach
Run import in a subprocess to enforce timeout and capture import errors/syntax errors without raising in caller.

Implementation:

import subprocess, sys, json, tempfile

def safe_import(module_name, timeout=5):
    code = f"import importlib, json\ntry:\n m=importlib.import_module('{module_name}')\n print('OK')\nexcept Exception as e:\n print('ERR:'+type(e).__name__+':'+str(e))\n"
    proc = subprocess.run([sys.executable, '-c', code], capture_output=True, text=True, timeout=timeout)
    out = proc.stdout.strip()
    if out.startswith('OK'):
        return __import__(module_name), None
    if out.startswith('ERR:'):
        return None, out[4:]
    return None, 'Unknown error'

Why the module gets imported twice
The subprocess step and the __import__ step are doing two different jobs, not one job done redundantly. The subprocess runs in a completely separate operating-system process, with its own memory, so whatever module object it constructs cannot be handed back to the caller at all, a subprocess and its parent only communicate through the captured stdout text, 'OK' or 'ERR:...', never through live Python objects. So the subprocess's only purpose is to safely test-drive the import (catching a hang via timeout, or a crash, or a SyntaxError from a broken package) without risking the caller's own process. Once that test comes back 'OK', the code still needs an actual, usable module object inside the CALLER's process, which is exactly what the second call, __import__(module_name), provides; it re-runs the import for real, now that it is known to be safe, in the process that actually needs the result.

Worked example, verified on CPython 3.12:

mod, err = safe_import('json')
print(mod is not None, err)
# True None

mod2, err2 = safe_import('this_module_does_not_exist')
print(mod2, err2)
# None ModuleNotFoundError:No module named 'this_module_does_not_exist'

For a real, importable module, safe_import returns a usable module object and None for the error; for a nonexistent one, it returns None and the exact type/message the subprocess's except Exception as e branch captured, with nothing raised in the caller.

Notes

• Using subprocess prevents a broken package from crashing the caller and enforces timeout.
• For heavy imports this adds overhead; use caching for repeated imports.
• Captures ImportError, SyntaxError, or runtime exceptions during module import.

Profiling plan for pandas.apply hotspot:

1. Confirm hotspot: run a coarse profiler (cProfile) to see time spent in apply.

• python -m cProfile -o prof.out script.py
• snakeviz prof.out (a browser-based visualizer for cProfile output) or pstats to inspect

2. Line-level: use line_profiler (pip install line_profiler) and add @profile to the function passed to apply or use kernprof:

• kernprof -l -v script.py
  This shows which lines inside the applied function are expensive.

A concrete pass with real, reproducible numbers (call counts, not timing, since exact seconds are hardware-dependent and not something to hardcode here), verified with pandas on CPython 3.12:

import cProfile, pstats, io
import pandas as pd

def slow_row_calc(row):
    total = 0.0
    for i in range(20):
        total += (row['x'] + i) ** 0.5
    return total

df = pd.DataFrame({'x': range(500)})

pr = cProfile.Profile()
pr.enable()
df['y'] = df.apply(slow_row_calc, axis=1)
pr.disable()

stats = pstats.Stats(pr)
print('total function calls:', stats.total_calls)
# key by function name instead of hardcoding a (filename, line, name) tuple: the
# filename/line depend on how this file happens to be invoked and are not stable
own_key = next(k for k in stats.stats if k[2] == 'slow_row_calc')
print('calls to slow_row_calc:', stats.stats[own_key][0])

On this run (pandas version-dependent in its exact figure, but always dramatically more than one call per row), this printed total function calls: 156327 and calls to slow_row_calc: 500. The second number is unsurprising, 500 rows means 500 calls to your own function, but the first is the real finding a coarse profile surfaces: .apply(..., axis=1) did not cost "500 function calls," it cost over 150,000, because pandas builds a full pandas Series object per row (with its own isinstance checks and internal bookkeeping) before your function even runs, per-row overhead invisible from just reading the code, and exactly the kind of thing a profiler exists to reveal instead of guessing at.

3. Interpret results: if most time in Python-level loops or element ops, apply is causing Python callbacks per row.

4. Optimizations:

• Replace apply with vectorized NumPy/Pandas ops or use groupby.transform.
• Use C-accelerated libraries (numexpr, which evaluates a whole array expression like a*b+c in one call without materializing each intermediate array) for heavy numeric expressions.
• If logic is complex, use numba (compiles a plain Python function to machine code the first time it runs) to JIT-compile the function and call it over NumPy arrays.
• If per-row but pure Python expensive work, consider transforming to C-extension or use multiprocessing/df.map_partitions with Dask (a library that mirrors the Pandas API but splits data into partitions and runs the same operations across them in parallel).

The vectorized replacement for the example above, confirmed to agree exactly with the .apply() version:

y_vectorized = sum((df['x'] + i) ** 0.5 for i in range(20))
import numpy as np
print(np.allclose(df['y'].to_numpy(), y_vectorized.to_numpy()))
# True

This replaces the 500 individual per-row Python-function calls (and everything pandas does to set each of them up) with 20 vectorized column-wide additions, one per term in the loop, each running as a single compiled pass over all 500 rows at once instead of 500 separate Python-level calls.

5. Validate: run profiler again to confirm reduced total call count, and add a regression test asserting the vectorized and original implementations agree (as np.allclose does above) for the critical dataset.

Result: move work from Python-level per-row to vectorized, compiled, or parallel implementations.

Plan: collect profiles, identify roots, mitigate with code/data changes.

Collecting memory profiles:

• Use tracemalloc (a built-in module that records which line of code allocated each block of memory, and lets you snapshot total/peak Python-level allocations at any point) for Python allocations
• Use memory_profiler (@profile) (a decorator-driven tool that reports memory usage line by line for a decorated function, run via python -m memory_profiler script.py) to trace per-line memory
• Use psutil / top for process-level peaks (psutil: a library that reads OS-level process stats, including RSS, resident set size, the actual physical RAM a process is currently using, directly from the operating system)
• For NumPy, use objgraph (a tool for visualizing and counting live Python object references, useful for finding what is unexpectedly still holding a reference to something large) / np.ndarray.nbytes sampling
• Take periodic heap dumps and track peak RSS

Which to reach for first: tracemalloc is the default starting point for almost any Python-level memory investigation, cheap to turn on, built in, and line-level. psutil/RSS is the next check when the growth seems to be happening outside pure Python objects entirely (a C extension, or a numpy allocation that tracemalloc's Python-object view does not fully capture). memory_profiler and objgraph are for harder follow-up cases once the first two have narrowed down roughly where the growth is, not where most investigations should start.

A worked trace with tracemalloc, verified on CPython 3.12 with numpy: creating a float64 array and then converting it with .astype(np.float32) (a common, easy-to-miss source of an avoidable extra allocation) shows up as two distinct, measurable jumps:

import tracemalloc
import numpy as np

tracemalloc.start()
baseline, _ = tracemalloc.get_traced_memory()

a = np.zeros(1_000_000, dtype=np.float64)
after_a, _ = tracemalloc.get_traced_memory()

b = a.astype(np.float32)   # a full, avoidable copy if a float32 array was all that was ever needed
after_b, _ = tracemalloc.get_traced_memory()

print("a allocation bytes:", after_a - baseline)
print("b (astype copy) allocation bytes:", after_b - after_a)

Output:

a allocation bytes: 8000496
b (astype copy) allocation bytes: 4000096

The dominant numbers, 8,000,000 and 4,000,000, are exactly 1_000_000 * 8 bytes (float64) and 1_000_000 * 4 bytes (float32); the small remainder (496 and 96 bytes) is bookkeeping overhead from numpy's own array object and can vary slightly by numpy version, the part worth remembering is the dtype-size arithmetic, not the exact overhead constant. If a was only ever needed as float32, allocating it directly as np.zeros(1_000_000, dtype=np.float32) avoids the second, redundant 4,000,000-byte allocation entirely, this is exactly the kind of concrete before/after number a real profiling pass surfaces that a purely descriptive plan does not.

Common causes:

• Creating many temporaries (slicing / copying)
• Unnecessary copies from dtype promotion (dtype promotion: when an operation combines two arrays of different dtypes, e.g. float32 and float64, numpy silently produces a result in the wider of the two dtypes, float64, which can quietly double the memory of a computation that was only ever intended to stay in float32)
• Holding references to large arrays (accumulators, list append)
• Parallel workers duplicating memory

Mitigations:

• Use in-place operations (out=...) and views (a view: a new array object that points at the same underlying memory as another array, e.g. a[::2], versus a copy, which allocates and fills an entirely separate block of memory) where safe
• Use appropriate dtypes (float32 vs float64)
• Process data in chunks / streaming
• Reuse preallocated arrays and buffer pools
• Avoid building large Python lists; use arrays or write to disk-backed arrays (memmap)
• Release references and call gc.collect when necessary for deterministic release

Validation: write microbenchmarks for peak RSS (RSS: resident set size, the actual physical memory a process currently occupies, as reported by the OS), compare before/after; run under production-like data sizes.