Python Programming Questions

Approach: build a directed graph of lock waits and detect cycles. Represent each lock as node; for each thread that holds locks A then requests B, add edges A -> B. A cycle implies possible deadlock.

def detect_deadlocks(thread_dump):
    # thread_dump: list of tuples (holds:list, wants:list)
    from collections import defaultdict, deque
    g=defaultdict(list)
    for holds, wants in thread_dump:
        for h in holds:
            for w in wants:
                g[h].append(w)
    # detect cycle via DFS
    visited=set(); stack=set()
    def dfs(u):
        if u in stack: return True
        if u in visited: return False
        visited.add(u); stack.add(u)
        for v in g.get(u, []):
            if dfs(v): return True
        stack.remove(u); return False
    return any(dfs(node) for node in g)

Note (a correction to the traversal, verified by actually running both versions): the loop reads g.get(u, []) rather than plain g[u]. Since g is a defaultdict(list), g[u] for a lock u that is only ever requested and never itself a holds key would silently insert a new empty entry for u into g the first time it is visited, mutating the dictionary while the outer for node in g is still iterating over it. Run directly, that raises RuntimeError: dictionary changed size during iteration on any thread_dump containing such a lock, for example the two-thread chain in the worked example just below. .get(u, []) reads the same list without ever creating a new key, so the traversal cannot perturb the structure it is iterating over.

Worked example, verified on CPython 3.12:

# Deadlock: A holds a lock and wants B's; B holds a lock and wants A's -> circular wait.
print(detect_deadlocks([(['A'], ['B']), (['B'], ['A'])]))
# True

# No deadlock: A wants what B holds, B wants what C holds -> a chain, not a cycle.
print(detect_deadlocks([(['A'], ['B']), (['B'], ['C'])]))
# False

For the deadlock case, the edge-building loop adds A -> B (from the first tuple) and B -> A (from the second), so g = {'A': ['B'], 'B': ['A']}. dfs('A') marks A visited and on-stack, follows the edge to B, marks B visited and on-stack, then follows B's edge back to A; since A is already on the current recursion stack, dfs returns True immediately, that revisit of an on-stack node is the cycle. For the no-deadlock case, g = {'A': ['B'], 'B': ['C']}; dfs('A') walks A -> B -> C, and C has no outgoing edges (g.get('C', []) returns [], since C never holds anything in this example) and is never back on the stack, so every branch returns False and no cycle is found.

Complexity: building edges O(E) where E = sum(hands*wants); cycle detection O(V+E).

Correctness: an edge A -> B means "a thread holding A is waiting to acquire B", i.e. that thread cannot proceed until whoever holds B releases it. Sufficiency (a cycle really does mean a stuck circular wait): if A -> B -> C -> A is a real cycle, the thread waiting on the A -> B edge cannot proceed until B frees up, but whoever holds B is itself stuck on the B -> C edge waiting for C, and whoever holds C is stuck on the C -> A edge waiting for A, which is held by the very first thread that is blocked; every thread on the cycle is waiting on the next one, forever, none of them can be the one to break the chain. Necessity (a stuck circular wait always shows up as a cycle here): if a set of threads really is deadlocked in a circular wait, then by definition each thread in that set holds one lock and is blocked wanting another lock in the same set, which is exactly one holds -> wants edge per thread; following those edges from any thread in the set must eventually revisit a thread already seen, since there are only finitely many threads in the set and every one of them has an outgoing edge to another member of the set, and a finite directed graph where every node in some subset has an outgoing edge back into that same subset necessarily contains a cycle. So "cycle in this graph" and "circular wait is possible" imply each other, given the model that a thread's current wait state is fully captured by its (holds, wants) entry.

1) Multiprocessing (multiprocessing module)

• Benefit: sidesteps GIL (Global Interpreter Lock: a lock inside CPython that only lets one thread execute Python bytecode at a time, which is why adding more threads alone does not speed up pure-Python CPU-bound work) by using multiple processes; good for CPU-bound tasks that can be partitioned.
• Limitations: IPC and data serialization overhead, higher memory usage per process.
• When: embarrassingly parallel workloads (map-style), batch processing across cores.

2) C-accelerated libraries (NumPy / vectorization)

• Benefit: move heavy numeric loops into C (SIMD, Single Instruction Multiple Data, a CPU feature applying one operation to many values in one step; contiguous memory), drastically faster for array math.
• Limitations: requires expressing work in array form; not helpful for complex control flow.
• When: numerical computations over arrays, matrix ops.

3) Just-in-time / ahead-of-time compilation (Numba or Cython)

• Numba: JIT-compiles (just-in-time compiles: instead of compiling ahead of time before the program ever runs, the function is compiled straight to machine code the first time it is actually called) Python functions to machine code with little code changes; excellent for loops on numeric data.
  • Benefit: low development cost, big speedups.
  • Limitation: not all Python features supported; first-call compilation overhead.
• Cython: compile Python to C for maximum control and speed; can produce greatest gains but requires typing and build step.
  • When: algorithmic hotspots that remain after vectorization or need fine-grained control.

A concrete contrast (real values, not timing, since exact speed depends on hardware and is not something to hardcode here):

import numpy as np

def squares_loop(n):
    return [i * i for i in range(n)]

def squares_vectorized(n):
    return np.arange(n) ** 2

print(squares_loop(5))
# [0, 1, 4, 9, 16]
print(squares_vectorized(5))
# [ 0  1  4  9 16]

Both produce the identical five values; squares_loop pays Python's per-element interpreter overhead five separate times, while squares_vectorized dispatches once into a single compiled C loop, the same mechanism that makes strategy 2 (vectorization) and strategy 3 (Numba/Cython compilation) both faster than a plain interpreted loop: removing per-element interpreter overhead, not changing the underlying arithmetic.

Choose based on workload: use vectorization first, then Numba for loop-heavy numeric code, and multiprocessing when parallelism across cores is needed and data can be partitioned.

The Global Interpreter Lock (GIL) is a mutex in CPython that ensures only one native thread executes Python bytecode at a time. It simplifies memory management but serializes CPU-bound Python code.

Workloads where threads still help despite the GIL:

• I/O-bound workloads: threads waiting on network, disk, or database I/O release the GIL during blocking calls, improving throughput (e.g., web crawlers, concurrent HTTP clients). Concretely, verified on CPython 3.12: two threads that each print, sleep (standing in for a blocking network call), then print again:

import threading, time

order = []

def worker(name, delay):
    order.append(f"{name} start")
    time.sleep(delay)      # GIL released for the duration of the sleep
    order.append(f"{name} done")

t1 = threading.Thread(target=worker, args=("A", 0.1))
t2 = threading.Thread(target=worker, args=("B", 0.1))
t1.start(); t2.start()
t1.join(); t2.join()
print(order)

A representative run printed ['A start', 'B start', 'B done', 'A done']: both start entries appear before either done, which is only possible because thread A released the GIL during its time.sleep(0.1), letting thread B start and run its own code during that wait, rather than A blocking B out entirely the way a genuine CPU-bound loop with no sleep/I-O would (there, the two threads would still only ever have one of them executing Python bytecode at a time, and the interleaving above would not occur). The exact order of the two done entries is not guaranteed (it depends on OS scheduling), but the invariant that matters, both starts landing before either finish, reliably demonstrates the GIL being released during the blocking wait, not a specific timing.

• Native-code parallelism: when threads call C extensions that release the GIL (numpy heavy computations, image processing libraries, cryptography), CPU work runs in parallel across cores.

For CPU-bound pure-Python tasks use multiprocessing or native extensions to achieve parallelism.

Approach

I would stream the file line by line so memory stays bounded, meaning it stays roughly constant no matter how large the file is. For each line, trim whitespace, skip blanks, try to parse the fields, and update counters. Malformed lines go into a small issue count or log so the job can continue.

from dataclasses import dataclass


@dataclass
class Report:
    total: int = 0
    overdue: int = 0
    days_overdue_sum: int = 0
    malformed: int = 0


def process_tickets(path: str) -> Report:
    report = Report()

    with open(path, 'r', encoding='utf-8') as f:
        for line_no, line in enumerate(f, start=1):
            line = line.strip()
            if not line:
                continue

            parts = line.split(',')
            if len(parts) != 3:
                report.malformed += 1
                continue

            try:
                status = parts[1].strip().lower()
                days = int(parts[2].strip())
            except ValueError:
                report.malformed += 1
                continue

            if not parts[0].strip():
                report.malformed += 1
                continue

            report.total += 1
            if status == 'overdue':
                report.overdue += 1
                report.days_overdue_sum += days

    return report

Worked example: with A1,overdue,3, a blank line, and B2,open,0, the report counts 2 valid records, 1 overdue ticket, and 3 total overdue days.

Tradeoffs

• Fast and memory-safe for huge files
• One pass only, so you should add any needed counters up front
• If operators need examples of bad lines, log the first few rather than storing every bad line

Complexity: O(n) time and O(1) memory for the counters.

Why vectorized ops are faster:

• Vectorized NumPy/Pandas operations run in optimized C/Fortran loops avoiding Python per-element overhead.
• They leverage contiguous memory, CPU cache, SIMD (Single Instruction, Multiple Data: a CPU feature that applies one instruction to several values in a single step, instead of one value at a time), and multi-threaded BLAS (Basic Linear Algebra Subprogram, a standard library of fast, hardware-tuned matrix/vector math routines that numpy calls into) for numeric work.
• Example: adding two arrays uses a single C loop vs Python loop calling millions of Python operations.

Worked example, verified on CPython 3.12 (showing what actually gets computed, not a timing benchmark: exact speed depends on hardware and array size, so run time.perf_counter() yourself to see the real gap on your own machine rather than trusting a hardcoded number here):

import numpy as np

a = np.array([1.0, 2.0, 3.0, 4.0])
b = np.array([10.0, 20.0, 30.0, 40.0])

def add_loop(a, b):
    return [x + y for x, y in zip(a, b)]

def add_vectorized(a, b):
    return a + b

print(add_loop(a, b))
# [11.0, 22.0, 33.0, 44.0]
print(add_vectorized(a, b))
# [11. 22. 33. 44.]

Both compute the identical values; add_loop does it via four separate Python-level additions, each one dispatched, type-checked, and reference-counted by the interpreter, while add_vectorized's a + b dispatches once into a single compiled C loop over the two contiguous memory buffers, which is the mechanism, not a specific timing number, that makes the vectorized version pull ahead as array size grows.

When vectorization can be slower:

• Small arrays: overhead of creating intermediate arrays and function call overhead can dominate; a simple Python loop or in-place updates may be faster.
• Complex element-wise logic with branching: vectorization may require multiple large temporaries or complex masking, increasing memory bandwidth and runtime.

Where vectorization stops winning, made concrete: a piecewise function like "if x < 0, return 0; elif x < 10, return x; else return x**2" vectorizes via np.where, but every branch has to be computed for every element before the mask selects which result to keep:

x = np.array([-3.0, 5.0, 15.0])
result = np.where(x < 0, 0.0, np.where(x < 10, x, x ** 2))
print(result)
# [  0.   5. 225.]

Even though only one branch is "true" for any given element, x ** 2 is computed for all three elements (including -3.0 and 5.0, whose squared values are simply discarded by the outer np.where), and each np.where call allocates a full temporary array the size of x. For a genuinely small array or a function with many branches, that wasted work and temporary-array allocation can cost more than a plain Python loop that only evaluates the one branch each element actually needs; this is the concrete shape of "vectorization can be slower."

Example: computing a piecewise function with many branches can be slower vectorized due to multiple masks and temporary arrays; using numba (a library that compiles a plain Python function to machine code the first time it runs, letting a genuinely branchy per-element loop run at near-C speed without vectorizing it at all) for a compiled loop may be faster and more memory-efficient.