Python Programming Questions

Approach: build a directed graph of lock waits and detect cycles. Represent each lock as node; for each thread that holds locks A then requests B, add edges A -> B. A cycle implies possible deadlock.

def detect_deadlocks(thread_dump):
    # thread_dump: list of tuples (holds:list, wants:list)
    from collections import defaultdict, deque
    g=defaultdict(list)
    for holds, wants in thread_dump:
        for h in holds:
            for w in wants:
                g[h].append(w)
    # detect cycle via DFS
    visited=set(); stack=set()
    def dfs(u):
        if u in stack: return True
        if u in visited: return False
        visited.add(u); stack.add(u)
        for v in g.get(u, []):
            if dfs(v): return True
        stack.remove(u); return False
    return any(dfs(node) for node in g)

Note (a correction to the traversal, verified by actually running both versions): the loop reads g.get(u, []) rather than plain g[u]. Since g is a defaultdict(list), g[u] for a lock u that is only ever requested and never itself a holds key would silently insert a new empty entry for u into g the first time it is visited, mutating the dictionary while the outer for node in g is still iterating over it. Run directly, that raises RuntimeError: dictionary changed size during iteration on any thread_dump containing such a lock, for example the two-thread chain in the worked example just below. .get(u, []) reads the same list without ever creating a new key, so the traversal cannot perturb the structure it is iterating over.

Worked example, verified on CPython 3.12:

# Deadlock: A holds a lock and wants B's; B holds a lock and wants A's -> circular wait.
print(detect_deadlocks([(['A'], ['B']), (['B'], ['A'])]))
# True

# No deadlock: A wants what B holds, B wants what C holds -> a chain, not a cycle.
print(detect_deadlocks([(['A'], ['B']), (['B'], ['C'])]))
# False

For the deadlock case, the edge-building loop adds A -> B (from the first tuple) and B -> A (from the second), so g = {'A': ['B'], 'B': ['A']}. dfs('A') marks A visited and on-stack, follows the edge to B, marks B visited and on-stack, then follows B's edge back to A; since A is already on the current recursion stack, dfs returns True immediately, that revisit of an on-stack node is the cycle. For the no-deadlock case, g = {'A': ['B'], 'B': ['C']}; dfs('A') walks A -> B -> C, and C has no outgoing edges (g.get('C', []) returns [], since C never holds anything in this example) and is never back on the stack, so every branch returns False and no cycle is found.

Complexity: building edges O(E) where E = sum(hands*wants); cycle detection O(V+E).

Correctness: an edge A -> B means "a thread holding A is waiting to acquire B", i.e. that thread cannot proceed until whoever holds B releases it. Sufficiency (a cycle really does mean a stuck circular wait): if A -> B -> C -> A is a real cycle, the thread waiting on the A -> B edge cannot proceed until B frees up, but whoever holds B is itself stuck on the B -> C edge waiting for C, and whoever holds C is stuck on the C -> A edge waiting for A, which is held by the very first thread that is blocked; every thread on the cycle is waiting on the next one, forever, none of them can be the one to break the chain. Necessity (a stuck circular wait always shows up as a cycle here): if a set of threads really is deadlocked in a circular wait, then by definition each thread in that set holds one lock and is blocked wanting another lock in the same set, which is exactly one holds -> wants edge per thread; following those edges from any thread in the set must eventually revisit a thread already seen, since there are only finitely many threads in the set and every one of them has an outgoing edge to another member of the set, and a finite directed graph where every node in some subset has an outgoing edge back into that same subset necessarily contains a cycle. So "cycle in this graph" and "circular wait is possible" imply each other, given the model that a thread's current wait state is fully captured by its (holds, wants) entry.

Profiling plan for pandas.apply hotspot:

1. Confirm hotspot: run a coarse profiler (cProfile) to see time spent in apply.

• python -m cProfile -o prof.out script.py
• snakeviz prof.out (a browser-based visualizer for cProfile output) or pstats to inspect

2. Line-level: use line_profiler (pip install line_profiler) and add @profile to the function passed to apply or use kernprof:

• kernprof -l -v script.py
  This shows which lines inside the applied function are expensive.

A concrete pass with real, reproducible numbers (call counts, not timing, since exact seconds are hardware-dependent and not something to hardcode here), verified with pandas on CPython 3.12:

import cProfile, pstats, io
import pandas as pd

def slow_row_calc(row):
    total = 0.0
    for i in range(20):
        total += (row['x'] + i) ** 0.5
    return total

df = pd.DataFrame({'x': range(500)})

pr = cProfile.Profile()
pr.enable()
df['y'] = df.apply(slow_row_calc, axis=1)
pr.disable()

stats = pstats.Stats(pr)
print('total function calls:', stats.total_calls)
# key by function name instead of hardcoding a (filename, line, name) tuple: the
# filename/line depend on how this file happens to be invoked and are not stable
own_key = next(k for k in stats.stats if k[2] == 'slow_row_calc')
print('calls to slow_row_calc:', stats.stats[own_key][0])

On this run (pandas version-dependent in its exact figure, but always dramatically more than one call per row), this printed total function calls: 156327 and calls to slow_row_calc: 500. The second number is unsurprising, 500 rows means 500 calls to your own function, but the first is the real finding a coarse profile surfaces: .apply(..., axis=1) did not cost "500 function calls," it cost over 150,000, because pandas builds a full pandas Series object per row (with its own isinstance checks and internal bookkeeping) before your function even runs, per-row overhead invisible from just reading the code, and exactly the kind of thing a profiler exists to reveal instead of guessing at.

3. Interpret results: if most time in Python-level loops or element ops, apply is causing Python callbacks per row.

4. Optimizations:

• Replace apply with vectorized NumPy/Pandas ops or use groupby.transform.
• Use C-accelerated libraries (numexpr, which evaluates a whole array expression like a*b+c in one call without materializing each intermediate array) for heavy numeric expressions.
• If logic is complex, use numba (compiles a plain Python function to machine code the first time it runs) to JIT-compile the function and call it over NumPy arrays.
• If per-row but pure Python expensive work, consider transforming to C-extension or use multiprocessing/df.map_partitions with Dask (a library that mirrors the Pandas API but splits data into partitions and runs the same operations across them in parallel).

The vectorized replacement for the example above, confirmed to agree exactly with the .apply() version:

y_vectorized = sum((df['x'] + i) ** 0.5 for i in range(20))
import numpy as np
print(np.allclose(df['y'].to_numpy(), y_vectorized.to_numpy()))
# True

This replaces the 500 individual per-row Python-function calls (and everything pandas does to set each of them up) with 20 vectorized column-wide additions, one per term in the loop, each running as a single compiled pass over all 500 rows at once instead of 500 separate Python-level calls.

5. Validate: run profiler again to confirm reduced total call count, and add a regression test asserting the vectorized and original implementations agree (as np.allclose does above) for the critical dataset.

Result: move work from Python-level per-row to vectorized, compiled, or parallel implementations.

Go vocabulary, translated for a Python reader: a goroutine is Go's lightweight, concurrently-running function, similar in spirit to a Python thread but far cheaper to start and typically used in much greater numbers in real Go code; a channel is a typed, thread-safe queue you send values into and receive values out of, roughly like a queue.Queue shared between Python threads, except the compiler enforces the type of what flows through it; a WaitGroup is a counter that lets the caller block until every worker goroutine has signaled it is done, similar to calling .join() on a list of Python Thread objects; ctx (short for context) carries a shared cancellation signal through the call tree, and ctx.Done() returns a channel that closes the moment that signal fires, so any goroutine can cheaply check "has someone asked everything to stop?" without polling a shared boolean, comparable to checking a Python threading.Event.

Structure

I’d use a bounded worker pool. A feeder sends files with an index into a jobs channel. Each worker validates one file, sends {index, result, err} to a results channel, and watches ctx.Done() so context cancellation, meaning a cooperative stop signal, is fast.

Ordering
A single collector keeps nextIndex and a map of out-of-order results. If result 7 arrives before 6, store it until 6 is ready, then flush in submission order.

Fatal parse error
If a worker sees an unrecoverable parse error, it sends the error and calls cancel(). That stops the feeder, makes workers exit on ctx.Done(), and prevents new work from starting.

Shutdown

• close jobs after feeding stops
• WaitGroup waits for workers
• close results after workers finish
• collector drains until closed or canceled

Worked example: files 1, 2, 3, 4 arrive. If file 3 has a fatal parse error, 1 and 2 can still be emitted, 4 is never started, and the summary reports the error immediately.

Shape of the code (a sketch of the structure, not a full compiled program):

type job struct {
    index int
    path  string
}
type result struct {
    index  int
    output string
    err    error
}

func run(ctx context.Context, paths []string, numWorkers int) []result {
    ctx, cancel := context.WithCancel(ctx)
    defer cancel()

    jobs := make(chan job)
    results := make(chan result)
    var wg sync.WaitGroup

    for w := 0; w < numWorkers; w++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for j := range jobs {
                out, err := validate(j.path)
                if err != nil && isFatal(err) {
                    cancel()
                }
                results <- result{index: j.index, output: out, err: err}
            }
        }()
    }

    go func() {
        for i, p := range paths {
            select {
            case jobs <- job{index: i, path: p}:
            case <-ctx.Done():
                close(jobs)
                return
            }
        }
        close(jobs)
    }()

    go func() {
        wg.Wait()
        close(results)
    }()

    return collectInOrder(results)
}

The jobs and results lines are the channel declarations; the three go func() { ... }() blocks are the goroutine launches, one pool of workers, one feeder, one closer. select { case jobs <- job{...}: ... case <-ctx.Done(): ... } is how the feeder stays responsive to cancellation even while trying to send a job that a worker isn't ready to receive yet, instead of blocking forever on a full channel after a fatal error.

This gives parallelism, ordered output, and fast failure without deadlocks.

Approach

I would read the file line by line with bufio.Scanner. Opening errors are fatal, because there is no file to process. Line parse errors are recoverable, meaning a bad line does not stop the whole file, so I return them in a slice and keep the good records. If scanning itself fails, I return the partial data plus the scan error so the caller can decide whether to fail the job.

package main

import (
    "bufio"
    "fmt"
    "os"
    "strconv"
    "strings"
)

type Record struct {
    Name string
    Age  int
}

type ParseIssue struct {
    Line int
    Raw  string
    Err  error
}

func ParseFile(path string) ([]Record, []ParseIssue, error) {
    f, err := os.Open(path)
    if err != nil {
        return nil, nil, err
    }
    defer f.Close()

    scanner := bufio.NewScanner(f)
    scanner.Buffer(make([]byte, 1024), 1024*1024)

    var records []Record
    var issues []ParseIssue
    lineNum := 0

    for scanner.Scan() {
        lineNum++
        line := strings.TrimSpace(scanner.Text())
        if line == "" {
            continue
        }

        parts := strings.Split(line, ",")
        if len(parts) != 2 {
            issues = append(issues, ParseIssue{Line: lineNum, Raw: line, Err: fmt.Errorf("expected name,age")})
            continue
        }

        age, err := strconv.Atoi(strings.TrimSpace(parts[1]))
        if err != nil {
            issues = append(issues, ParseIssue{Line: lineNum, Raw: line, Err: err})
            continue
        }

        records = append(records, Record{
            Name: strings.TrimSpace(parts[0]),
            Age:  age,
        })
    }

    if err := scanner.Err(); err != nil {
        return records, issues, err
    }

    return records, issues, nil
}

Key points

• open failure: return immediately
• parsing failure: collect issue and continue
• scan failure: return partial results and an error

Example: with Ann,32, bad-line, and Bob,41, you get 2 records and 1 issue.

Complexity: O(n) time, O(k) memory for stored records and issues.

The Global Interpreter Lock (GIL) is a mutex in CPython that ensures only one native thread executes Python bytecode at a time. It simplifies memory management but serializes CPU-bound Python code.

Workloads where threads still help despite the GIL:

• I/O-bound workloads: threads waiting on network, disk, or database I/O release the GIL during blocking calls, improving throughput (e.g., web crawlers, concurrent HTTP clients). Concretely, verified on CPython 3.12: two threads that each print, sleep (standing in for a blocking network call), then print again:

import threading, time

order = []

def worker(name, delay):
    order.append(f"{name} start")
    time.sleep(delay)      # GIL released for the duration of the sleep
    order.append(f"{name} done")

t1 = threading.Thread(target=worker, args=("A", 0.1))
t2 = threading.Thread(target=worker, args=("B", 0.1))
t1.start(); t2.start()
t1.join(); t2.join()
print(order)

A representative run printed ['A start', 'B start', 'B done', 'A done']: both start entries appear before either done, which is only possible because thread A released the GIL during its time.sleep(0.1), letting thread B start and run its own code during that wait, rather than A blocking B out entirely the way a genuine CPU-bound loop with no sleep/I-O would (there, the two threads would still only ever have one of them executing Python bytecode at a time, and the interleaving above would not occur). The exact order of the two done entries is not guaranteed (it depends on OS scheduling), but the invariant that matters, both starts landing before either finish, reliably demonstrates the GIL being released during the blocking wait, not a specific timing.

• Native-code parallelism: when threads call C extensions that release the GIL (numpy heavy computations, image processing libraries, cryptography), CPU work runs in parallel across cores.

For CPU-bound pure-Python tasks use multiprocessing or native extensions to achieve parallelism.