Python Programming Questions

Experiment design
Compare three functions in a tight loop: (A) raise/catch exception on error, (B) return error code and check, (C) pre-validated path (no error). Use timeit and large N.

Code:

import timeit

def raise_path(n):
    for i in range(n):
        try:
            if i%100==0: raise ValueError
        except ValueError:
            pass

def errorcode_path(n):
    for i in range(n):
        ok = True            # the normal, no-error case
        if i%100==0: ok = False   # the simulated error case
        if not ok: pass

def prevalidated_path(n):
    for i in range(n):
        pass                 # no error branch at all: the baseline

n=1000000
print(timeit.timeit(lambda: raise_path(n), number=3))
print(timeit.timeit(lambda: errorcode_path(n), number=3))
print(timeit.timeit(lambda: prevalidated_path(n), number=3))

A correction to the code above (a genuine bug, not just a style choice): as originally sketched, errorcode_path set ok = False unconditionally at the top of every iteration and never set it back to True on the normal path, so if not ok: was True on every single iteration, not just the 1-in-100 simulated errors. That does not model "check an error code" at all, it just runs the pass branch every time. The fix is the one shown above: ok = True by default (the common, no-error case), flipped to False only on the simulated error, so errorcode_path and raise_path are actually testing the same 1% error frequency against each other. The third function, prevalidated_path, is also added here: it has no error branch whatsoever, and exists specifically as the baseline "cost of the loop itself, with no error handling of any kind" that the question's three-way comparison (raise/catch, return-code, pre-validated) asks for; the original sketch defined only the first two.

Interpretation

• Run this at varying simulated error frequencies (change i%100==0 to i%1==0 for 100% errors, or remove the if entirely for 0%) rather than trusting a single frequency; the shape of the gap between raise_path and errorcode_path as a function of error frequency is the actual finding, a single absolute number from one machine is not portable or reproducible elsewhere and should not be reported as "the" result. prevalidated_path's time is the floor: the cost of iterating n times with no error-handling machinery of any kind, useful as the baseline both other paths are measured against.
• The mechanism, not a specific number, is what to lead with: constructing and raising a real exception involves allocating an exception object, populating a traceback, and unwinding the stack to find a matching except, real, nonzero work that a plain boolean check never does; at 0% error frequency, the try block itself still costs something (setting up the exception-handling frame) even though nothing is ever raised, so raise_path at 0% errors is a fair comparison of that baseline try-frame overhead against errorcode_path's baseline if overhead. As error frequency rises toward 100%, the actual raise/unwind cost starts to dominate raise_path's total time in a way it never does for errorcode_path, which does the same constant amount of work (ok = True, one comparison) whether or not that iteration's error flag ends up True.
• For production ETL, prefer error-code or pre-validation for expected, frequent errors, reserve exceptions for truly exceptional control flow, situations that are rare enough that even a real per-raise cost barely matters in aggregate, and where the cleaner control-flow and forced handling (you cannot silently ignore a raised exception the way an unchecked error code can be ignored) outweigh that cost.
• Profile memory and CPU (not just the timeit total) to ensure GC or traceback construction isn't dominating the comparison for reasons unrelated to the actual branch being tested, for example, a test harness that also does unrelated allocation inside the timed loop.

Approach: implement LRU with a dict for storage and a doubly-linked list for order; use threading.RLock (a lock that the SAME thread can safely acquire again without deadlocking itself, unlike a plain threading.Lock, in case the cache logic ever needs to re-enter the lock while already holding it) for concurrency. Decorator returns wrapper that locks around lookups and updates, minimizing lock hold time.

Implementation:

import threading
from functools import wraps

def lru_cache(maxsize=128):
    def deco(func):
        cache = {}
        head = tail = None
        lock = threading.RLock()
        class Node:
            __slots__=('key','val','prev','next')
            def __init__(self,k,v):
                self.key=k;self.val=v;self.prev=self.next=None
        def _move_to_front(node):
            nonlocal head, tail
            if node is head:
                return  # already the most-recently-used entry, nothing to do
            # unlink node from wherever it currently sits
            if node.prev:
                node.prev.next = node.next
            if node.next:
                node.next.prev = node.prev
            if node is tail:
                tail = node.prev
            # relink it at the head (the most-recently-used end)
            node.prev = None
            node.next = head
            if head:
                head.prev = node
            head = node
            if tail is None:
                tail = node
        @wraps(func)
        def wrapper(*args, **kwargs):
            nonlocal head, tail   # wrapper reassigns both below on eviction; without
                                   # this declaration LEGB makes them locals instead
            key=(args,tuple(sorted(kwargs.items())))
            with lock:
                node=cache.get(key)
                if node:
                    _move_to_front(node); return node.val
            val=func(*args, **kwargs)
            with lock:
                if key in cache: return cache[key].val
                node=Node(key,val); cache[key]=node; _move_to_front(node)
                if len(cache)>maxsize:
                    # cache is over capacity: evict the true tail, the
                    # least-recently-used entry, from both the linked list
                    # and the dict
                    lru_node = tail
                    tail = lru_node.prev
                    if tail:
                        tail.next = None
                    else:
                        head = None
                    del cache[lru_node.key]
            return val
        return wrapper
    return deco

Worked example: verified on CPython 3.12, calling the decorated function through a full eviction cycle so the policy can actually be watched working, not just taken on faith.

calls = []

@lru_cache(maxsize=2)
def square(n):
    calls.append(n)
    return n * n

print(square(1))   # 1  (miss: cache empty, computed and cached; order (MRU->LRU): [1])
print(square(2))   # 4  (miss: computed and cached; order: [2, 1])
print(square(1))   # 1  (hit: served from cache, 1 moves back to the front; order: [1, 2])
print(square(3))   # 9  (miss: cache was full at {1, 2}; since 1 was just reused, 2 is now
                    # the least-recently-used entry and gets evicted to make room for 3; order: [3, 1])
print(square(2))   # 4  (miss again: 2 was evicted in the previous step, so this recomputes
                    # instead of hitting the cache; order: [2, 3])
print(calls)        # [1, 2, 3, 2]  -- 2 appears twice: proof it was actually evicted and
                     # had to be recomputed, not just a claim

Complexity: O(1) average get/set.

Contention: a single global lock, one shared threading.RLock protecting the whole cache, serializes cache access: only one thread at a time can even check whether something is cached, regardless of which key it wants, which becomes a bottleneck under heavy concurrent traffic. Two ways to reduce that: a read-mostly optimistic check (read the dict for a hit without taking the lock first, since a plain dict read is safe to race on for a snapshot lookup, and only take the lock to confirm the hit and update the linked-list ordering, so the common cache-hit path spends less time holding the lock), or shard locks (split one cache into several smaller caches, each with its own separate lock, and route each key to one shard by hashing it, e.g. shard = hash(key) % num_shards; two threads reading keys that land in different shards no longer contend for the same lock at all, at the cost of maxsize now being enforced per shard rather than globally).

Profiling plan for pandas.apply hotspot:

1. Confirm hotspot: run a coarse profiler (cProfile) to see time spent in apply.

• python -m cProfile -o prof.out script.py
• snakeviz prof.out (a browser-based visualizer for cProfile output) or pstats to inspect

2. Line-level: use line_profiler (pip install line_profiler) and add @profile to the function passed to apply or use kernprof:

• kernprof -l -v script.py
  This shows which lines inside the applied function are expensive.

A concrete pass with real, reproducible numbers (call counts, not timing, since exact seconds are hardware-dependent and not something to hardcode here), verified with pandas on CPython 3.12:

import cProfile, pstats, io
import pandas as pd

def slow_row_calc(row):
    total = 0.0
    for i in range(20):
        total += (row['x'] + i) ** 0.5
    return total

df = pd.DataFrame({'x': range(500)})

pr = cProfile.Profile()
pr.enable()
df['y'] = df.apply(slow_row_calc, axis=1)
pr.disable()

stats = pstats.Stats(pr)
print('total function calls:', stats.total_calls)
# key by function name instead of hardcoding a (filename, line, name) tuple: the
# filename/line depend on how this file happens to be invoked and are not stable
own_key = next(k for k in stats.stats if k[2] == 'slow_row_calc')
print('calls to slow_row_calc:', stats.stats[own_key][0])

On this run (pandas version-dependent in its exact figure, but always dramatically more than one call per row), this printed total function calls: 156327 and calls to slow_row_calc: 500. The second number is unsurprising, 500 rows means 500 calls to your own function, but the first is the real finding a coarse profile surfaces: .apply(..., axis=1) did not cost "500 function calls," it cost over 150,000, because pandas builds a full pandas Series object per row (with its own isinstance checks and internal bookkeeping) before your function even runs, per-row overhead invisible from just reading the code, and exactly the kind of thing a profiler exists to reveal instead of guessing at.

3. Interpret results: if most time in Python-level loops or element ops, apply is causing Python callbacks per row.

4. Optimizations:

• Replace apply with vectorized NumPy/Pandas ops or use groupby.transform.
• Use C-accelerated libraries (numexpr, which evaluates a whole array expression like a*b+c in one call without materializing each intermediate array) for heavy numeric expressions.
• If logic is complex, use numba (compiles a plain Python function to machine code the first time it runs) to JIT-compile the function and call it over NumPy arrays.
• If per-row but pure Python expensive work, consider transforming to C-extension or use multiprocessing/df.map_partitions with Dask (a library that mirrors the Pandas API but splits data into partitions and runs the same operations across them in parallel).

The vectorized replacement for the example above, confirmed to agree exactly with the .apply() version:

y_vectorized = sum((df['x'] + i) ** 0.5 for i in range(20))
import numpy as np
print(np.allclose(df['y'].to_numpy(), y_vectorized.to_numpy()))
# True

This replaces the 500 individual per-row Python-function calls (and everything pandas does to set each of them up) with 20 vectorized column-wide additions, one per term in the loop, each running as a single compiled pass over all 500 rows at once instead of 500 separate Python-level calls.

5. Validate: run profiler again to confirm reduced total call count, and add a regression test asserting the vectorized and original implementations agree (as np.allclose does above) for the critical dataset.

Result: move work from Python-level per-row to vectorized, compiled, or parallel implementations.

Approach summary
Use an atomic filesystem operation to create a lockfile and store owner PID. "Atomic" here means the operating system guarantees that checking whether the file already exists and creating it happen as a single, indivisible step: the kernel resolves any race internally, so it is impossible for two processes to both be told "it didn't exist, you just created it" for the same file. Concretely, os.open(path, O_CREAT|O_EXCL) either creates the file and returns a valid file descriptor (this process is now the owner), or, if the file already exists, raises OSError with errno.EEXIST and creates nothing, there is no window in between where a second process could sneak in and also succeed. On Unix use os.open with O_EXCL|O_CREAT; on Windows use msvcrt.locking or CreateFile with exclusive flags. Implement acquire(timeout) with retries and stale-lock detection via PID and age.

Implementation (Unix-first, cross-platform fallback)

import os, time, errno
from pathlib import Path

class FileLock:
    def __init__(self, path):
        self.path = Path(path)
    def acquire(self, timeout=10):
        end = time.time()+timeout
        while time.time()<end:
            try:
                fd = os.open(self.path, os.O_CREAT|os.O_EXCL|os.O_WRONLY)
                os.write(fd, str(os.getpid()).encode())
                os.close(fd)
                return True
            except OSError as e:
                if e.errno!=errno.EEXIST: raise
            time.sleep(0.1)
        return False
    def release(self):
        try: self.path.unlink()
        except FileNotFoundError: pass

Worked trace: two processes racing to acquire the same lock, simulated in one process for illustration (the real guarantee comes from the OS's atomic O_CREAT|O_EXCL, not from anything special in this simulation, but the sequence of return values is exactly what would happen with two real, separate processes):

lock_a = FileLock("/tmp/demo.lock")
lock_b = FileLock("/tmp/demo.lock")

print(lock_a.acquire(timeout=1))   # True: lock_a's os.open call wins the race, file now exists
print(lock_b.acquire(timeout=0.3)) # False: lock_b's os.open calls all hit EEXIST and it
                                    # gives up once the 0.3s timeout elapses
lock_a.release()
print(lock_b.acquire(timeout=1))   # True: now that lock_a released (deleted the file),
                                    # lock_b's next os.open call succeeds

The first acquire call's os.open(..., O_CREAT|O_EXCL) either wins outright (file did not exist, now it does, True) or loses outright (EEXIST, False after retries time out); there is no third outcome where both processes believe they created the file, which is exactly the race the atomic system call rules out by construction.

Platform notes

• Unix: O_EXCL is atomic across processes. Use fcntl.flock for advisory locks (a lock that only stops other processes if they also choose to check for it before touching the file; unlike a mandatory, OS-enforced lock, the OS itself does not block a process that simply ignores the lock and opens the file directly) when needed.
• Windows: O_EXCL behaves differently; prefer msvcrt.locking or pywin32 CreateFile for exclusive access.

Stale locks
Check PID from file and whether process exists; remove stale if safe. For production, add jitter, robust error handling, and optional directory-level locking for network filesystems.

Why vectorized ops are faster:

• Vectorized NumPy/Pandas operations run in optimized C/Fortran loops avoiding Python per-element overhead.
• They leverage contiguous memory, CPU cache, SIMD (Single Instruction, Multiple Data: a CPU feature that applies one instruction to several values in a single step, instead of one value at a time), and multi-threaded BLAS (Basic Linear Algebra Subprogram, a standard library of fast, hardware-tuned matrix/vector math routines that numpy calls into) for numeric work.
• Example: adding two arrays uses a single C loop vs Python loop calling millions of Python operations.

Worked example, verified on CPython 3.12 (showing what actually gets computed, not a timing benchmark: exact speed depends on hardware and array size, so run time.perf_counter() yourself to see the real gap on your own machine rather than trusting a hardcoded number here):

import numpy as np

a = np.array([1.0, 2.0, 3.0, 4.0])
b = np.array([10.0, 20.0, 30.0, 40.0])

def add_loop(a, b):
    return [x + y for x, y in zip(a, b)]

def add_vectorized(a, b):
    return a + b

print(add_loop(a, b))
# [11.0, 22.0, 33.0, 44.0]
print(add_vectorized(a, b))
# [11. 22. 33. 44.]

Both compute the identical values; add_loop does it via four separate Python-level additions, each one dispatched, type-checked, and reference-counted by the interpreter, while add_vectorized's a + b dispatches once into a single compiled C loop over the two contiguous memory buffers, which is the mechanism, not a specific timing number, that makes the vectorized version pull ahead as array size grows.

When vectorization can be slower:

• Small arrays: overhead of creating intermediate arrays and function call overhead can dominate; a simple Python loop or in-place updates may be faster.
• Complex element-wise logic with branching: vectorization may require multiple large temporaries or complex masking, increasing memory bandwidth and runtime.

Where vectorization stops winning, made concrete: a piecewise function like "if x < 0, return 0; elif x < 10, return x; else return x**2" vectorizes via np.where, but every branch has to be computed for every element before the mask selects which result to keep:

x = np.array([-3.0, 5.0, 15.0])
result = np.where(x < 0, 0.0, np.where(x < 10, x, x ** 2))
print(result)
# [  0.   5. 225.]

Even though only one branch is "true" for any given element, x ** 2 is computed for all three elements (including -3.0 and 5.0, whose squared values are simply discarded by the outer np.where), and each np.where call allocates a full temporary array the size of x. For a genuinely small array or a function with many branches, that wasted work and temporary-array allocation can cost more than a plain Python loop that only evaluates the one branch each element actually needs; this is the concrete shape of "vectorization can be slower."

Example: computing a piecewise function with many branches can be slower vectorized due to multiple masks and temporary arrays; using numba (a library that compiles a plain Python function to machine code the first time it runs, letting a genuinely branchy per-element loop run at near-C speed without vectorizing it at all) for a compiled loop may be faster and more memory-efficient.