Python Programming Questions

Experiment design
Compare three functions in a tight loop: (A) raise/catch exception on error, (B) return error code and check, (C) pre-validated path (no error). Use timeit and large N.

Code:

import timeit

def raise_path(n):
    for i in range(n):
        try:
            if i%100==0: raise ValueError
        except ValueError:
            pass

def errorcode_path(n):
    for i in range(n):
        ok = True            # the normal, no-error case
        if i%100==0: ok = False   # the simulated error case
        if not ok: pass

def prevalidated_path(n):
    for i in range(n):
        pass                 # no error branch at all: the baseline

n=1000000
print(timeit.timeit(lambda: raise_path(n), number=3))
print(timeit.timeit(lambda: errorcode_path(n), number=3))
print(timeit.timeit(lambda: prevalidated_path(n), number=3))

A correction to the code above (a genuine bug, not just a style choice): as originally sketched, errorcode_path set ok = False unconditionally at the top of every iteration and never set it back to True on the normal path, so if not ok: was True on every single iteration, not just the 1-in-100 simulated errors. That does not model "check an error code" at all, it just runs the pass branch every time. The fix is the one shown above: ok = True by default (the common, no-error case), flipped to False only on the simulated error, so errorcode_path and raise_path are actually testing the same 1% error frequency against each other. The third function, prevalidated_path, is also added here: it has no error branch whatsoever, and exists specifically as the baseline "cost of the loop itself, with no error handling of any kind" that the question's three-way comparison (raise/catch, return-code, pre-validated) asks for; the original sketch defined only the first two.

Interpretation

• Run this at varying simulated error frequencies (change i%100==0 to i%1==0 for 100% errors, or remove the if entirely for 0%) rather than trusting a single frequency; the shape of the gap between raise_path and errorcode_path as a function of error frequency is the actual finding, a single absolute number from one machine is not portable or reproducible elsewhere and should not be reported as "the" result. prevalidated_path's time is the floor: the cost of iterating n times with no error-handling machinery of any kind, useful as the baseline both other paths are measured against.
• The mechanism, not a specific number, is what to lead with: constructing and raising a real exception involves allocating an exception object, populating a traceback, and unwinding the stack to find a matching except, real, nonzero work that a plain boolean check never does; at 0% error frequency, the try block itself still costs something (setting up the exception-handling frame) even though nothing is ever raised, so raise_path at 0% errors is a fair comparison of that baseline try-frame overhead against errorcode_path's baseline if overhead. As error frequency rises toward 100%, the actual raise/unwind cost starts to dominate raise_path's total time in a way it never does for errorcode_path, which does the same constant amount of work (ok = True, one comparison) whether or not that iteration's error flag ends up True.
• For production ETL, prefer error-code or pre-validation for expected, frequent errors, reserve exceptions for truly exceptional control flow, situations that are rare enough that even a real per-raise cost barely matters in aggregate, and where the cleaner control-flow and forced handling (you cannot silently ignore a raised exception the way an unchecked error code can be ignored) outweigh that cost.
• Profile memory and CPU (not just the timeit total) to ensure GC or traceback construction isn't dominating the comparison for reasons unrelated to the actual branch being tested, for example, a test harness that also does unrelated allocation inside the timed loop.

Comparison:

• Method A (single-threaded generator): simple, low overhead, minimal memory, deterministic ordering, easy error handling; limited by single-core CPU and Python parsing speed.
• Method B (multi-process byte-range split): parallel CPU usage, faster parse throughput on multi-core, but more complex (must align splits to line boundaries), higher peak memory (multiple workers), and increased I/O contention.

Method A, concretely (the whole method really is this short, which is part of why it is the default):

import json

def read_jsonl(path):
    with open(path, 'r', encoding='utf-8') as f:
        for line in f:
            yield json.loads(line)

Method B, concretely, with real byte offsets so 'split misalignment' can be traced rather than only named: take a tiny 40-byte JSON-lines file, four records of exactly 10 bytes each, {"id": 1}\n{"id": 2}\n{"id": 3}\n{"id": 4}\n (bytes 0-9 are line 1, 10-19 line 2, 20-29 line 3, 30-39 line 4). Splitting this file evenly at byte 20 happens to land exactly on a line boundary, so nothing goes wrong there. Splitting unevenly, worker 0 gets raw bytes [0, 15), worker 1 gets [15, 40), lands worker 1's start offset at byte 15, verified to be the : character in the middle of line 2, not the start of any record:

def worker_range(path, start, end):
    with open(path, 'rb') as f:
        f.seek(start)
        if start != 0:
            f.readline()          # discard the partial line landed on, if any
        while f.tell() < end:
            line = f.readline()
            if not line:
                break
            yield json.loads(line)

Worker 1 seeks to byte 15 (mid-line-2), then its f.readline() reads and discards the remainder of line 2 (bytes 15-19, ': 2}\n'), landing the cursor exactly at byte 20, the true start of line 3; from there it reads whole lines normally. The other half of line 2 (bytes 10-15) was already skipped by worker 0, which stopped consuming at byte 20 without seeing it either, so line 2 as a whole is silently dropped. This is exactly the alignment bug: each worker must seek to (and discard up to) the next newline after its own raw start offset, or a record straddling a boundary is lost or corrupted.

When B is preferable:

• CPU-bound parsing (complex JSON) and available cores/IO bandwidth
• File stored on fast SSD or networked storage that supports concurrent reads

Pitfalls & failure modes:

• Split misalignment: worker starts mid-line: must seek to next newline
• Memory pressure: many workers each allocate buffers; can OOM
• Ordering: results may be out-of-order; need merging if order matters
• Partial line encoding issues (multibyte UTF-8 boundaries: UTF-8 is a variable-width encoding where one character can take anywhere from 1 to 4 bytes, so a byte-offset split chosen without regard to character boundaries can land in the middle of a multi-byte character, not just in the middle of a line; a correct implementation aligns on a newline byte, which is always a single, unambiguous ASCII byte and never appears as a continuation byte inside a multi-byte UTF-8 character, rather than aligning on an arbitrary byte count)
• Error recovery complexity: a worker crash loses its chunk

Guidelines: choose worker count to match CPU/IO balance, implement safe split alignment, stream results via queues, and fallback to single-threaded if resources constrained.

Situation: Overnight job reads a directory of daily status files. Some may be empty, malformed, or missing.

Task: Keep valid data moving, but tell the operator exactly what failed.

Action: I would loop over the expected file list and handle errors per file, not around the whole batch.

• If a file is missing, record missing and continue.
• If a file is empty, record empty and continue.
• If a line is malformed, skip just that file or just that line depending on the rule, and capture the filename and line number.
• Keep running totals in memory, plus a failed_files list for the final report.

Result: The job still produces a useful summary from good files, while the operator gets a clean failure section like status_2025-07-03.txt: malformed line 42 and status_2025-07-04.txt: missing. That is better than a hard stop because it keeps the daily workflow moving.

A small worked example, verified on CPython 3.12, showing the actual loop and its exact printed summary rather than only describing the shape:

def build_report(expected_files, file_contents):
    report = {"processed": 0, "total_value": 0, "failed": []}
    for name in expected_files:
        if name not in file_contents:
            report["failed"].append(f"{name}: missing")
            continue
        content = file_contents[name]
        if content.strip() == "":
            report["failed"].append(f"{name}: empty")
            continue
        try:
            value = int(content.strip())
        except ValueError:
            report["failed"].append(f"{name}: malformed line")
            continue
        report["processed"] += 1
        report["total_value"] += value
    return report

expected = [f"status_2025-07-{i:02d}.txt" for i in range(1, 11)]
contents = {name: "5" for name in expected}
contents["status_2025-07-05.txt"] = ""          # empty
del contents["status_2025-07-09.txt"]            # missing

print(build_report(expected, contents))

Output:

{'processed': 8, 'total_value': 40, 'failed': ['status_2025-07-05.txt: empty', 'status_2025-07-09.txt: missing']}

8 files processed (matching the 10 expected minus the 1 empty and 1 missing), each contributing a value of 5, giving total_value = 40; the two problem files are named explicitly in failed with the specific reason, exactly the breakdown a report shown to an operator needs.

The Global Interpreter Lock (GIL) is a mutex in CPython that ensures only one native thread executes Python bytecode at a time. It simplifies memory management but serializes CPU-bound Python code.

Workloads where threads still help despite the GIL:

• I/O-bound workloads: threads waiting on network, disk, or database I/O release the GIL during blocking calls, improving throughput (e.g., web crawlers, concurrent HTTP clients). Concretely, verified on CPython 3.12: two threads that each print, sleep (standing in for a blocking network call), then print again:

import threading, time

order = []

def worker(name, delay):
    order.append(f"{name} start")
    time.sleep(delay)      # GIL released for the duration of the sleep
    order.append(f"{name} done")

t1 = threading.Thread(target=worker, args=("A", 0.1))
t2 = threading.Thread(target=worker, args=("B", 0.1))
t1.start(); t2.start()
t1.join(); t2.join()
print(order)

A representative run printed ['A start', 'B start', 'B done', 'A done']: both start entries appear before either done, which is only possible because thread A released the GIL during its time.sleep(0.1), letting thread B start and run its own code during that wait, rather than A blocking B out entirely the way a genuine CPU-bound loop with no sleep/I-O would (there, the two threads would still only ever have one of them executing Python bytecode at a time, and the interleaving above would not occur). The exact order of the two done entries is not guaranteed (it depends on OS scheduling), but the invariant that matters, both starts landing before either finish, reliably demonstrates the GIL being released during the blocking wait, not a specific timing.

• Native-code parallelism: when threads call C extensions that release the GIL (numpy heavy computations, image processing libraries, cryptography), CPU work runs in parallel across cores.

For CPU-bound pure-Python tasks use multiprocessing or native extensions to achieve parallelism.

Approach: Use a min-heap (a min-heap is a tree-shaped data structure that always keeps its smallest element accessible at the top/root, retrievable in O(1), and restores that property in O(log n) after every push or pop; that is what lets this algorithm always grab the smallest available head-of-iterable next without rescanning every input from scratch) to pull the smallest next element from heads of each iterable. Support key by pushing (key(val), idx, val, iterator). Works with generators and streams and only holds one item per input plus heap overhead.

Implementation:

import heapq
import itertools

def merge_sorted_iterables(iters, key=None):
    key = (lambda x: x) if key is None else key
    heap = []
    for idx, it in enumerate(map(iter, iters)):
        try:
            val = next(it)
            heapq.heappush(heap, (key(val), idx, val, it))
        except StopIteration:
            continue
    while heap:
        _, idx, val, it = heapq.heappop(heap)
        yield val
        try:
            nxt = next(it)
            heapq.heappush(heap, (key(nxt), idx, nxt, it))
        except StopIteration:
            continue

Worked example, verified on CPython 3.12:

print(list(merge_sorted_iterables([[1, 4, 7], [2, 3, 9]])))
# [1, 2, 3, 4, 7, 9]

Tracing the heap: it starts holding one entry per input, (1, 0, 1, iter1) and (2, 1, 2, iter2), the smallest head of each list. Popping the smaller one yields 1 and immediately pushes that same iterator's next value, 4, back in, so the heap always holds exactly one "candidate next value" per iterable that still has values left. Popping the current minimum of that small set is always correct because every value not yet pulled from a given iterable is guaranteed to be >= the value currently sitting in the heap for it, since each input is already sorted; the full sequence of pops is 1, 2, 3, 4, 7, 9, matching the printed output exactly.

Key points:

• Memory: O(m) where m = number of input iterables (one element per iterator + heap metadata).
• Works with infinite streams and generators.
• Stable per iterator due to idx tie-breaker.

Edge cases: empty iterables, differing element types (ensure key handles them), expensive key (consider caching keys).