Python Programming Questions

Comparison:

• Method A (single-threaded generator): simple, low overhead, minimal memory, deterministic ordering, easy error handling; limited by single-core CPU and Python parsing speed.
• Method B (multi-process byte-range split): parallel CPU usage, faster parse throughput on multi-core, but more complex (must align splits to line boundaries), higher peak memory (multiple workers), and increased I/O contention.

Method A, concretely (the whole method really is this short, which is part of why it is the default):

import json

def read_jsonl(path):
    with open(path, 'r', encoding='utf-8') as f:
        for line in f:
            yield json.loads(line)

Method B, concretely, with real byte offsets so 'split misalignment' can be traced rather than only named: take a tiny 40-byte JSON-lines file, four records of exactly 10 bytes each, {"id": 1}\n{"id": 2}\n{"id": 3}\n{"id": 4}\n (bytes 0-9 are line 1, 10-19 line 2, 20-29 line 3, 30-39 line 4). Splitting this file evenly at byte 20 happens to land exactly on a line boundary, so nothing goes wrong there. Splitting unevenly, worker 0 gets raw bytes [0, 15), worker 1 gets [15, 40), lands worker 1's start offset at byte 15, verified to be the : character in the middle of line 2, not the start of any record:

def worker_range(path, start, end):
    with open(path, 'rb') as f:
        f.seek(start)
        if start != 0:
            f.readline()          # discard the partial line landed on, if any
        while f.tell() < end:
            line = f.readline()
            if not line:
                break
            yield json.loads(line)

Worker 1 seeks to byte 15 (mid-line-2), then its f.readline() reads and discards the remainder of line 2 (bytes 15-19, ': 2}\n'), landing the cursor exactly at byte 20, the true start of line 3; from there it reads whole lines normally. The other half of line 2 (bytes 10-15) was already skipped by worker 0, which stopped consuming at byte 20 without seeing it either, so line 2 as a whole is silently dropped. This is exactly the alignment bug: each worker must seek to (and discard up to) the next newline after its own raw start offset, or a record straddling a boundary is lost or corrupted.

When B is preferable:

• CPU-bound parsing (complex JSON) and available cores/IO bandwidth
• File stored on fast SSD or networked storage that supports concurrent reads

Pitfalls & failure modes:

• Split misalignment: worker starts mid-line: must seek to next newline
• Memory pressure: many workers each allocate buffers; can OOM
• Ordering: results may be out-of-order; need merging if order matters
• Partial line encoding issues (multibyte UTF-8 boundaries: UTF-8 is a variable-width encoding where one character can take anywhere from 1 to 4 bytes, so a byte-offset split chosen without regard to character boundaries can land in the middle of a multi-byte character, not just in the middle of a line; a correct implementation aligns on a newline byte, which is always a single, unambiguous ASCII byte and never appears as a continuation byte inside a multi-byte UTF-8 character, rather than aligning on an arbitrary byte count)
• Error recovery complexity: a worker crash loses its chunk

Guidelines: choose worker count to match CPU/IO balance, implement safe split alignment, stream results via queues, and fallback to single-threaded if resources constrained.

1) Multiprocessing (multiprocessing module)

• Benefit: sidesteps GIL (Global Interpreter Lock: a lock inside CPython that only lets one thread execute Python bytecode at a time, which is why adding more threads alone does not speed up pure-Python CPU-bound work) by using multiple processes; good for CPU-bound tasks that can be partitioned.
• Limitations: IPC and data serialization overhead, higher memory usage per process.
• When: embarrassingly parallel workloads (map-style), batch processing across cores.

2) C-accelerated libraries (NumPy / vectorization)

• Benefit: move heavy numeric loops into C (SIMD, Single Instruction Multiple Data, a CPU feature applying one operation to many values in one step; contiguous memory), drastically faster for array math.
• Limitations: requires expressing work in array form; not helpful for complex control flow.
• When: numerical computations over arrays, matrix ops.

3) Just-in-time / ahead-of-time compilation (Numba or Cython)

• Numba: JIT-compiles (just-in-time compiles: instead of compiling ahead of time before the program ever runs, the function is compiled straight to machine code the first time it is actually called) Python functions to machine code with little code changes; excellent for loops on numeric data.
  • Benefit: low development cost, big speedups.
  • Limitation: not all Python features supported; first-call compilation overhead.
• Cython: compile Python to C for maximum control and speed; can produce greatest gains but requires typing and build step.
  • When: algorithmic hotspots that remain after vectorization or need fine-grained control.

A concrete contrast (real values, not timing, since exact speed depends on hardware and is not something to hardcode here):

import numpy as np

def squares_loop(n):
    return [i * i for i in range(n)]

def squares_vectorized(n):
    return np.arange(n) ** 2

print(squares_loop(5))
# [0, 1, 4, 9, 16]
print(squares_vectorized(5))
# [ 0  1  4  9 16]

Both produce the identical five values; squares_loop pays Python's per-element interpreter overhead five separate times, while squares_vectorized dispatches once into a single compiled C loop, the same mechanism that makes strategy 2 (vectorization) and strategy 3 (Numba/Cython compilation) both faster than a plain interpreted loop: removing per-element interpreter overhead, not changing the underlying arithmetic.

Choose based on workload: use vectorization first, then Numba for loop-heavy numeric code, and multiprocessing when parallelism across cores is needed and data can be partitioned.

Tests and benchmarks:

• Correctness tests: unit tests comparing NumPy reference outputs vs Numba outputs on varied inputs including edge cases (NaN, infinities, zero-length arrays, non-contiguous/strided arrays). Use property-based tests (the hypothesis library: instead of writing individual test inputs by hand, you describe a rule the output must always satisfy, and the library generates many random inputs trying to break that rule) for broader coverage.
• Numerical accuracy: assert tolerances (rtol/atol, relative and absolute tolerance) for floating ops, using np.testing.assert_allclose rather than exact equality, since compiled and interpreted floating-point code can differ in the last few bits even when both are correct.
• Performance benchmarks: microbenchmarks measuring time and compilation overhead (first call vs steady-state) using asv (airspeed velocity, a benchmarking tool that tracks a function's performance across commits, so a regression shows up tied to a specific commit rather than just "it got slower at some point") or pytest-benchmark.

A minimal correctness check, concretely (verified on CPython 3.12; the pattern below is the standard, documented Numba usage, whether or not numba happens to be installed in the environment reading this):

import numpy as np
from numba import njit

def f_numpy(x):
    return np.sqrt(x ** 2 + 1)

@njit
def f_numba(x):
    out = np.empty_like(x)
    for i in range(x.shape[0]):
        out[i] = (x[i] ** 2 + 1) ** 0.5
    return out

x = np.array([0.0, 1.0, 2.0, 3.0])
np.testing.assert_allclose(f_numpy(x), f_numba(x))
# passes silently if the two implementations agree within tolerance;
# assert_allclose raises AssertionError with the offending values if they don't

This is the shape every correctness test in this answer means: compute the same thing both ways, on real (including edge-case) input, and assert they agree, rather than eyeballing the code and assuming they do.

Gotchas & how to test:

• Compilation overhead: measure both cold (first call, which includes the one-time cost of compiling the function for that argument type) and hot runs (every call after, which reuse the already-compiled machine code); prefer caching compiled functions when appropriate.
• Unsupported Python features: ensure no reliance on Python objects in hot loops; test for object-mode fallback by enabling nopython=True in tests and catching TypingError. Numba's default @njit mode (short for nopython=True) compiles the whole function straight to machine code with no Python-object fallback; if some part of the function cannot be compiled that way, @njit raises TypingError immediately rather than silently degrading, which is exactly what you want in tests: a raised TypingError tells you precisely which line Numba could not compile, instead of the function quietly running slow through the interpreter (the older, more permissive default mode's "object-mode fallback").
• ABI/dtype differences: test with different dtypes and memory layouts (C/F contiguous, non-contiguous views). ABI (application binary interface) is the compiled-code calling convention two pieces of native code, here a NumPy build and a Numba-compiled function, must agree on to call each other correctly; a mismatch is rare but shows up as crashes or garbage values rather than a clean error. C-contiguous means an array's elements sit one after another in memory in row-major order (the default for a freshly created array); F-contiguous (Fortran-contiguous) means the same but column-major; a strided or non-contiguous view (e.g. arr[::2]) is neither, and a Numba function compiled assuming one layout can behave incorrectly or need an implicit, costly copy when handed the other, which is exactly why testing with a strided view as input matters, not just a fresh contiguous array.
• Thread-safety: test with multithreaded calls if using parallel=True; check for race conditions.

Automation: integrate benchmarks in CI (nightly) and gate correctness on PRs; document performance expectations and maintain regression alerts.

Vectorized mask for rows containing any NaN:

import numpy as np

# a is shape (N, M)
mask = np.isnan(a).any(axis=1)

Explanation:

• np.isnan(a) produces boolean array shape (N, M).
• .any(axis=1) reduces per row to True if any element is NaN. A numpy axis names which direction a reduction collapses across for a 2D array: axis=0 collapses DOWN each column (one result per column), axis=1 collapses ACROSS each row (one result per row); since the question asks for a per-row answer, one True/False per row of the (N, M) array, axis=1 is the one that produces exactly N results, matching the number of rows.

Worked example, verified on CPython 3.12:

a = np.array([[1.0, 2.0], [np.nan, 4.0], [5.0, 6.0]])
print(np.isnan(a))
# [[False False]
#  [ True False]
#  [False False]]
print(np.isnan(a).any(axis=1))
# [False  True False]

Row 0 ([1.0, 2.0]) has no NaN, so it reduces to False; row 1 ([nan, 4.0]) has one, so it reduces to True; row 2 ([5.0, 6.0]) has none, so False again, exactly the boolean mask a caller would use to filter out or flag the bad row.

This is fully vectorized, memory-efficient (single boolean array) and avoids Python loops. For very large arrays consider processing in chunks to limit peak memory.

Solution (Pandas): use pivot_table taking last value for duplicates and use efficient grouping to limit memory.

import pandas as pd

df = pd.DataFrame({
    "user": ["u1", "u1", "u1", "u2", "u2"],
    "metric": ["score", "score", "clicks", "score", "clicks"],
    "value": [10, 15, 3, 20, 7],
})

# pivot using last observation
wide = df.pivot_table(index='user', columns='metric', values='value', aggfunc='last')
# optional: reset index
wide = wide.reset_index()

Worked example, verified with pandas on CPython 3.12, including a genuine duplicate (user, metric) pair so the tie-breaking behavior is visible:

df = pd.DataFrame({
    "user": ["u1", "u1", "u1", "u2", "u2"],
    "metric": ["score", "score", "clicks", "score", "clicks"],
    "value":  [10,     15,      3,       20,     7],
})
print(df)
wide = df.pivot_table(index='user', columns='metric', values='value', aggfunc='last').reset_index()
print(wide)

Input:

  user  metric  value
0   u1   score     10
1   u1   score     15
2   u1  clicks      3
3   u2   score     20
4   u2  clicks      7

Output:

metric user  clicks  score
0        u1       3      15
1        u2       7      20

u1 has two score rows (10 then 15); aggfunc='last' keeps whichever one appears LAST in df's row order, 15, and silently drops 10. This is the actual behavior the question asks about: the row order of the input DataFrame determines which duplicate value survives, so if "last" is meant to be "most recent by time" rather than "however the rows happened to arrive," the DataFrame must be sorted by a timestamp column first, pivot_table itself has no notion of time, only of row order.

Efficient for large data:

• If df is very large, pre-sort so last() works correctly: df.sort_values(['user','metric','timestamp'], inplace=True) then drop_duplicates keeping='last' then pivot.

# memory-friendly: deduplicate then pivot
df = pd.DataFrame({
    "user":      ["u1", "u1", "u1", "u2", "u2"],
    "metric":    ["score", "score", "clicks", "score", "clicks"],
    "value":     [10, 15, 3, 20, 7],
    "timestamp": pd.to_datetime([
        "2024-01-01", "2024-01-03", "2024-01-02", "2024-01-01", "2024-01-03"
    ]),
})
df2 = df.sort_values(['user','metric','timestamp']).drop_duplicates(['user','metric'], keep='last')
wide = df2.pivot(index='user', columns='metric', values='value').reset_index()

This second code block is an ALTERNATIVE to the pivot_table(aggfunc='last') call above for large data, not a required second step after it: pivot_table with aggfunc='last' already handles duplicates correctly on its own, this version exists purely because pre-deduplicating (drop_duplicates) before a plain pivot (which requires already-unique index/column pairs and raises otherwise) can be cheaper at scale than letting pivot_table do the deduplication and aggregation together internally.

Notes: drop_duplicates reduces rows before pivoting to lower memory. Use categorical for 'metric' to reduce memory. For extreme scale, use Dask DataFrame (a library that mirrors the Pandas API but splits data into partitions and runs the same operations, including pivot-style reshaping, across them in parallel) with same operations, once the table genuinely no longer fits on one machine; plain Pandas, as shown above, is enough below that point.