Python Programming Questions

Experiment design
Compare three functions in a tight loop: (A) raise/catch exception on error, (B) return error code and check, (C) pre-validated path (no error). Use timeit and large N.

Code:

import timeit

def raise_path(n):
    for i in range(n):
        try:
            if i%100==0: raise ValueError
        except ValueError:
            pass

def errorcode_path(n):
    for i in range(n):
        ok = True            # the normal, no-error case
        if i%100==0: ok = False   # the simulated error case
        if not ok: pass

def prevalidated_path(n):
    for i in range(n):
        pass                 # no error branch at all: the baseline

n=1000000
print(timeit.timeit(lambda: raise_path(n), number=3))
print(timeit.timeit(lambda: errorcode_path(n), number=3))
print(timeit.timeit(lambda: prevalidated_path(n), number=3))

A correction to the code above (a genuine bug, not just a style choice): as originally sketched, errorcode_path set ok = False unconditionally at the top of every iteration and never set it back to True on the normal path, so if not ok: was True on every single iteration, not just the 1-in-100 simulated errors. That does not model "check an error code" at all, it just runs the pass branch every time. The fix is the one shown above: ok = True by default (the common, no-error case), flipped to False only on the simulated error, so errorcode_path and raise_path are actually testing the same 1% error frequency against each other. The third function, prevalidated_path, is also added here: it has no error branch whatsoever, and exists specifically as the baseline "cost of the loop itself, with no error handling of any kind" that the question's three-way comparison (raise/catch, return-code, pre-validated) asks for; the original sketch defined only the first two.

Interpretation

• Run this at varying simulated error frequencies (change i%100==0 to i%1==0 for 100% errors, or remove the if entirely for 0%) rather than trusting a single frequency; the shape of the gap between raise_path and errorcode_path as a function of error frequency is the actual finding, a single absolute number from one machine is not portable or reproducible elsewhere and should not be reported as "the" result. prevalidated_path's time is the floor: the cost of iterating n times with no error-handling machinery of any kind, useful as the baseline both other paths are measured against.
• The mechanism, not a specific number, is what to lead with: constructing and raising a real exception involves allocating an exception object, populating a traceback, and unwinding the stack to find a matching except, real, nonzero work that a plain boolean check never does; at 0% error frequency, the try block itself still costs something (setting up the exception-handling frame) even though nothing is ever raised, so raise_path at 0% errors is a fair comparison of that baseline try-frame overhead against errorcode_path's baseline if overhead. As error frequency rises toward 100%, the actual raise/unwind cost starts to dominate raise_path's total time in a way it never does for errorcode_path, which does the same constant amount of work (ok = True, one comparison) whether or not that iteration's error flag ends up True.
• For production ETL, prefer error-code or pre-validation for expected, frequent errors, reserve exceptions for truly exceptional control flow, situations that are rare enough that even a real per-raise cost barely matters in aggregate, and where the cleaner control-flow and forced handling (you cannot silently ignore a raised exception the way an unchecked error code can be ignored) outweigh that cost.
• Profile memory and CPU (not just the timeit total) to ensure GC or traceback construction isn't dominating the comparison for reasons unrelated to the actual branch being tested, for example, a test harness that also does unrelated allocation inside the timed loop.

Comparison:

• Method A (single-threaded generator): simple, low overhead, minimal memory, deterministic ordering, easy error handling; limited by single-core CPU and Python parsing speed.
• Method B (multi-process byte-range split): parallel CPU usage, faster parse throughput on multi-core, but more complex (must align splits to line boundaries), higher peak memory (multiple workers), and increased I/O contention.

Method A, concretely (the whole method really is this short, which is part of why it is the default):

import json

def read_jsonl(path):
    with open(path, 'r', encoding='utf-8') as f:
        for line in f:
            yield json.loads(line)

Method B, concretely, with real byte offsets so 'split misalignment' can be traced rather than only named: take a tiny 40-byte JSON-lines file, four records of exactly 10 bytes each, {"id": 1}\n{"id": 2}\n{"id": 3}\n{"id": 4}\n (bytes 0-9 are line 1, 10-19 line 2, 20-29 line 3, 30-39 line 4). Splitting this file evenly at byte 20 happens to land exactly on a line boundary, so nothing goes wrong there. Splitting unevenly, worker 0 gets raw bytes [0, 15), worker 1 gets [15, 40), lands worker 1's start offset at byte 15, verified to be the : character in the middle of line 2, not the start of any record:

def worker_range(path, start, end):
    with open(path, 'rb') as f:
        f.seek(start)
        if start != 0:
            f.readline()          # discard the partial line landed on, if any
        while f.tell() < end:
            line = f.readline()
            if not line:
                break
            yield json.loads(line)

Worker 1 seeks to byte 15 (mid-line-2), then its f.readline() reads and discards the remainder of line 2 (bytes 15-19, ': 2}\n'), landing the cursor exactly at byte 20, the true start of line 3; from there it reads whole lines normally. The other half of line 2 (bytes 10-15) was already skipped by worker 0, which stopped consuming at byte 20 without seeing it either, so line 2 as a whole is silently dropped. This is exactly the alignment bug: each worker must seek to (and discard up to) the next newline after its own raw start offset, or a record straddling a boundary is lost or corrupted.

When B is preferable:

• CPU-bound parsing (complex JSON) and available cores/IO bandwidth
• File stored on fast SSD or networked storage that supports concurrent reads

Pitfalls & failure modes:

• Split misalignment: worker starts mid-line: must seek to next newline
• Memory pressure: many workers each allocate buffers; can OOM
• Ordering: results may be out-of-order; need merging if order matters
• Partial line encoding issues (multibyte UTF-8 boundaries: UTF-8 is a variable-width encoding where one character can take anywhere from 1 to 4 bytes, so a byte-offset split chosen without regard to character boundaries can land in the middle of a multi-byte character, not just in the middle of a line; a correct implementation aligns on a newline byte, which is always a single, unambiguous ASCII byte and never appears as a continuation byte inside a multi-byte UTF-8 character, rather than aligning on an arbitrary byte count)
• Error recovery complexity: a worker crash loses its chunk

Guidelines: choose worker count to match CPU/IO balance, implement safe split alignment, stream results via queues, and fallback to single-threaded if resources constrained.

Solution (Pandas): use pivot_table taking last value for duplicates and use efficient grouping to limit memory.

import pandas as pd

df = pd.DataFrame({
    "user": ["u1", "u1", "u1", "u2", "u2"],
    "metric": ["score", "score", "clicks", "score", "clicks"],
    "value": [10, 15, 3, 20, 7],
})

# pivot using last observation
wide = df.pivot_table(index='user', columns='metric', values='value', aggfunc='last')
# optional: reset index
wide = wide.reset_index()

Worked example, verified with pandas on CPython 3.12, including a genuine duplicate (user, metric) pair so the tie-breaking behavior is visible:

df = pd.DataFrame({
    "user": ["u1", "u1", "u1", "u2", "u2"],
    "metric": ["score", "score", "clicks", "score", "clicks"],
    "value":  [10,     15,      3,       20,     7],
})
print(df)
wide = df.pivot_table(index='user', columns='metric', values='value', aggfunc='last').reset_index()
print(wide)

Input:

  user  metric  value
0   u1   score     10
1   u1   score     15
2   u1  clicks      3
3   u2   score     20
4   u2  clicks      7

Output:

metric user  clicks  score
0        u1       3      15
1        u2       7      20

u1 has two score rows (10 then 15); aggfunc='last' keeps whichever one appears LAST in df's row order, 15, and silently drops 10. This is the actual behavior the question asks about: the row order of the input DataFrame determines which duplicate value survives, so if "last" is meant to be "most recent by time" rather than "however the rows happened to arrive," the DataFrame must be sorted by a timestamp column first, pivot_table itself has no notion of time, only of row order.

Efficient for large data:

• If df is very large, pre-sort so last() works correctly: df.sort_values(['user','metric','timestamp'], inplace=True) then drop_duplicates keeping='last' then pivot.

# memory-friendly: deduplicate then pivot
df = pd.DataFrame({
    "user":      ["u1", "u1", "u1", "u2", "u2"],
    "metric":    ["score", "score", "clicks", "score", "clicks"],
    "value":     [10, 15, 3, 20, 7],
    "timestamp": pd.to_datetime([
        "2024-01-01", "2024-01-03", "2024-01-02", "2024-01-01", "2024-01-03"
    ]),
})
df2 = df.sort_values(['user','metric','timestamp']).drop_duplicates(['user','metric'], keep='last')
wide = df2.pivot(index='user', columns='metric', values='value').reset_index()

This second code block is an ALTERNATIVE to the pivot_table(aggfunc='last') call above for large data, not a required second step after it: pivot_table with aggfunc='last' already handles duplicates correctly on its own, this version exists purely because pre-deduplicating (drop_duplicates) before a plain pivot (which requires already-unique index/column pairs and raises otherwise) can be cheaper at scale than letting pivot_table do the deduplication and aggregation together internally.

Notes: drop_duplicates reduces rows before pivoting to lower memory. Use categorical for 'metric' to reduce memory. For extreme scale, use Dask DataFrame (a library that mirrors the Pandas API but splits data into partitions and runs the same operations, including pivot-style reshaping, across them in parallel) with same operations, once the table genuinely no longer fits on one machine; plain Pandas, as shown above, is enough below that point.

Goal: compute rank within each 'category' in descending order efficiently.

Solution:

import pandas as pd

# assuming df has columns ['category', 'value']
df = pd.DataFrame({
    "category": ["A", "A", "A", "B", "B"],
    "value":    [10,  10,  8,   5,   5],
})

# use groupby + rank with method='dense' or 'first' depending on tie behavior
df['rank_within_group'] = df.groupby('category')['value']\
    .rank(method='dense', ascending=False).astype(int)

method='dense' is the crux of the tie-breaking behavior here: it means tied values SHARE a rank, and the next distinct value continues at the very next rank number with no gap. That is different from method='first' (ties broken by row order, so no two rows ever share a rank) and from the default method='average' (ties get the average of the ranks they would have spanned). For descending values like 10, 10, 8, dense ranks them 1, 1, 2, not 1, 2, 3 (that would be method='first' or 'min'-adjacent behavior) and not 1.5, 1.5, 3 (that would be the average default).

Worked example, verified with pandas on CPython 3.12, including a genuine tie so the behavior is visible rather than only named:

df = pd.DataFrame({
    "category": ["A", "A", "A", "B", "B"],
    "value":    [10,  10,  8,   5,   5],
})
df['rank_within_group'] = df.groupby('category')['value']\
    .rank(method='dense', ascending=False).astype(int)
print(df)

Output:

  category  value  rank_within_group
0        A     10                  1
1        A     10                  1
2        A      8                  2
3        B      5                  1
4        B      5                  1

Category A's two tied 10s both get rank 1 (the highest, since ascending=False), and the next distinct value, 8, gets rank 2 immediately, no rank 3 is skipped to "make room" for the tie. Category B's two tied 5s both get rank 1, since within group B, 5 is the only (and therefore highest) value present.

Performance & memory:

• groupby.rank is implemented in C and avoids Python-level loops; it's fast and memory-efficient compared to apply.
• For millions of rows, ensure 'category' is categorical to save memory and speed grouping:

df['category'] = df['category'].astype('category')

• If memory is tight, process in chunks by categories: iterate over df.groupby('category') or sort by category and value and rank per chunk.

Trade-offs:

• In-memory groupby requires holding column arrays; converting to categorical reduces memory but adds preprocessing cost.
• Chunking reduces peak RAM but loses single-call simplicity and may require external sorting.

Definitions:

• Shallow copy: creates a new container but inserts references to the same nested objects.
• Deep copy: recursively copies nested objects so modifications don't affect the original.

Example:

import copy
orig = {'nums': [1, 2]}
sh = copy.copy(orig)
dp = copy.deepcopy(orig)

sh['nums'].append(3)
print(orig)  # {'nums': [1,2,3]} -> changed via shallow copy

dp['nums'].append(4)
print(orig)  # unchanged by deep copy

When to use: shallow copy is fine for immutable nested objects or when shared mutation is intended; deep copy when you need independent mutable nested structures. Be cautious: deep copy can be expensive and may fail for unpicklable objects.