Python Programming Questions

Plan: collect profiles, identify roots, mitigate with code/data changes.

Collecting memory profiles:

• Use tracemalloc (a built-in module that records which line of code allocated each block of memory, and lets you snapshot total/peak Python-level allocations at any point) for Python allocations
• Use memory_profiler (@profile) (a decorator-driven tool that reports memory usage line by line for a decorated function, run via python -m memory_profiler script.py) to trace per-line memory
• Use psutil / top for process-level peaks (psutil: a library that reads OS-level process stats, including RSS, resident set size, the actual physical RAM a process is currently using, directly from the operating system)
• For NumPy, use objgraph (a tool for visualizing and counting live Python object references, useful for finding what is unexpectedly still holding a reference to something large) / np.ndarray.nbytes sampling
• Take periodic heap dumps and track peak RSS

Which to reach for first: tracemalloc is the default starting point for almost any Python-level memory investigation, cheap to turn on, built in, and line-level. psutil/RSS is the next check when the growth seems to be happening outside pure Python objects entirely (a C extension, or a numpy allocation that tracemalloc's Python-object view does not fully capture). memory_profiler and objgraph are for harder follow-up cases once the first two have narrowed down roughly where the growth is, not where most investigations should start.

A worked trace with tracemalloc, verified on CPython 3.12 with numpy: creating a float64 array and then converting it with .astype(np.float32) (a common, easy-to-miss source of an avoidable extra allocation) shows up as two distinct, measurable jumps:

import tracemalloc
import numpy as np

tracemalloc.start()
baseline, _ = tracemalloc.get_traced_memory()

a = np.zeros(1_000_000, dtype=np.float64)
after_a, _ = tracemalloc.get_traced_memory()

b = a.astype(np.float32)   # a full, avoidable copy if a float32 array was all that was ever needed
after_b, _ = tracemalloc.get_traced_memory()

print("a allocation bytes:", after_a - baseline)
print("b (astype copy) allocation bytes:", after_b - after_a)

Output:

a allocation bytes: 8000496
b (astype copy) allocation bytes: 4000096

The dominant numbers, 8,000,000 and 4,000,000, are exactly 1_000_000 * 8 bytes (float64) and 1_000_000 * 4 bytes (float32); the small remainder (496 and 96 bytes) is bookkeeping overhead from numpy's own array object and can vary slightly by numpy version, the part worth remembering is the dtype-size arithmetic, not the exact overhead constant. If a was only ever needed as float32, allocating it directly as np.zeros(1_000_000, dtype=np.float32) avoids the second, redundant 4,000,000-byte allocation entirely, this is exactly the kind of concrete before/after number a real profiling pass surfaces that a purely descriptive plan does not.

Common causes:

• Creating many temporaries (slicing / copying)
• Unnecessary copies from dtype promotion (dtype promotion: when an operation combines two arrays of different dtypes, e.g. float32 and float64, numpy silently produces a result in the wider of the two dtypes, float64, which can quietly double the memory of a computation that was only ever intended to stay in float32)
• Holding references to large arrays (accumulators, list append)
• Parallel workers duplicating memory

Mitigations:

• Use in-place operations (out=...) and views (a view: a new array object that points at the same underlying memory as another array, e.g. a[::2], versus a copy, which allocates and fills an entirely separate block of memory) where safe
• Use appropriate dtypes (float32 vs float64)
• Process data in chunks / streaming
• Reuse preallocated arrays and buffer pools
• Avoid building large Python lists; use arrays or write to disk-backed arrays (memmap)
• Release references and call gc.collect when necessary for deterministic release

Validation: write microbenchmarks for peak RSS (RSS: resident set size, the actual physical memory a process currently occupies, as reported by the OS), compare before/after; run under production-like data sizes.

Requirements & idea: Provide a lazy, composable API that records string ops and applies them in a single pass to avoid temporaries. Offer a lightweight proxy object wrapping Series with an operation pipeline executed in-place or chunked.

API sketch, with a working _apply_pipeline (the earlier sketch left this function unimplemented; here it actually runs each recorded op vectorized, once, over the whole chunk, rather than materializing an intermediate Series between every step):

class StrChain:
    def __init__(self, series):
        self.series = series
        self.ops = []
    def lower(self):
        self.ops.append(('lower', None)); return self
    def replace(self, pat, repl):
        self.ops.append(('replace', (pat, repl))); return self
    def strip(self):
        self.ops.append(('strip', None)); return self
    def compute(self, chunk_size=10_000):
        # apply ops chunk-wise to avoid temporaries
        return _apply_pipeline(self.series, self.ops, chunk_size)


def _apply_pipeline(series, ops, chunk_size):
    parts = []
    for start in range(0, len(series), chunk_size):
        chunk = series.iloc[start:start + chunk_size]
        for op_name, arg in ops:
            if op_name == 'lower':
                chunk = chunk.str.lower()
            elif op_name == 'replace':
                pat, repl = arg
                chunk = chunk.str.replace(pat, repl, regex=False)
            elif op_name == 'strip':
                chunk = chunk.str.strip()
        parts.append(chunk)
    return type(series)(pd.concat(parts)) if parts else series

_apply_pipeline walks the recorded op list once per chunk (default: the whole Series in one chunk, for anything that fits in memory), calling the real vectorized Series.str method for each recorded op in sequence; "chunk-wise" here means each chunk only ever holds one intermediate Series at a time (reassigned to chunk on each step) rather than every stage's output existing simultaneously the way s.str.lower().str.replace(...).str.strip() chained directly would briefly do.

Worked example, verified with pandas on CPython 3.12:

import pandas as pd

s = pd.Series(['  Hello World  ', '  FOO-BAR  ', '  Already lower  '])
result = StrChain(s).lower().replace('-', ' ').strip().compute()
print(list(result))

Output:

['hello world', 'foo bar', 'already lower']

Each string is lowercased, has - replaced with a space, and is stripped of surrounding whitespace, in that recorded order, confirming the chain actually runs end to end and produces the same result s.str.lower().str.replace('-', ' ', regex=False).str.strip() would, just without materializing three separate full-Series temporaries to get there.

Implementation strategies:

• Represent ops as vectorized functions (use Series.str methods or numpy.char).
• Apply pipeline per chunk: read chunk, apply all ops in sequence in-place (reuse buffers), write out to result array or new Series with preallocated dtype.
• For unicode/regex heavy ops, compile regex ahead.
• Use numba (compiles a plain Python function to machine code the first time it runs) or cython (compiles Python-like code all the way down to C, with explicit type declarations, for the most control and the largest potential speedup) for the hot path once profiling shows plain vectorized Series.str ops are the actual bottleneck; for most chains, the vectorized ops above are already enough and neither is needed by default.

Minimize allocations:

• Reuse a single buffer per chunk; preallocate numpy object/bytes arrays when possible.
• Fuse operations into one pass (e.g., lower+strip -> single routine, implemented as one numpy.char or Cython function operating character-by-character instead of two separate full passes over the data).

Trade-offs: chunking adds overhead but limits peak memory; fusing ops increases implementation complexity.

Requirements: per-user 7-day rolling sum by event_time, scalable.

Idiomatic, vectorized Pandas (vectorized meaning the loop over rows runs as fast, compiled bulk code instead of a slow Python-level for loop, one row at a time):

import pandas as pd

df = pd.DataFrame({
    "user_id": ["u1", "u1", "u1", "u1", "u2"],
    "event_time": pd.to_datetime([
        "2024-01-01", "2024-01-03", "2024-01-05", "2024-01-09", "2024-01-02"
    ]),
    "value": [10, 20, 30, 40, 5],
})

# ensure datetime and sort
df['event_time'] = pd.to_datetime(df['event_time'])
df = df.sort_values(['user_id', 'event_time'])

# set index for time-based rolling and compute 7-day sum per user
result = (
    df.set_index('event_time')
      .groupby('user_id')['value']
      .rolling('7D')
      .sum()
      .reset_index(name='rolling_7d_sum')
)

Worked example, verified with pandas on CPython 3.12:

df = pd.DataFrame({
    "user_id": ["u1", "u1", "u1", "u1", "u2"],
    "event_time": pd.to_datetime([
        "2024-01-01", "2024-01-03", "2024-01-05", "2024-01-09", "2024-01-02"
    ]),
    "value": [10, 20, 30, 40, 5],
})
print(result)  # (built by running df through the code above)

Output:

  user_id event_time  rolling_7d_sum
0      u1 2024-01-01            10.0
1      u1 2024-01-03            30.0
2      u1 2024-01-05            60.0
3      u1 2024-01-09            90.0
4      u2 2024-01-02             5.0

Tracing u1's rows: Jan 1 has no prior rows in its trailing 7-day window, so the sum is just its own value, 10. Jan 3's window (Jan 3 back to Dec 27) includes Jan 1 and Jan 3, 10 + 20 = 30. Jan 5's window (back to Dec 29) includes Jan 1, 3, and 5, 10 + 20 + 30 = 60. Jan 9's window (back to Jan 2) EXCLUDES Jan 1, which is 8 days earlier, outside the 7-day cutoff, so it sums only Jan 3, 5, and 9: 20 + 30 + 40 = 90. This is exactly what makes it a time-based, not row-count-based, window: the number of prior rows included varies depending on how many actually fall within the trailing 7 real days, not a fixed count of the last N rows.

Notes: Uses time-based rolling with groupby which is vectorized and memory-efficient for large data. For millions of rows, ensure event_time is datetime64, operate on chunked parquet files (parquet: a compressed, columnar file format for tabular data, well suited to reading only the columns and row-groups a job actually needs) if memory constrained, and consider Dask or PySpark for distributed scaling once the data genuinely no longer fits on one machine: Dask mirrors the Pandas API but splits data into partitions and runs the same groupby/rolling-style operations across them in parallel; PySpark is a different, JVM-backed distributed engine with its own (similar but not identical) DataFrame API. Plain, chunked Pandas is enough for anything that still fits on a single machine's memory; reach for Dask/PySpark only once it genuinely does not.

Approach

I’d normalize by lowercasing, removing punctuation, and collapsing repeated spaces. Then I’d use a set of normalized keys to keep only the first occurrence. A set is a data structure that gives fast membership checks, so this stays simple and efficient for a few thousand strings.

import re
from typing import List


def normalize_title(title: str) -> str:
    cleaned = re.sub(r'[\W_]+', ' ', title.lower())
    return ' '.join(cleaned.split())


def dedupe_titles(titles: List[str]) -> List[str]:
    seen = set()
    result = []

    for title in titles:
        norm = normalize_title(title)
        if norm in seen:
            continue
        seen.add(norm)
        result.append(norm)

    return result


titles = ['  Weekly Update!', 'weekly update', 'Budget Review', 'Budget review  ']
print(dedupe_titles(titles))

The regex pattern r'[\W_]+' is what actually strips punctuation: \W (capital W) matches any character that is NOT a letter, digit, or underscore, so it catches punctuation, symbols, and stray whitespace runs in one go; adding _ to the character class folds underscores into that same "replace with a space" rule too, since \W alone would NOT match an underscore (underscores count as word characters). The + means one-or-more, so any run of these unwanted characters collapses to a single space rather than leaving multiple spaces behind, and ' '.join(cleaned.split()) then trims leading/trailing spaces and collapses any remaining internal runs down to exactly one space each.

Example

Input: [' Weekly Update!', 'weekly update', 'Budget Review', 'Budget review ']

Output:

['weekly update', 'budget review']

That is the literal stdout of print(dedupe_titles(titles)) from the code block above.

Why this works: the first weekly update is kept, and the later duplicate is skipped because it normalizes to the same key.

Complexity: O(n * m) time, where m is string length, and O(n) extra space for the set.

If the team wants to keep original casing, I would store the first original string in result instead of the normalized version.

The Global Interpreter Lock (GIL) is a mutex in CPython that ensures only one native thread executes Python bytecode at a time. It simplifies memory management but serializes CPU-bound Python code.

Workloads where threads still help despite the GIL:

• I/O-bound workloads: threads waiting on network, disk, or database I/O release the GIL during blocking calls, improving throughput (e.g., web crawlers, concurrent HTTP clients). Concretely, verified on CPython 3.12: two threads that each print, sleep (standing in for a blocking network call), then print again:

import threading, time

order = []

def worker(name, delay):
    order.append(f"{name} start")
    time.sleep(delay)      # GIL released for the duration of the sleep
    order.append(f"{name} done")

t1 = threading.Thread(target=worker, args=("A", 0.1))
t2 = threading.Thread(target=worker, args=("B", 0.1))
t1.start(); t2.start()
t1.join(); t2.join()
print(order)

A representative run printed ['A start', 'B start', 'B done', 'A done']: both start entries appear before either done, which is only possible because thread A released the GIL during its time.sleep(0.1), letting thread B start and run its own code during that wait, rather than A blocking B out entirely the way a genuine CPU-bound loop with no sleep/I-O would (there, the two threads would still only ever have one of them executing Python bytecode at a time, and the interleaving above would not occur). The exact order of the two done entries is not guaranteed (it depends on OS scheduling), but the invariant that matters, both starts landing before either finish, reliably demonstrates the GIL being released during the blocking wait, not a specific timing.

• Native-code parallelism: when threads call C extensions that release the GIL (numpy heavy computations, image processing libraries, cryptography), CPU work runs in parallel across cores.

For CPU-bound pure-Python tasks use multiprocessing or native extensions to achieve parallelism.