Python Programming Questions

Overview of approaches:

• requirements.txt + pip: Simple, exact package names; use pip freeze to pin. Reproducible only if you pin hashes (pip hash or requirements with hashes).
• pip-tools (pip-compile): Maintains human-editable top-level requirements.in and generates fully pinned requirements.txt (including sub-deps).
• Pipenv: Combines virtualenv + Pipfile/Pipfile.lock (a lock file records the exact version, and a hash, of every package actually installed, including indirect dependencies pulled in by your direct ones, so a second install reproduces the identical environment instead of silently picking up newer, still-technically-compatible versions); locking for reproducibility but has had stability/performance issues historically.
• Poetry: Modern tool for dependency resolution, pyproject.toml, and lock file; good for packaging and reproducible installs across environments.
• Conda: Manages packages and binaries (C libs) and environments; best when native dependencies (numpy, scipy) are needed.

Which of these is baseline knowledge versus niche: for most purposes, pip + a lockfile-pinned requirements.txt and Poetry are the two worth knowing well; pip-tools and Pipenv are reasonable alternatives worth recognizing as "another lockfile-based tool" rather than needing a strong independent opinion on; Conda (and micromamba, below) only becomes necessary once native, non-Python dependencies enter the picture.

My pick for reproducible data-science project: Poetry + Conda hybrid depending on needs.

• If heavy native deps: use Conda to create base env (python, BLAS, the compiled linear-algebra library numpy/scipy rely on internally for fast matrix math, and something Poetry itself cannot install since it only manages Python-level packages, not compiled system libraries), then Poetry for Python deps inside that env, or use micromamba (a faster, lighter-weight, drop-in alternative to the conda command line tool, installing from the same package channels) + poetry. Concretely: conda create -n proj python=3.12 numpy scipy to get the compiled base right, then poetry init and poetry add pandas inside that environment to layer the pure-Python dependencies on top with a proper lockfile. Poetry gives clear lockfile, semantic project layout, and reproducible installs. pip-tools is a lighter alternative if you prefer pip. Ensure CI records exact lockfile and Python version.

Definitions:

• Shallow copy: creates a new container but inserts references to the same nested objects.
• Deep copy: recursively copies nested objects so modifications don't affect the original.

Example:

import copy
orig = {'nums': [1, 2]}
sh = copy.copy(orig)
dp = copy.deepcopy(orig)

sh['nums'].append(3)
print(orig)  # {'nums': [1,2,3]} -> changed via shallow copy

dp['nums'].append(4)
print(orig)  # unchanged by deep copy

When to use: shallow copy is fine for immutable nested objects or when shared mutation is intended; deep copy when you need independent mutable nested structures. Be cautious: deep copy can be expensive and may fail for unpicklable objects.

Plan: collect profiles, identify roots, mitigate with code/data changes.

Collecting memory profiles:

• Use tracemalloc (a built-in module that records which line of code allocated each block of memory, and lets you snapshot total/peak Python-level allocations at any point) for Python allocations
• Use memory_profiler (@profile) (a decorator-driven tool that reports memory usage line by line for a decorated function, run via python -m memory_profiler script.py) to trace per-line memory
• Use psutil / top for process-level peaks (psutil: a library that reads OS-level process stats, including RSS, resident set size, the actual physical RAM a process is currently using, directly from the operating system)
• For NumPy, use objgraph (a tool for visualizing and counting live Python object references, useful for finding what is unexpectedly still holding a reference to something large) / np.ndarray.nbytes sampling
• Take periodic heap dumps and track peak RSS

Which to reach for first: tracemalloc is the default starting point for almost any Python-level memory investigation, cheap to turn on, built in, and line-level. psutil/RSS is the next check when the growth seems to be happening outside pure Python objects entirely (a C extension, or a numpy allocation that tracemalloc's Python-object view does not fully capture). memory_profiler and objgraph are for harder follow-up cases once the first two have narrowed down roughly where the growth is, not where most investigations should start.

A worked trace with tracemalloc, verified on CPython 3.12 with numpy: creating a float64 array and then converting it with .astype(np.float32) (a common, easy-to-miss source of an avoidable extra allocation) shows up as two distinct, measurable jumps:

import tracemalloc
import numpy as np

tracemalloc.start()
baseline, _ = tracemalloc.get_traced_memory()

a = np.zeros(1_000_000, dtype=np.float64)
after_a, _ = tracemalloc.get_traced_memory()

b = a.astype(np.float32)   # a full, avoidable copy if a float32 array was all that was ever needed
after_b, _ = tracemalloc.get_traced_memory()

print("a allocation bytes:", after_a - baseline)
print("b (astype copy) allocation bytes:", after_b - after_a)

Output:

a allocation bytes: 8000496
b (astype copy) allocation bytes: 4000096

The dominant numbers, 8,000,000 and 4,000,000, are exactly 1_000_000 * 8 bytes (float64) and 1_000_000 * 4 bytes (float32); the small remainder (496 and 96 bytes) is bookkeeping overhead from numpy's own array object and can vary slightly by numpy version, the part worth remembering is the dtype-size arithmetic, not the exact overhead constant. If a was only ever needed as float32, allocating it directly as np.zeros(1_000_000, dtype=np.float32) avoids the second, redundant 4,000,000-byte allocation entirely, this is exactly the kind of concrete before/after number a real profiling pass surfaces that a purely descriptive plan does not.

Common causes:

• Creating many temporaries (slicing / copying)
• Unnecessary copies from dtype promotion (dtype promotion: when an operation combines two arrays of different dtypes, e.g. float32 and float64, numpy silently produces a result in the wider of the two dtypes, float64, which can quietly double the memory of a computation that was only ever intended to stay in float32)
• Holding references to large arrays (accumulators, list append)
• Parallel workers duplicating memory

Mitigations:

• Use in-place operations (out=...) and views (a view: a new array object that points at the same underlying memory as another array, e.g. a[::2], versus a copy, which allocates and fills an entirely separate block of memory) where safe
• Use appropriate dtypes (float32 vs float64)
• Process data in chunks / streaming
• Reuse preallocated arrays and buffer pools
• Avoid building large Python lists; use arrays or write to disk-backed arrays (memmap)
• Release references and call gc.collect when necessary for deterministic release

Validation: write microbenchmarks for peak RSS (RSS: resident set size, the actual physical memory a process currently occupies, as reported by the OS), compare before/after; run under production-like data sizes.

Requirements

• Watch a shared folder, validate each report, aggregate metrics, and write one daily summary.
• Keep running if one file is bad.

Design
I would prefer a scheduled scanner over a pure file watcher, because shared folders can miss events. The CLI would:

1. list files in incoming/
2. move each file atomically to processing/
3. validate and parse it
4. write metrics to a temp summary file
5. rename the temp file to the final output when complete

Idempotent design
Idempotent means running it twice produces the same result, with no duplicate output. I would track a checksum or a (date, filename) key in a small state file or SQLite table. If the same file appears twice, skip it. If the run dies midway, rerun only unprocessed files.

Debuggability

• structured logs with filename, checksum, line number, and error
• quarantine bad files in failed/ with a reason file
• counters for processed, skipped, and failed
• a --file replay mode for one broken input

Worked example: if ops_2025-07-03.csv has a bad line 18, the run should still finish the other files, mark that one as failed, and include the failure in the final summary. That makes the job observable instead of mysterious.

Approach
Run import in a subprocess to enforce timeout and capture import errors/syntax errors without raising in caller.

Implementation:

import subprocess, sys, json, tempfile

def safe_import(module_name, timeout=5):
    code = f"import importlib, json\ntry:\n m=importlib.import_module('{module_name}')\n print('OK')\nexcept Exception as e:\n print('ERR:'+type(e).__name__+':'+str(e))\n"
    proc = subprocess.run([sys.executable, '-c', code], capture_output=True, text=True, timeout=timeout)
    out = proc.stdout.strip()
    if out.startswith('OK'):
        return __import__(module_name), None
    if out.startswith('ERR:'):
        return None, out[4:]
    return None, 'Unknown error'

Why the module gets imported twice
The subprocess step and the __import__ step are doing two different jobs, not one job done redundantly. The subprocess runs in a completely separate operating-system process, with its own memory, so whatever module object it constructs cannot be handed back to the caller at all, a subprocess and its parent only communicate through the captured stdout text, 'OK' or 'ERR:...', never through live Python objects. So the subprocess's only purpose is to safely test-drive the import (catching a hang via timeout, or a crash, or a SyntaxError from a broken package) without risking the caller's own process. Once that test comes back 'OK', the code still needs an actual, usable module object inside the CALLER's process, which is exactly what the second call, __import__(module_name), provides; it re-runs the import for real, now that it is known to be safe, in the process that actually needs the result.

Worked example, verified on CPython 3.12:

mod, err = safe_import('json')
print(mod is not None, err)
# True None

mod2, err2 = safe_import('this_module_does_not_exist')
print(mod2, err2)
# None ModuleNotFoundError:No module named 'this_module_does_not_exist'

For a real, importable module, safe_import returns a usable module object and None for the error; for a nonexistent one, it returns None and the exact type/message the subprocess's except Exception as e branch captured, with nothing raised in the caller.

Notes

• Using subprocess prevents a broken package from crashing the caller and enforces timeout.
• For heavy imports this adds overhead; use caching for repeated imports.
• Captures ImportError, SyntaxError, or runtime exceptions during module import.