Python Programming Questions

Approach

I would stream the file line by line so memory stays bounded, meaning it stays roughly constant no matter how large the file is. For each line, trim whitespace, skip blanks, try to parse the fields, and update counters. Malformed lines go into a small issue count or log so the job can continue.

from dataclasses import dataclass


@dataclass
class Report:
    total: int = 0
    overdue: int = 0
    days_overdue_sum: int = 0
    malformed: int = 0


def process_tickets(path: str) -> Report:
    report = Report()

    with open(path, 'r', encoding='utf-8') as f:
        for line_no, line in enumerate(f, start=1):
            line = line.strip()
            if not line:
                continue

            parts = line.split(',')
            if len(parts) != 3:
                report.malformed += 1
                continue

            try:
                status = parts[1].strip().lower()
                days = int(parts[2].strip())
            except ValueError:
                report.malformed += 1
                continue

            if not parts[0].strip():
                report.malformed += 1
                continue

            report.total += 1
            if status == 'overdue':
                report.overdue += 1
                report.days_overdue_sum += days

    return report

Worked example: with A1,overdue,3, a blank line, and B2,open,0, the report counts 2 valid records, 1 overdue ticket, and 3 total overdue days.

Tradeoffs

• Fast and memory-safe for huge files
• One pass only, so you should add any needed counters up front
• If operators need examples of bad lines, log the first few rather than storing every bad line

Complexity: O(n) time and O(1) memory for the counters.

Goal: compute rank within each 'category' in descending order efficiently.

Solution:

import pandas as pd

# assuming df has columns ['category', 'value']
df = pd.DataFrame({
    "category": ["A", "A", "A", "B", "B"],
    "value":    [10,  10,  8,   5,   5],
})

# use groupby + rank with method='dense' or 'first' depending on tie behavior
df['rank_within_group'] = df.groupby('category')['value']\
    .rank(method='dense', ascending=False).astype(int)

method='dense' is the crux of the tie-breaking behavior here: it means tied values SHARE a rank, and the next distinct value continues at the very next rank number with no gap. That is different from method='first' (ties broken by row order, so no two rows ever share a rank) and from the default method='average' (ties get the average of the ranks they would have spanned). For descending values like 10, 10, 8, dense ranks them 1, 1, 2, not 1, 2, 3 (that would be method='first' or 'min'-adjacent behavior) and not 1.5, 1.5, 3 (that would be the average default).

Worked example, verified with pandas on CPython 3.12, including a genuine tie so the behavior is visible rather than only named:

df = pd.DataFrame({
    "category": ["A", "A", "A", "B", "B"],
    "value":    [10,  10,  8,   5,   5],
})
df['rank_within_group'] = df.groupby('category')['value']\
    .rank(method='dense', ascending=False).astype(int)
print(df)

Output:

  category  value  rank_within_group
0        A     10                  1
1        A     10                  1
2        A      8                  2
3        B      5                  1
4        B      5                  1

Category A's two tied 10s both get rank 1 (the highest, since ascending=False), and the next distinct value, 8, gets rank 2 immediately, no rank 3 is skipped to "make room" for the tie. Category B's two tied 5s both get rank 1, since within group B, 5 is the only (and therefore highest) value present.

Performance & memory:

• groupby.rank is implemented in C and avoids Python-level loops; it's fast and memory-efficient compared to apply.
• For millions of rows, ensure 'category' is categorical to save memory and speed grouping:

df['category'] = df['category'].astype('category')

• If memory is tight, process in chunks by categories: iterate over df.groupby('category') or sort by category and value and rank per chunk.

Trade-offs:

• In-memory groupby requires holding column arrays; converting to categorical reduces memory but adds preprocessing cost.
• Chunking reduces peak RAM but loses single-call simplicity and may require external sorting.

How they work together:

• CPython uses reference counting: each object tracks references; when count hits zero it's reclaimed immediately.
• The cyclic GC (generational, meaning it groups objects by how long they've survived so far and scans the youngest group most often, since most garbage dies young and rarely needs a second look) complements this by detecting reference cycles that reference counting alone can't free.

Example needing GC:
Two objects referencing each other (cycle) with del avoided or handled by GC. Example:

class Node:
    def __init__(self):
        self.ref = None

a = Node()
b = Node()
a.ref = b
b.ref = a
# If no external refs remain, ref counts are non-zero due to cycle -> GC must collect

Actually watching the reclamation happen, verified on CPython 3.12, extending the example above with a __del__ and an explicit gc.collect():

import gc

class Node:
    def __init__(self, name):
        self.name = name
        self.ref = None
    def __del__(self):
        print(f'__del__ called for {self.name}')

a = Node('a')
b = Node('b')
a.ref = b
b.ref = a
del a, b            # neither refcount reaches zero; both still reference each other

gc.collect()         # the cyclic GC finds the unreachable pair and reclaims it

Output:

__del__ called for a
__del__ called for b

Both finalizers ran as part of gc.collect() reclaiming the cycle, confirming the cycle really was found and cleaned up, not just theoretically collectible.

Reference counting handles most deallocations quickly; the cyclic GC periodically finds unreachable cycles and reclaims them. Cycles involving objects with __del__ used to require special handling: on Python versions before 3.4, the cyclic GC could not safely decide what order to finalize such objects in, so it left them permanently uncollected in gc.garbage instead of guessing. Since Python 3.4 (PEP 442), __del__ is called safely as part of collecting the cycle, exactly as demonstrated above, and the objects are then reclaimed normally; citing the old "cycles with __del__ can't be collected" claim as current fact is a dated answer today.

Approach: stream the file line-by-line using an internal buffer of chunk_size, parse CSV incrementally, yield namedtuples and convert numeric-looking strings to int/float lazily.

Implementation:

import csv
from collections import namedtuple
from io import TextIOBase

def stream_csv(path_or_file, chunk_size=65536):
    f = open(path_or_file, 'r', newline='') if isinstance(path_or_file, str) else path_or_file
    reader = csv.reader(f)
    headers = next(reader)
    Row = namedtuple('Row', [h.replace(' ', '_') for h in headers])
    def conv(s):
        if s == '':
            return None
        try:
            i = int(s)
            return i
        except ValueError:
            try:
                return float(s)
            except ValueError:
                return s
    for raw in reader:
        yield Row(*[conv(field) for field in raw])
    if isinstance(path_or_file, str):
        f.close()

A namedtuple (created above via namedtuple('Row', [...])) is a lightweight, tuple-backed class whose fields you can access by name, row.age, instead of only by position, row[1], which is why it is used here rather than a plain tuple or a dict: it is as compact and fast as a tuple, but each yielded row reads like a normal object with named fields.

Worked example, verified on CPython 3.12:

import io

sample_csv = "name,age,score\nAlice,30,9.5\nBob,,7\n"
rows = list(stream_csv(io.StringIO(sample_csv)))
print(rows)
# [Row(name='Alice', age=30, score=9.5), Row(name='Bob', age=None, score=7)]
print(rows[0].name, rows[1].age)
# Alice None

conv turns '30' into the int 30 and '9.5' into the float 9.5; Bob's empty age field hits the s == '' check first and becomes None rather than being run through int()/float() at all, and rows[0].name shows the named-field access a plain tuple could not offer without remembering which index held which column.

Key points: single-pass, constant memory per-row, converts numbers eagerly per-field. For truly huge fields, use streaming CSV parsers (csv module is already streaming).

Complexity: O(n) time where n is total fields; O(1) extra space per row.

Edge cases: quoted newlines, malformed rows, large field sizes: consider using csv.DictReader and incremental file objects if needed.

Why vectorized ops are faster:

• Vectorized NumPy/Pandas operations run in optimized C/Fortran loops avoiding Python per-element overhead.
• They leverage contiguous memory, CPU cache, SIMD (Single Instruction, Multiple Data: a CPU feature that applies one instruction to several values in a single step, instead of one value at a time), and multi-threaded BLAS (Basic Linear Algebra Subprogram, a standard library of fast, hardware-tuned matrix/vector math routines that numpy calls into) for numeric work.
• Example: adding two arrays uses a single C loop vs Python loop calling millions of Python operations.

Worked example, verified on CPython 3.12 (showing what actually gets computed, not a timing benchmark: exact speed depends on hardware and array size, so run time.perf_counter() yourself to see the real gap on your own machine rather than trusting a hardcoded number here):

import numpy as np

a = np.array([1.0, 2.0, 3.0, 4.0])
b = np.array([10.0, 20.0, 30.0, 40.0])

def add_loop(a, b):
    return [x + y for x, y in zip(a, b)]

def add_vectorized(a, b):
    return a + b

print(add_loop(a, b))
# [11.0, 22.0, 33.0, 44.0]
print(add_vectorized(a, b))
# [11. 22. 33. 44.]

Both compute the identical values; add_loop does it via four separate Python-level additions, each one dispatched, type-checked, and reference-counted by the interpreter, while add_vectorized's a + b dispatches once into a single compiled C loop over the two contiguous memory buffers, which is the mechanism, not a specific timing number, that makes the vectorized version pull ahead as array size grows.

When vectorization can be slower:

• Small arrays: overhead of creating intermediate arrays and function call overhead can dominate; a simple Python loop or in-place updates may be faster.
• Complex element-wise logic with branching: vectorization may require multiple large temporaries or complex masking, increasing memory bandwidth and runtime.

Where vectorization stops winning, made concrete: a piecewise function like "if x < 0, return 0; elif x < 10, return x; else return x**2" vectorizes via np.where, but every branch has to be computed for every element before the mask selects which result to keep:

x = np.array([-3.0, 5.0, 15.0])
result = np.where(x < 0, 0.0, np.where(x < 10, x, x ** 2))
print(result)
# [  0.   5. 225.]

Even though only one branch is "true" for any given element, x ** 2 is computed for all three elements (including -3.0 and 5.0, whose squared values are simply discarded by the outer np.where), and each np.where call allocates a full temporary array the size of x. For a genuinely small array or a function with many branches, that wasted work and temporary-array allocation can cost more than a plain Python loop that only evaluates the one branch each element actually needs; this is the concrete shape of "vectorization can be slower."

Example: computing a piecewise function with many branches can be slower vectorized due to multiple masks and temporary arrays; using numba (a library that compiles a plain Python function to machine code the first time it runs, letting a genuinely branchy per-element loop run at near-C speed without vectorizing it at all) for a compiled loop may be faster and more memory-efficient.