Python Programming Questions

My decision

I would lean Go for this tool if the parsing is truly CPU-heavy, meaning the machine spends most of its time computing rather than waiting on disk, and the team wants one deployable binary. Go's compiled binary, static typing, and built-in concurrency fit unattended daily jobs well.

What matters beyond raw speed

• team familiarity and onboarding
• library support for the file formats you need
• packaging and deployment simplicity
• error handling and testability
• memory footprint and startup time
• how often the rules change

When I would pick Python instead
If the job mostly glues together existing Python libraries, or if the engineers need to change parsing rules every week, Python may be faster to evolve and easier to read.

Worked example
If the tool reads 500 large files every morning, I would favor Go when parsing and parallel file handling are the bottlenecks. If the same tool is edited often by a small team that values quick iteration over strict compilation, Python may win because maintenance cost matters more than raw throughput.

So I choose the language that minimizes total operating cost, not just the fastest loop.

The Global Interpreter Lock (GIL) is a mutex in CPython that ensures only one native thread executes Python bytecode at a time. It simplifies memory management but serializes CPU-bound Python code.

Workloads where threads still help despite the GIL:

• I/O-bound workloads: threads waiting on network, disk, or database I/O release the GIL during blocking calls, improving throughput (e.g., web crawlers, concurrent HTTP clients). Concretely, verified on CPython 3.12: two threads that each print, sleep (standing in for a blocking network call), then print again:

import threading, time

order = []

def worker(name, delay):
    order.append(f"{name} start")
    time.sleep(delay)      # GIL released for the duration of the sleep
    order.append(f"{name} done")

t1 = threading.Thread(target=worker, args=("A", 0.1))
t2 = threading.Thread(target=worker, args=("B", 0.1))
t1.start(); t2.start()
t1.join(); t2.join()
print(order)

A representative run printed ['A start', 'B start', 'B done', 'A done']: both start entries appear before either done, which is only possible because thread A released the GIL during its time.sleep(0.1), letting thread B start and run its own code during that wait, rather than A blocking B out entirely the way a genuine CPU-bound loop with no sleep/I-O would (there, the two threads would still only ever have one of them executing Python bytecode at a time, and the interleaving above would not occur). The exact order of the two done entries is not guaranteed (it depends on OS scheduling), but the invariant that matters, both starts landing before either finish, reliably demonstrates the GIL being released during the blocking wait, not a specific timing.

• Native-code parallelism: when threads call C extensions that release the GIL (numpy heavy computations, image processing libraries, cryptography), CPU work runs in parallel across cores.

For CPU-bound pure-Python tasks use multiprocessing or native extensions to achieve parallelism.

Bottleneck reasoning

A bottleneck is the part of the system that dominates total runtime. If the report generator scans the same task records over and over, each new question forces another full pass through the data. That scales poorly as the team grows.

5 scans * 100000 rows = 500000 row visits
1 scan * 100000 rows = 100000 row visits

Intuition: one pass is 5x less work in this example.

What I would change
I would switch the data flow from repeated scans to one ingestion pass that builds summary structures as it goes:

• counters for totals by status, owner, or date
• maps indexed by the fields used in reports
• one reducer function that updates all needed aggregates together

Then each summary question reads from the precomputed aggregates instead of touching the raw records again. That keeps queries fast and the code maintainable because the aggregation logic lives in one place.

Tradeoff
The main cost is a bit more memory for the summary maps. I would accept that tradeoff because it is much cheaper than rescanning the whole dataset repeatedly.

Example
If ten report questions all need ticket counts by team, I would build that count once during ingestion, then reuse it for every report instead of looping over the task table ten times.

Experiment design
Compare three functions in a tight loop: (A) raise/catch exception on error, (B) return error code and check, (C) pre-validated path (no error). Use timeit and large N.

Code:

import timeit

def raise_path(n):
    for i in range(n):
        try:
            if i%100==0: raise ValueError
        except ValueError:
            pass

def errorcode_path(n):
    for i in range(n):
        ok = True            # the normal, no-error case
        if i%100==0: ok = False   # the simulated error case
        if not ok: pass

def prevalidated_path(n):
    for i in range(n):
        pass                 # no error branch at all: the baseline

n=1000000
print(timeit.timeit(lambda: raise_path(n), number=3))
print(timeit.timeit(lambda: errorcode_path(n), number=3))
print(timeit.timeit(lambda: prevalidated_path(n), number=3))

A correction to the code above (a genuine bug, not just a style choice): as originally sketched, errorcode_path set ok = False unconditionally at the top of every iteration and never set it back to True on the normal path, so if not ok: was True on every single iteration, not just the 1-in-100 simulated errors. That does not model "check an error code" at all, it just runs the pass branch every time. The fix is the one shown above: ok = True by default (the common, no-error case), flipped to False only on the simulated error, so errorcode_path and raise_path are actually testing the same 1% error frequency against each other. The third function, prevalidated_path, is also added here: it has no error branch whatsoever, and exists specifically as the baseline "cost of the loop itself, with no error handling of any kind" that the question's three-way comparison (raise/catch, return-code, pre-validated) asks for; the original sketch defined only the first two.

Interpretation

• Run this at varying simulated error frequencies (change i%100==0 to i%1==0 for 100% errors, or remove the if entirely for 0%) rather than trusting a single frequency; the shape of the gap between raise_path and errorcode_path as a function of error frequency is the actual finding, a single absolute number from one machine is not portable or reproducible elsewhere and should not be reported as "the" result. prevalidated_path's time is the floor: the cost of iterating n times with no error-handling machinery of any kind, useful as the baseline both other paths are measured against.
• The mechanism, not a specific number, is what to lead with: constructing and raising a real exception involves allocating an exception object, populating a traceback, and unwinding the stack to find a matching except, real, nonzero work that a plain boolean check never does; at 0% error frequency, the try block itself still costs something (setting up the exception-handling frame) even though nothing is ever raised, so raise_path at 0% errors is a fair comparison of that baseline try-frame overhead against errorcode_path's baseline if overhead. As error frequency rises toward 100%, the actual raise/unwind cost starts to dominate raise_path's total time in a way it never does for errorcode_path, which does the same constant amount of work (ok = True, one comparison) whether or not that iteration's error flag ends up True.
• For production ETL, prefer error-code or pre-validation for expected, frequent errors, reserve exceptions for truly exceptional control flow, situations that are rare enough that even a real per-raise cost barely matters in aggregate, and where the cleaner control-flow and forced handling (you cannot silently ignore a raised exception the way an unchecked error code can be ignored) outweigh that cost.
• Profile memory and CPU (not just the timeit total) to ensure GC or traceback construction isn't dominating the comparison for reasons unrelated to the actual branch being tested, for example, a test harness that also does unrelated allocation inside the timed loop.

Approach: stream the file line-by-line using an internal buffer of chunk_size, parse CSV incrementally, yield namedtuples and convert numeric-looking strings to int/float lazily.

Implementation:

import csv
from collections import namedtuple
from io import TextIOBase

def stream_csv(path_or_file, chunk_size=65536):
    f = open(path_or_file, 'r', newline='') if isinstance(path_or_file, str) else path_or_file
    reader = csv.reader(f)
    headers = next(reader)
    Row = namedtuple('Row', [h.replace(' ', '_') for h in headers])
    def conv(s):
        if s == '':
            return None
        try:
            i = int(s)
            return i
        except ValueError:
            try:
                return float(s)
            except ValueError:
                return s
    for raw in reader:
        yield Row(*[conv(field) for field in raw])
    if isinstance(path_or_file, str):
        f.close()

A namedtuple (created above via namedtuple('Row', [...])) is a lightweight, tuple-backed class whose fields you can access by name, row.age, instead of only by position, row[1], which is why it is used here rather than a plain tuple or a dict: it is as compact and fast as a tuple, but each yielded row reads like a normal object with named fields.

Worked example, verified on CPython 3.12:

import io

sample_csv = "name,age,score\nAlice,30,9.5\nBob,,7\n"
rows = list(stream_csv(io.StringIO(sample_csv)))
print(rows)
# [Row(name='Alice', age=30, score=9.5), Row(name='Bob', age=None, score=7)]
print(rows[0].name, rows[1].age)
# Alice None

conv turns '30' into the int 30 and '9.5' into the float 9.5; Bob's empty age field hits the s == '' check first and becomes None rather than being run through int()/float() at all, and rows[0].name shows the named-field access a plain tuple could not offer without remembering which index held which column.

Key points: single-pass, constant memory per-row, converts numbers eagerly per-field. For truly huge fields, use streaming CSV parsers (csv module is already streaming).

Complexity: O(n) time where n is total fields; O(1) extra space per row.

Edge cases: quoted newlines, malformed rows, large field sizes: consider using csv.DictReader and incremental file objects if needed.