Python Programming Questions

Requirements

• Watch a shared folder, validate each report, aggregate metrics, and write one daily summary.
• Keep running if one file is bad.

Design
I would prefer a scheduled scanner over a pure file watcher, because shared folders can miss events. The CLI would:

1. list files in incoming/
2. move each file atomically to processing/
3. validate and parse it
4. write metrics to a temp summary file
5. rename the temp file to the final output when complete

Idempotent design
Idempotent means running it twice produces the same result, with no duplicate output. I would track a checksum or a (date, filename) key in a small state file or SQLite table. If the same file appears twice, skip it. If the run dies midway, rerun only unprocessed files.

Debuggability

• structured logs with filename, checksum, line number, and error
• quarantine bad files in failed/ with a reason file
• counters for processed, skipped, and failed
• a --file replay mode for one broken input

Worked example: if ops_2025-07-03.csv has a bad line 18, the run should still finish the other files, mark that one as failed, and include the failure in the final summary. That makes the job observable instead of mysterious.

Approach

Ask forgiveness, not permission: normalize away the known messy formatting (thousands separators, a currency symbol, a K/M/B magnitude suffix, parenthesized negatives), then attempt the actual conversion with float() inside a try/except ValueError, returning None on failure. Look-before-you-leap (LBYL) would mean writing a regex or a hand-rolled validator that decides in advance whether the string looks convertible, which in practice means re-implementing everything float()'s own parser already does correctly, just to decide whether to call it; that duplicated logic is extra surface area that can drift from what float() actually accepts, and it does not save the try/except anyway, since malformed input mixing multiple messy features ("$1,2M3") will still need a runtime check. EAFP here means: only do the normalization steps that resolve known, named messy formats, then let Python's own float parser be the single source of truth on whether the result is actually a valid number.

Code (Python 3.12)

import re

_SUFFIX_MULTIPLIER = {"K": 1e3, "M": 1e6, "B": 1e9}
_CURRENCY_CHARS = re.compile(r"[$,\s]")

def parse_number(raw):
    '''Convert a messy numeric string to float; return None if unparseable.
    Handles thousands separators (1,234.56), a leading currency symbol ($),
    a K/M/B magnitude suffix, and parenthesized negatives ((3.5K) == -3500).
    Treats 'nan' and '' as missing data (None), not as float('nan'): in this
    pipeline a NaN token means "value absent", not "the value is not-a-number".
    '''
    if raw is None or not isinstance(raw, str):
        return None
    text = raw.strip()
    if not text or text.lower() == "nan":
        return None

    negative = text.startswith("(") and text.endswith(")")
    if negative:
        text = text[1:-1]

    multiplier = 1.0
    if text and text[-1].upper() in _SUFFIX_MULTIPLIER:
        multiplier = _SUFFIX_MULTIPLIER[text[-1].upper()]
        text = text[:-1]

    text = _CURRENCY_CHARS.sub("", text)

    try:
        value = float(text)
    except ValueError:
        return None

    value *= multiplier
    return -value if negative else value

cases = ["1,234.56", "$1.2M", "NaN", "", "  42  ", "(3.5K)", "abc", None, "3.1e2"]
for c in cases:
    print(repr(c), "->", parse_number(c))

# Absorbed variant: average a batch of messy values while ignoring missing ones.
def average_ignoring_missing(raw_values):
    parsed = [v for v in (parse_number(x) for x in raw_values) if v is not None]
    return sum(parsed) / len(parsed) if parsed else None

print(average_ignoring_missing(["10", "", "20", "NaN", "30"]))

Output:

'1,234.56' -> 1234.56
'$1.2M' -> 1200000.0
'NaN' -> None
'' -> None
'  42  ' -> 42.0
'(3.5K)' -> -3500.0
'abc' -> None
None -> None
'3.1e2' -> 310.0
20.0

Key points

• Every normalization step (strip whitespace, strip a currency symbol, strip a magnitude suffix, strip parentheses) is applied unconditionally and cheaply; only the final float(text) call is wrapped in try/except, so the EAFP boundary is drawn as tightly as possible around the one operation whose success genuinely cannot be known in advance without duplicating its logic.
• "NaN" is deliberately treated as None (missing), not passed through to float("nan") (which would succeed and produce an actual NaN float): the two are different data-quality signals, "this field was never populated" versus "this field holds the floating-point value NaN," and conflating them would let a NaN silently poison a downstream sum() or comparison (nan != nan in every comparison, including equality, which produces confusing bugs if it leaks into arithmetic unexpectedly).
• isinstance(raw, str) up front is LBYL for exactly one thing: rejecting non-string input (a list, an int already) before doing string operations on it that would raise AttributeError/TypeError rather than the ValueError this function is designed to swallow. Mixing that one type check into an otherwise EAFP function is a normal, common pattern, not a contradiction: LBYL and EAFP are not mutually exclusive within a single function, the choice is made per failure mode.

Complexity and edge cases

$O(k)$ where $k$ is the length of the string, dominated by the regex substitution and float()'s own parsing.

• Whitespace-only strings (" ") become "" after .strip() and correctly return None.
• A currency symbol combined with a suffix and a thousands separator all at once ("$1,234.5K") is handled correctly because the three normalization steps are independent and composable.
• Absorbed variant, averaging while ignoring missing values (shown in the code above via average_ignoring_missing): parse every value with the same parse_number, filter out the Nones, then average what remains, so a missing reading never silently counts as zero and never crashes the average.
• Absorbed variant, a safe money sum: float accumulates binary floating-point rounding error across many additions, which is unacceptable for currency; the EAFP shape stays the same but the target type changes to decimal.Decimal, which represents decimal fractions exactly:

import re
from decimal import Decimal, InvalidOperation

_CURRENCY_CHARS = re.compile(r"[$,\s]")

def parse_money(raw):
    '''Like parse_number but returns Decimal for exact currency arithmetic.'''
    if raw is None or not isinstance(raw, str):
        return None
    text = _CURRENCY_CHARS.sub("", raw.strip())
    if not text:
        return None
    try:
        return Decimal(text)
    except InvalidOperation:
        return None

prices = ["$19.99", "$5.01"]
total = sum((parse_money(p) for p in prices), start=Decimal("0"))
print(total)  # 25.00, exact

Goal: resample timezone-aware timestamps to hourly UTC bins, forward-fill, preserve tz-aware dtype.

Code:

import pandas as pd

def resample_hourly_utc(df):
    # ensure tz-aware
    ts = pd.to_datetime(df['timestamp'])
    if ts.dt.tz is None:
        raise ValueError('timestamp must be timezone-aware')
    # convert to UTC
    df = df.copy()
    df['timestamp'] = ts.dt.tz_convert('UTC')
    df = df.set_index('timestamp').sort_index()
    # resample hourly and forward-fill
    out = df.resample('h').ffill()   # lowercase 'h': pandas removed the uppercase hour alias
    # ensure index tz-aware and dtype preserved
    out.index = out.index.tz_localize(None).tz_localize('UTC') if out.index.tz is None else out.index
    return out

Worked example, verified with pandas on CPython 3.12:

df = pd.DataFrame({
    "timestamp": pd.to_datetime(
        ["2024-11-03 04:15", "2024-11-03 06:40", "2024-11-03 08:05"]
    ).tz_localize("America/New_York"),
    "value": [10, 20, 30],
})
print(resample_hourly_utc(df))

Output:

                           value
timestamp
2024-11-03 09:00:00+00:00    NaN
2024-11-03 10:00:00+00:00   10.0
2024-11-03 11:00:00+00:00   10.0
2024-11-03 12:00:00+00:00   20.0
2024-11-03 13:00:00+00:00   20.0

The three original local readings, converted internally to UTC before resampling, become five hourly UTC bins: the first bin has no data yet before the first real reading, so it stays NaN (nothing earlier to forward-fill from), and each later hour repeats the most recent value until a newer one arrives (10.0 covers two consecutive bins because the next real reading does not land until the following hour).

DST pitfalls

• Convert to UTC before resampling to avoid ambiguous/missing local times.
• If you must resample in local tz, handle ambiguous times with tz_localize(ambiguous=...) and nonexistent with nonexistent=...; otherwise conversions across DST will produce duplicate or missing hours.

Why DST actually produces duplicate or missing local hours, made concrete: on 2024-11-03, US Eastern clocks "fall back" from 2:00 AM EDT to 1:00 AM EST, so local wall-clock time 01:00-01:59 happens twice, once while still on EDT (UTC-4) and once on EST (UTC-5):

naive = pd.Timestamp("2024-11-03 01:30:00")
first = naive.tz_localize("America/New_York", ambiguous=True)    # earlier occurrence, still EDT
second = naive.tz_localize("America/New_York", ambiguous=False)  # later occurrence, now EST
print(first, "->", first.tz_convert("UTC"))
print(second, "->", second.tz_convert("UTC"))

Output:

2024-11-03 01:30:00-04:00 -> 2024-11-03 05:30:00+00:00
2024-11-03 01:30:00-05:00 -> 2024-11-03 06:30:00+00:00

Both print the identical local wall-clock string, 01:30:00, but they are genuinely different instants, one hour apart in UTC; without ambiguous=, pandas has no way to know which of the two real moments a bare "2024-11-03 01:30:00" local string refers to, and that unresolved ambiguity is the "duplicate hour" pitfall.

The mirror case, a nonexistent time: on 2025-03-09, US Eastern clocks "spring forward" from 2:00 AM directly to 3:00 AM, so the entire local hour 02:00-02:59 never occurs at all:

try:
    pd.Timestamp("2025-03-09 02:30:00").tz_localize("America/New_York")  # expected: 02:30 never happened locally that day
except ValueError as e:
    print(f'raises as expected: ValueError: {e}')

print(pd.Timestamp("2025-03-09 02:30:00").tz_localize("America/New_York", nonexistent="shift_forward"))
# 2025-03-09 03:00:00-04:00

Asking for 02:30 local time on that date raises by default, since no such wall-clock moment ever happened; nonexistent="shift_forward" is one explicit policy for handling it (push the timestamp forward past the gap), rather than silently guessing what was meant.

This approach keeps data consistent for ETL and downstream aggregation.

Direct answer

List comprehensions are the clearest default for a single, simple transform-and-filter over data you'll use right away; generator expressions are the same idea but lazy (they compute each value only when you ask for it, one at a time, instead of building the whole result up front), for one-pass or large-data situations; map/filter are functionally equivalent to a comprehension but read worse the moment you need more than one simple function, because chaining them nests function calls instead of reading left to right; and plain for loops are the right choice once the body needs branching, multiple side effects, early exit, or error handling that a single expression cannot express cleanly.

Structured elaboration

Definitions and equivalent forms

data = [1, 2, 3, 4, 5, 6]

# list comprehension
evens_squared = [x * x for x in data if x % 2 == 0]

# generator expression: identical logic, lazy
evens_squared_lazy = (x * x for x in data if x % 2 == 0)

# map/filter equivalent, absorbed one-line coding variant
evens_squared_mf = list(map(lambda x: x * x, filter(lambda x: x % 2 == 0, data)))

print(evens_squared)       # [4, 16, 36]
print(evens_squared_mf)    # [4, 16, 36]

All three produce the identical result here; the difference is purely readability and (for the generator) laziness, not correctness.

Worked example: where map/filter actively hurts readability

The single-condition case above reads fine either way. It stops being fine the moment you need a second transform or condition, because map/filter compose by nesting calls, which has to be read from the inside out, while a comprehension reads left to right with all its clauses visible at the top level:

words = ["apple", "Bob", "cherry", "Dave", "elderberry"]

# comprehension: reads in the order things happen
result_comp = [w.upper() for w in words if len(w) > 3 if w[0].islower()]

# map/filter equivalent: two nested filters, then a map, read inside-out
result_mf = list(
    map(lambda w: w.upper(),
        filter(lambda w: w[0].islower(),
               filter(lambda w: len(w) > 3, words)))
)

print(result_comp)   # ['APPLE', 'CHERRY', 'ELDERBERRY']
print(result_mf)      # ['APPLE', 'CHERRY', 'ELDERBERRY']

Both are correct and produce the same list, but result_mf requires unwinding three nested calls to see the order operations actually happen in (innermost filter first, then the next filter, then map), while result_comp lists its two filter clauses and its transform in the order a reader executes them mentally. This is the concrete case where reaching for map/filter "because it's more functional" costs real readability for no benefit, since a comprehension expresses the identical logic more directly.

Trade-offs & pitfalls

• Comprehensions and generator expressions support arbitrarily many if clauses and nested for clauses in one flat, left-to-right expression; map/filter require nesting calls to combine more than one step, which inverts the reading order.
• map/filter earn their keep when you already have a named function (no lambda needed) and only one stage: map(str.upper, words) is arguably cleaner than [w.upper() for w in words] to some readers, precisely because there's no lambda and only one operation; this is a matter of team convention more than a hard rule.
• A plain for loop is strictly more powerful than any of the expression forms (it can break, continue, use try/except per item, update multiple variables, or log at each step) and should be preferred the moment a comprehension needs a helper function purely to work around the fact that its body can only be one expression.
• All of eager comprehensions, lazy generators, and map/filter (which return iterators, not lists, in Python 3) do a single pass over the input; the memory profile difference is comprehension (materializes fully) versus generator/map/filter (lazy, one item at a time), independent of which of the three lazy forms you pick.

Direct answer

A metaclass is the class of a class: just as an ordinary class controls how its instances are created, a metaclass controls how classes themselves are created. type is the default metaclass for every class in Python; overriding type.__new__ (or __init__) lets you intercept class creation itself and inject behavior, such as recording every subclass in a registry, the moment a class statement finishes executing. __init_subclass__ is a lighter, more modern hook (added in Python 3.6, PEP 487) that runs after a subclass object already exists, which covers plain subclass registration perfectly well and is what most engineers should reach for first. The honest reason to reach for a full metaclass instead is that __init_subclass__ cannot act before the class object is built: it cannot rewrite the namespace or base classes, enforce a shared metaclass across unrelated hierarchies that must interoperate, or run validation that has to see the class dictionary before Python assembles it into a type.

Structured elaboration

Approach

• Define RegistryMeta(type) and override __new__, which runs once per class statement, before the class object is returned.
• Build the class as normal via super().__new__(...), then, unless the class opts out (a sentinel attribute), record it in a module-level dict keyed by class name.
• Attach the metaclass to a base class via class Base(metaclass=RegistryMeta): ...; every subclass of Base automatically goes through RegistryMeta.__new__.

Worked example

# registry_meta.py
CLASS_REGISTRY = {}

class RegistryMeta(type):
    def __new__(mcls, name, bases, namespace, **kwargs):
        cls = super().__new__(mcls, name, bases, namespace)
        if namespace.get("__abstract__", False):
            return cls          # skip the abstract base itself
        CLASS_REGISTRY[name] = cls
        return cls

class Plugin(metaclass=RegistryMeta):
    __abstract__ = True

class CsvPlugin(Plugin):
    pass

class JsonPlugin(Plugin):
    pass

print(list(CLASS_REGISTRY.keys()))

Verified on CPython 3.12, this prints:

['CsvPlugin', 'JsonPlugin']

Plugin itself never appears, since it sets __abstract__ = True in its own namespace.

Key points

• __new__ on a metaclass runs before the class fully exists in the enclosing scope, so it is the correct place to reject or rewrite a class definition, not just observe it.
• The __abstract__ sentinel pattern keeps the base class out of the registry without special-casing its name.
• The registry itself is a plain module-level dict: O(1) lookup by class name, useful for plugin discovery and factory patterns (CLASS_REGISTRY["CsvPlugin"]()).

Why reach for the metaclass here at all, given __init_subclass__ exists:

SIMPLE_REGISTRY = {}

class SimpleBase:
    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        SIMPLE_REGISTRY[cls.__name__] = cls

class Alpha(SimpleBase):
    pass

class Beta(SimpleBase):
    pass

print(list(SIMPLE_REGISTRY.keys()))   # ['Alpha', 'Beta'], verified on CPython 3.12

For exactly the registration behavior asked about, this is fewer moving parts and does the identical job, no metaclass required. The genuine reason to prefer a metaclass is when you need to act before the class object exists, for example rejecting a malformed class outright instead of letting it get created and then complaining. __init_subclass__ only ever runs on an already-built class; it can veto by raising, but it cannot change what gets built. A metaclass's __new__ can inspect and reject the raw namespace before type.__new__ ever constructs the class:

class ValidatingMeta(type):
    def __new__(mcls, name, bases, namespace, **kwargs):
        if namespace.get("__abstract__", False):
            return super().__new__(mcls, name, bases, namespace)
        if "handle" not in namespace:
            raise TypeError(f"{name} must define a 'handle' method (checked before class creation)")
        return super().__new__(mcls, name, bases, namespace)

class Handler(metaclass=ValidatingMeta):
    __abstract__ = True

class GoodHandler(Handler):
    def handle(self):
        return "ok"

try:
    class BadHandler(Handler):    # expected: no `handle` method, ValidatingMeta.__new__ rejects it
        pass
except TypeError as e:
    print(f'raises as expected: TypeError: {e}')

Verified on CPython 3.12: GoodHandler is created normally; the class BadHandler(Handler): pass statement itself raises immediately with TypeError: BadHandler must define a 'handle' method (checked before class creation), before BadHandler is ever bound as a name. __init_subclass__ can also raise on a missing method, but only after type.__new__ has already built the class object; the difference matters when the class body itself needs to be inspected or rewritten (adding methods, changing bases) rather than merely rejected.

Trade-offs & pitfalls

• The honest default: for plain subclass registration, prefer __init_subclass__. It is simpler, needs no metaclass= boilerplate, and composes more predictably with multiple inheritance. Choosing a metaclass for a job __init_subclass__ already does is over-engineering that a senior reviewer will flag.
• Metaclasses justify themselves when you need pre-creation control (rewriting the namespace, enforcing constraints across classes that do not share a common base, or making the metaclass itself the unit of composition across independent hierarchies) or when integrating with a library that already defines its own metaclass and yours must cooperate with it (both must derive from a common metaclass, or Python raises TypeError: metaclass conflict).
• Name collisions in a class name -> class registry silently overwrite an earlier entry; keying by (module, name) or storing a list per name avoids losing an earlier plugin with the same class name defined in a different module.
• A metaclass applies to every subclass in the hierarchy automatically, which is a feature for enforced registration but a footgun if a downstream user subclasses without realizing they inherited the metaclass's side effects (an unexpected registry entry, or an unexpected validation failure at class-definition time rather than at instantiation time).
• Complexity: registration is $O(1)$ per class creation (a dict insert); the registry lookup is $O(1)$. Edge cases: skip the sentinel base correctly (an off-by-one in the skip condition silently registers the abstract base itself); decide up front whether abstract intermediate classes in a deeper hierarchy should also be excluded, not just the root.