Python and Pandas for Data Analysis Questions

Direct answer

Write each stage (read, filter, impute, feature creation, write) as a small pure function that takes a DataFrame and returns one, and compose them with .pipe() so the pipeline reads top-to-bottom as its own table of contents. Each function can then be unit-tested in complete isolation with a two- or three-row DataFrame, with no need to run the rest of the pipeline or touch the filesystem.

Approach

import pandas as pd

def read_csv(path):
    return pd.read_csv(path, parse_dates=["event_time"])

def filter_rows(df):
    return df[df["event_time"].notna() & (df["revenue"] > 0)]

def impute(df):
    df["country"] = df["country"].fillna("unknown")
    df["revenue"] = df["revenue"].fillna(0)
    return df

def create_features(df):
    # cast the period to str: parquet's partition_cols can't hash a raw Period dtype column
    df["month"] = df["event_time"].dt.to_period("M").astype(str)
    df["rev_per_item"] = df["revenue"] / df["items"].replace(0, 1)
    return df

def run_pipeline(input_path, output_dir):
    result = (
        read_csv(input_path)   # the chain starts from the function that produces the first DataFrame,
        .pipe(filter_rows)     # not from an undefined variable piped through read_csv
        .pipe(impute)
        .pipe(create_features)
    )
    result.to_parquet(output_dir, partition_cols=["month", "country"], engine="pyarrow")
    return result

Key points:

• The chain has to start from a call that actually produces a DataFrame (read_csv("input.csv")), not from .pipe()-ing an undefined name through a reader function; .pipe() threads an existing DataFrame through the next step, it can't conjure the first one.
• create_features casts month to a plain string before it becomes a partition column: to_period("M") produces a pandas-specific Period dtype, and PyArrow's write_to_dataset (which to_parquet(partition_cols=...) uses under the hood) cannot hash that extension dtype as a partition key, so writing the raw Period column raises ArrowNotImplementedError: Keys of type extension<pandas.period<ArrowPeriodType>>.
• impute and create_features each receive the DataFrame that the previous .pipe() call already produced fresh (from filter_rows's boolean-indexed slice, which is itself already a new object under Copy-on-Write), so a plain df["col"] = ... inside them is an ordinary column assignment, not the chained-slice pattern that pandas 3.0 flags.

Worked example

pd.DataFrame({
    "event_time": ["2026-01-05", "2026-01-20", "2026-02-01", None, "2026-02-10"],
    "revenue": [10.0, 0.0, 25.0, 15.0, -5.0],
    "items": [2, 1, 0, 3, 1],
    "country": ["US", None, "DE", "US", "FR"],
}).to_csv("input.csv", index=False)

result = run_pipeline("input.csv", "out/")
print(result)

Output (verified against pandas 3.0.3, pyarrow):

  event_time  revenue  items country    month  rev_per_item
0 2026-01-05     10.0      2      US  2026-01           5.0
2 2026-02-01     25.0      0      DE  2026-02          25.0

Of the five input rows, three are filtered out by filter_rows: the row with revenue == 0, the row with a missing event_time, and the row with negative revenue (a data-quality problem, not something impute should paper over). The items == 0 row on 2026-02-01 survives filtering (its revenue is positive) and rev_per_item correctly falls back to dividing by 1 instead of 0 via .replace(0, 1). Writing this result with partition_cols=["month", "country"] succeeds and reading it back with pd.read_parquet("out/") reproduces the same two rows.

Complexity and edge cases

Complexity: each stage is a single O(n) pass over the rows it touches; .pipe() itself adds no algorithmic cost, it is function composition. Memory: each .pipe() call's result is a fresh DataFrame (a shallow copy for the untouched columns plus whatever the step added or filtered), so peak memory during the chain is bounded by roughly the size of the largest single intermediate, not the sum across every stage, since Python frees each intermediate once the next step has consumed it.

Edge cases: an items value of exactly 0 needs the .replace(0, 1) guard or rev_per_item divides by zero; a missing event_time is dropped by the filter rather than imputed, since a synthetic timestamp would be a worse assumption than dropping the row; an already-unknown or already-zero-filled value passing through impute a second time is a no-op, so the function is safe to call more than once (idempotent) if the pipeline is ever re-run on partially-processed data.

Trade-offs and pitfalls

Unit-testing each function in isolation is the main payoff of this shape: create_features can be tested with a two-row DataFrame asserting the exact rev_per_item value and that month is a plain string (not a Period), without needing a real CSV file or a filesystem to write parquet to; impute can be tested by asserting that a null country becomes "unknown" and a null revenue becomes 0, again with no dependency on the rest of the chain. Mocking I/O becomes trivial too, since read_csv and the final to_parquet call are the only two functions that touch the filesystem, everything in between is a pure DataFrame-to-DataFrame transformation that pytest fixtures can exercise directly. The pitfall in a .pipe() chain specifically: because each step only runs when the chain executes, not when it's defined, a bug like the undefined-starting-variable above or a missing import inside a step function stays completely invisible until someone actually runs the full pipeline end-to-end, which is exactly why testing each function independently (rather than trusting that the chain "looks right") is the point of building it this way in the first place.

Direct answer

Copy-on-Write (CoW) is now mandatory in pandas 3.0, it cannot be turned off. Under CoW, every pandas object behaves as if it holds independent data: reading through a result that used to be called a "view" is free, but the moment either the original or the derived object is written to, pandas makes a physical copy at that instant so the two can never silently diverge into each other. The old question "is this a view or a copy?" is no longer something you can predict from the syntax you used, and you no longer need to predict it, because a write can never leak across objects either way.

Structured elaboration

Before pandas 2.0, some operations (plain-column selection, some slicing) returned a "view" backed by the same underlying NumPy array as the source. Whether mutating that view also mutated the source depended on internal implementation details the documentation itself called unreliable, SettingWithCopyWarning existed precisely because pandas could not always tell you which case you were in. That warning was heuristic: it could fire on code that was actually fine, and stay silent on code that silently corrupted data.

Under mandatory CoW, SettingWithCopyWarning does not exist anymore. This was verified directly: pandas.errors.SettingWithCopyWarning raises AttributeError, it is not deprecated-but-present, it is gone, because the ambiguous partial-mutation scenario it warned about can no longer happen.

The two-statement form now silently succeeds on a copy

import pandas as pd

df = pd.DataFrame({'a': [1, 2, 3, 4, 5], 'b': [10, 20, 30, 40, 50]})
subset = df[df['a'] > 2]
subset['b'] = 999

Verified by running this exact code: no warning, no error, subset['b'] is 999 for the matching rows, and df is completely unchanged. This is the single biggest behavior change from the pre-3.0 mental model. What used to be "sometimes this mutates df, sometimes it doesn't, watch for the warning" is now deterministic: it never mutates df. subset is an independent object the instant it is assigned to a variable, so writing to it can only ever affect subset.

Only the single-expression chained form warns, and it is only a warning

import pandas as pd

df = pd.DataFrame({'a': [1, 2, 3, 4, 5], 'b': [10, 20, 30, 40, 50]})
df[df['a'] > 2]['b'] = 999

Verified by running this exact code: it raises no exception, df is unchanged, but a ChainedAssignmentError is emitted. Its class hierarchy (ChainedAssignmentError.__mro__) confirms it is a subclass of the built-in Warning, not a hard exception, so by default it prints to stderr and execution continues, exactly like any other uncaught Python warning. Write "warns via ChainedAssignmentError," not "raises ChainedAssignmentError," since the latter implies the program stops, and by default it does not.

Pandas can only catch this pattern in the single-expression form because that is the only shape where its internals can observe, within one call, that the object receiving the write was produced moments earlier by an indexing operation with no other references. Once the intermediate result is stored in a variable across two statements, as in the block above, subset is just an ordinary DataFrame by the time you write to it, so no warning path fires at all. That means the coverage of ChainedAssignmentError is narrower than most people assume: the two-statement form, which is the shape most engineers actually write, and the one most "avoid SettingWithCopyWarning" tutorials describe, is exactly the one pandas 3.0 cannot warn about.

Worked example

import pandas as pd

df = pd.DataFrame({'a': [1, 2, 3, 4, 5], 'b': [10, 20, 30, 40, 50]})

# two-statement chained assignment: silently succeeds on a copy
subset = df[df['a'] > 2]
subset['b'] = 999
print(df)          # unchanged
print(subset)       # 'b' is 999 for rows where a > 2

# single-expression chained assignment: warns via ChainedAssignmentError, does not halt
df[df['a'] > 2]['b'] = 999
print(df)          # still unchanged

# the correct pattern: a single .loc assignment
df.loc[df['a'] > 2, 'b'] = 999
print(df)          # 'b' is 999 for rows where a > 2, this is the only reliable pattern

Running this exact code against pandas 3.0.3: the two-statement block leaves df's b column as [10, 20, 30, 40, 50] (unchanged) and subset's b column as [999, 999, 999] for the three matching rows. The single-expression line prints a ChainedAssignmentError warning to stderr and also leaves df unchanged. Only the final .loc[...] = ... line mutates df, giving b = [10, 20, 999, 999, 999].

Trade-offs and pitfalls

• Do not treat "no warning" as a correctness signal anymore, since the two-statement chained form now produces neither a warning nor a mutation. Correctness depends entirely on habitually writing through a single .loc/.iloc assignment, df.loc[mask, 'col'] = value, which is the only pattern guaranteed to mutate df in both the pre-3.0 and 3.0 worlds. Treat it as the default even outside interview settings.
• Study material that says "chained assignment sometimes works, watch for SettingWithCopyWarning" is wrong on both halves for pandas 3.0: the warning does not exist, and "sometimes works" is not a real behavior either, the two-statement form is now unconditionally a no-op on the original.
• Because CoW defers the physical copy until the first write, code that reads heavily but writes rarely is cheaper than eager-copy pandas used to be, you get view-like sharing on the read path for free. But once you do write, that write pays a full copy of the affected block at that moment, so a loop that repeatedly mutates the same derived frame can pay a copy cost on every iteration if pandas cannot prove there is only one reference to the underlying data. The fix is the same discipline as always: do the transformation as one vectorized .loc assignment, or build the result with .assign() or pd.concat rather than mutating iteratively.
• To avoid unnecessary copies and large temporaries on genuinely large data: stop calling .copy() defensively "just in case," CoW already protects you from cross-object mutation, so a defensive copy only doubles memory for no safety benefit; prefer a single .loc[...] = ... write over any chained form so you pay exactly one copy, not an unpredictable number; drop columns you do not need before doing a wide operation rather than after; and check df.memory_usage(deep=True) and downcast dtypes (categorical for low-cardinality strings, smaller integer or float widths), since dtype choice affects the size of any eventual copy far more than CoW's own bookkeeping does.

Direct answer

Reproduce both jobs against the same input snapshot, then work down a fixed checklist in order of how cheap and how likely each check is: schema and types first, then null handling, then sort/sample determinism, then join and aggregation semantics, then floating-point precision, then timezones and parsing, then any user-defined function (UDF). Row counts and a deterministic row-level checksum bracket the whole process: run them first to confirm there is a real divergence, and again after each fix to confirm it's closed.

Approach

Concrete checks, in the order to run them:

1. Schema and types. Compare column names, order, and dtypes between the pandas and PySpark (the distributed, Spark-based analogue of pandas) outputs. A pandas int column that Spark reads as a string, or vice versa, is a common and cheap-to-spot first divergence.
2. Null handling. Count nulls and null-like sentinel strings ('', 'NULL', 'NA') per column in both. Pandas and Spark can default to different dropna/fill behavior in equivalent-looking operations.
3. Sorting and sampling determinism. Any operation that depends on row order (deduplication, head(), a sampled preview) needs the same sort keys and tie-breakers in both engines, and any sampling needs the same seed.
4. Joins and deduplication semantics. Confirm join types match (inner/left/right) and that null-key handling matches; a duplicate key on one side silently fans out rows differently between engines if the join type or dedup step isn't identical.
5. Aggregation and floating-point precision. Compare groupby aggregates with the same rounding, and use a small epsilon tolerance rather than exact equality for floating-point sums.
6. Timezones and parsing. Confirm timestamp parsing and timezone handling match, and check for locale differences (decimal comma vs. decimal point) if either source is locale-sensitive.
7. UDFs and implicit conversions. Audit any custom function or .apply()/Spark UDF for behavior that differs between a vectorized pandas op and a row-at-a-time Spark UDF.
8. Row counts and a deterministic checksum. Bracket every other check: if row counts and a per-row checksum already agree, stop; if they don't, narrow which transformation step introduced the divergence.

from pyspark.sql import functions as F

# Side-by-side sketch of the same checks in each engine.
# Assumes df_pd and df_spark already hold the same input snapshot; not meant to run
# standalone (the fully self-contained, verified version is the worked example below).

# pandas side
print(df_pd.dtypes)
print(df_pd.isnull().sum())
row_hashes = pd.util.hash_pandas_object(df_pd, index=False)
print("checksum:", int(row_hashes.sum()))

# pyspark side (same checks, Spark API)
df_spark.printSchema()
df_spark.select([F.count(F.when(F.col(c).isNull(), c)).alias(c) for c in df_spark.columns]).show()
# Spark's own deterministic row hash: F.hash(*cols) or F.sha2(F.concat_ws("|", *cols), 256)

Key points:

• A per-row checksum is only useful for reconciliation if it is itself reproducible: Python's built-in hash() is randomized per process by default (hash randomization) and gives a different value on every run, so a checksum built from ''.join(row).apply(hash) cannot be compared across two separate job runs even on identical data. pandas.util.hash_pandas_object is deterministic across runs and is the right tool for this, and Spark's F.hash or F.sha2 is its equivalent.
• Reading types with df.dtypes versus df.printSchema() is the fastest single check to run first, since a surprising number of "different results" incidents trace back to one side reading a column as text and the other as numeric.

Worked example

import pandas as pd

df_pd = pd.DataFrame({
    "id": [1, 2, 3],
    "name": ["alice", "bob", None],
    "amount": [10.5, 20.0, 30.25],
})

print(df_pd.dtypes)
print(df_pd.isnull().sum())
row_hashes = pd.util.hash_pandas_object(df_pd, index=False)
print("checksum:", int(row_hashes.sum()))

Output (verified against pandas 3.0.3, and stable across repeated runs, which is the property being demonstrated):

id          int64
name          str
amount    float64
dtype: object

id        0
name      1
amount    0
dtype: int64

checksum: 1720822316984681584

Note the name column's dtype prints as str, pandas 3.0's dedicated string dtype (the default for text columns as of 3.0, replacing the old catch-all object dtype): if the equivalent Spark column arrives typed as Spark's StringType, this pandas-side dtype check is exactly what a naive "are these the same type" comparison would need to account for, since str and object can otherwise look interchangeable in older pandas code that still checks dtype == object.

Trade-offs and pitfalls

A checksum built from df.astype(str).agg(''.join, axis=1).apply(hash) is a pattern that shows up often in ad hoc reconciliation scripts, and it has two independent problems worth knowing by name rather than rediscovering under pressure: first, pandas 3.0's default string dtype leaves a missing value as an actual float NaN (not-a-number) even after astype(str), rather than stringifying it to the literal text "nan", so ''.join on a row containing a null raises TypeError: sequence item ...: expected str instance, float found unless nulls are explicitly filled first (.astype(str).fillna("NULL")); second, even once that's fixed, Python's hash() is randomized per process, so the resulting "checksum" changes on every run and cannot be diffed against a checksum computed in a separate process (a separate script invocation, a scheduled job's next run, or the Spark side entirely). pandas.util.hash_pandas_object avoids both problems: it handles nulls correctly and produces the same hash for the same data on every run. Once a checksum mismatch is confirmed, the fastest way to localize the cause is a binary search through the pipeline: save intermediate outputs after each transformation in both engines and diff progressively earlier stages until the two agree, rather than guessing which of the eight checks above is the culprit.

Direct answer: NumPy vectorized aggregation (integer-encode the category labels, then np.bincount) can meaningfully outperform pandas.groupby for simple numeric reductions like sum, count, or mean on a single column, because it skips the general-purpose machinery groupby carries for mixed dtypes, missing values, and multi-column, multi-aggregation pipelines. Once your aggregation needs any of that generality, pandas.groupby becomes the better choice again, both for correctness and for how much code you'd otherwise have to hand-write and maintain.

Structured elaboration, worked end to end (verified, pandas 3.0.3, numpy 2.5.1, seed 0, 20 rows, 4 groups, fully reproducible):

When NumPy wins: you have one (or a few) numeric column(s), a single simple reduction (sum, count, mean), category labels that can be mapped to small contiguous integers, and no missing values to reason about. np.bincount computes a per-group sum or count in one pass with none of groupby's dtype dispatch, index bookkeeping, or Python-level overhead per group.

import numpy as np
import pandas as pd

rng = np.random.RandomState(0)
N, G = 20, 4
groups = rng.randint(0, G, size=N).astype(np.int32)
values = rng.randn(N)

sums = np.bincount(groups, weights=values)   # per-group sum, one pass
counts = np.bincount(groups)                 # per-group count
means = sums / counts                        # handle any zero-count group separately
print('sums =', sums)
print('counts =', counts)
print('means =', means)

Verified output: sums = [-0.4656, -0.0095, 4.4562, 1.9245], counts = [6, 4, 3, 7], means = [-0.0776, -0.0024, 1.4854, 0.2749] for groups 0-3.

If you need to accumulate in place or handle repeated indices explicitly rather than via weights=, np.add.at is the safe alternative:

acc = np.zeros(G, dtype=np.float64)
np.add.at(acc, groups, values)
print('np.allclose(acc, sums) =', np.allclose(acc, sums))

Verified: acc equals sums above exactly (np.allclose(acc, sums) is True).

Correctness check against pandas.groupby, same data:

df = pd.DataFrame({'cat': groups, 'val': values})
means_pd = df.groupby('cat')['val'].mean()
print('means_pd (sorted by group) =', means_pd.sort_index().to_numpy())
print('max abs difference =', np.max(np.abs(means_pd.sort_index().to_numpy() - means)))

Verified output: means_pd (sorted by group) is [-0.077606, -0.002381, 1.485384, 0.274924], identical to means above to 15 decimal places, max absolute difference 5.6e-17 (floating-point rounding noise, not a real difference). This confirms both approaches compute the identical aggregation on real data, so the choice between them is purely about performance and generality, not correctness. (Wall-clock timing numbers are intentionally omitted here since they are hardware- and environment-dependent and this small fixture is too small to be a fair benchmark anyway; the mechanism below is what actually explains the gap at scale.)

Why NumPy is faster for this narrow case: np.bincount is a single specialized C loop over a pre-encoded integer array with no per-row dtype checks, no handling of missing group labels, and no construction of an intermediate grouped-object representation. pandas.groupby builds a grouping object, handles arbitrary key types (strings, categoricals, tuples of columns), preserves index alignment, and supports dozens of aggregation functions and multi-column .agg() dictionaries, all of which cost something even when your specific case doesn't need it.

When pandas.groupby is preferable despite the difference

• Mixed dtypes or non-numeric group keys, groupby handles string, categorical, and datetime keys directly; you would have to build your own integer encoding by hand for NumPy.
• Missing values that need to be preserved or explicitly excluded, groupby has consistent, well-tested NaN (not-a-number, pandas' missing-value marker) handling; a hand-rolled bincount pipeline requires you to reimplement that correctly.
• Multiple different aggregations per column or across columns in one pass, e.g. df.groupby('cat').agg({'sales': 'sum', 'user': 'nunique'}), this is one readable call in pandas versus several hand-written NumPy passes.
• Time-aware or hierarchical grouping (resampling, groupby(level=...) on a MultiIndex).
• Code clarity and fewer manual edge-case bugs: division-by-zero on empty groups, sort/stability guarantees, and correct alignment back to the original frame are all handled for you.

Complexity: both approaches are O(n) in the number of rows for a single-pass reduction like sum or count; the difference is in the constant factor, not the asymptotic order, bincount has a smaller constant because it avoids groupby's per-row dispatch and object overhead. Memory is O(G) for both, where G is the number of distinct groups (the accumulator array in one case, the grouped result index in the other), so this trade-off is about per-row overhead and generality, not about complexity class.

Trade-offs and pitfalls

• Reach for the NumPy path only when you can cleanly map categories to small contiguous integers up front (pd.factorize does this for you and returns exactly what bincount needs) and your aggregation genuinely is that simple. Building and maintaining that encoding for a case that doesn't actually need the speed is wasted complexity.
• Benchmark on your real data and hardware before committing to the NumPy path in a production pipeline, the size of the gap depends heavily on group count, row count, and dtype, and groupby's internal implementation improves across pandas releases.
• A common middle ground: use pandas.groupby for exploratory work and multi-aggregation reporting, and drop to a NumPy pre-aggregation pass only in a narrow, profiled hot path where it has been measured to matter.

Direct answer

Use pd.merge_asof with direction='backward' to attach each trade to the most recent price
at or before its own timestamp, passing by='symbol' to restrict matches to the same symbol.
The one hard requirement is that both DataFrames are sorted by the on timestamp column
itself, not by symbol first: merge_asof performs a single sorted-order scan, and by only
restricts which rows are eligible matches within that scan, it does not re-sort per group.

Structured elaboration

• Sort by the on key only. Sorting by ['symbol', 'timestamp'] groups rows by symbol
  first, which breaks global timestamp order the moment two symbols' events interleave in
  time, and merge_asof raises ValueError: left keys must be sorted (or silently returns
  wrong matches on older pandas that did not check). Sort by 'timestamp' alone; by='symbol'
  handles the per-symbol restriction internally.
• direction controls which side of "at or before" you want: 'backward' (the default)
  finds the last price at or before the trade; 'forward' finds the next one at or after;
  'nearest' finds whichever is closer in time.
• by= restricts matches to the same group, so an AAPL trade can never match an MSFT
  price, without you having to loop over symbols yourself.

Worked example

import pandas as pd

prices = pd.DataFrame({
    "timestamp": pd.to_datetime(["2025-01-01 09:30:00", "2025-01-01 09:31:00", "2025-01-01 09:30:00"]),
    "symbol": ["AAPL", "AAPL", "MSFT"],
    "price": [150.0, 150.5, 250.0],
})

trades = pd.DataFrame({
    "timestamp": pd.to_datetime(["2025-01-01 09:30:30", "2025-01-01 09:31:10", "2025-01-01 09:29:00"]),
    "symbol": ["AAPL", "AAPL", "MSFT"],
    "quantity": [10, 5, 20],
})

# sort by the 'on' key alone; do NOT sort by symbol first (see pitfalls)
prices = prices.sort_values("timestamp")
trades = trades.sort_values("timestamp")

merged = pd.merge_asof(
    trades,
    prices,
    on="timestamp",
    by="symbol",
    direction="backward",
    suffixes=("", "_price"),
)
print(merged)
#             timestamp symbol  quantity  price
# 0 2025-01-01 09:29:00   MSFT        20    NaN
# 1 2025-01-01 09:30:30   AAPL        10  150.0
# 2 2025-01-01 09:31:10   AAPL         5  150.5

The MSFT trade at 09:29 gets not-a-number (NaN) for price: its only MSFT price observation
is at 09:30, which is AFTER the trade, so direction='backward' correctly finds nothing.

Key points

• by='symbol' restricts matches without a manual per-symbol loop.
• direction and tolerance (a max allowed time gap) both shape which price counts as "close
  enough"; use tolerance if a match hours old should be treated as no match at all.

Complexity

O(n log n) for the sort, which dominates; the merge_asof scan itself is O(n + m) given
sorted input (n trades, m prices).

Edge cases

A trade earlier than any price observation for its symbol produces NaN in the price columns
(decide: drop, forward-fill from a seed price, or treat as invalid upstream); a symbol present
in one side but not the other never matches, by design; and duplicate timestamps within a
symbol on the price side, where direction='backward' takes the last one in sort order, so add
an explicit tiebreaker column if that ordering needs to be deterministic beyond timestamp.

Trade-offs and pitfalls

The pitfall worth naming explicitly, because it is easy to get backwards: sorting by
['symbol', 'timestamp'] looks like the natural thing to do when you are about to pass
by='symbol', but it is wrong. merge_asof needs the on column globally sorted across the
whole DataFrame; grouping by symbol first breaks that the moment two symbols' timestamps
interleave (an MSFT trade at 09:29 sorted after an AAPL trade at 09:31 under a
symbol-then-timestamp sort). Sort by timestamp alone and let by do the grouping.

• If you want only exact-timestamp matches instead of "most recent at or before", use a plain
  inner merge on both timestamp and symbol instead of merge_asof.
• Consider whether an unmatched trade (NaN price) should be dropped, flagged, or filled from
  a known opening price, since silently keeping NaN in downstream arithmetic will propagate
  it.