DoorDash Machine Learning Engineer Interview Preparation Guide - Senior Level

Use a pytest fixture that creates an isolated temporary directory (using tempfile.TemporaryDirectory or pytest's tmp_path_factory) and yields a pathlib.Path to the test; cleanup happens automatically after the test. This keeps model artifact writes isolated.

import shutil
import tempfile
from pathlib import Path
import pytest

@pytest.fixture
def isolated_model_dir():
    """
    Creates an isolated temporary directory for writing model artifacts.
    Yields a pathlib.Path pointing to the dir and ensures cleanup after test.
    """
    td = tempfile.TemporaryDirectory(prefix="test_model_artifacts_")
    path = Path(td.name)
    try:
        yield path
    finally:
        # ensure cleanup even if test fails
        td.cleanup()
        # extra safety: remove residual files if any
        if path.exists():
            shutil.rmtree(path, ignore_errors=True)

Example test using the fixture:

def test_save_and_load_model(isolated_model_dir):
    model_file = isolated_model_dir / "model.pt"
    # pretend to save a model artifact
    model_file.write_text("fake-model-bytes")
    assert model_file.exists()

    # simulate a load operation
    content = model_file.read_text()
    assert content == "fake-model-bytes"

Key points:

• Yields a Path to let tests read/write easily.
• Uses TemporaryDirectory for automatic cleanup; finalizer ensures removal on failures.
• Keeps tests isolated and deterministic; avoids polluting repo or CI workspace.

Direct answer

Sustaining velocity over a year without letting debt or burnout win requires governance that's structural, not a matter of good intentions each sprint. That means a protected capacity reservation that's defended the same way an incident response commitment would be, sprint rituals that surface debt before it's invisible, incentives that actually reward paying it down, and leading metrics that catch decay while it's still cheap to fix rather than after it becomes an outage.

Structured elaboration

Capacity allocation. I set a fixed percentage of every sprint, for example 20%, reserved specifically for maintenance and debt work, and I protect it the same way I'd protect an SLA (service-level agreement, a commitment on how quickly something gets addressed): it is not the first thing cut when a feature deadline gets tight. It applies per engineer, not as a single pooled bucket someone else absorbs.

Sprint-level tactics. Each sprint, one rotating team member acts as the toil owner, responsible for triaging that reserved capacity into either same-sprint quick fixes or larger scheduled items. Sprint planning includes a short debt-review slot where new debt items get scored (similar to feature prioritization) and slotted into either the reserved capacity or a future sprint, so debt doesn't just accumulate as unscored tickets. On top of the dedicated capacity, I reinforce a boy-scout norm in code review, leaving code measurably better than you found it on the way through, as small continuous paydown alongside the larger scheduled work.

Incentives. I make debt reduction visible in the same channels that reward feature delivery: performance review templates explicitly credit reliability and toil-reduction contributions alongside shipped features, not as an afterthought, and each quarter I highlight one concrete toil-reduction win in a team or org forum, the same way a feature launch would get highlighted. If the only thing that gets celebrated is shipped features, the reserved capacity erodes in practice even if it's still on paper.

Metrics. I track leading indicators, not just lagging ones: the trend in on-call page volume, the trend in CI (continuous integration) flake rate, and the age distribution of the debt backlog, specifically the share of tickets older than 90 days. A rising backlog age or a flattening page-volume trend is a decay signal well before it turns into an incident count worth reacting to.

Preventing decay while keeping delivery fast. I review these trends quarterly and adjust the reserved percentage based on the signal, not on anecdote: if the leading indicators worsen for two consecutive quarters, the reservation goes up temporarily; if they hold steady or improve for two consecutive quarters, it stays where it is. This keeps the governance responsive without letting a single bad week trigger a policy change.

Worked example

For an organization of three squads, fifteen engineers total, running two-week sprints: each engineer reserves two days out of the ten working days in a sprint, about 20% of capacity, for debt and maintenance work. Each squad rotates a toil owner every sprint who triages that reserved time across the squad. Sprint planning includes a ten-minute debt-review where new items get a rough severity score and either get pulled into this sprint's reserved time or added to a backlog with an age tracked from creation.

Over a year, the org tracks on-call page volume and CI flake rate quarterly. Starting from roughly 24 pages a week and an 8% flake rate in the first quarter, both trend down as the reserved capacity retires the noisiest recurring sources, reaching roughly 18 pages a week and 6% flake rate by the second quarter, 13 pages and 4% by the third, and under 10 pages a week with a 2% flake rate by the fourth. Debt-backlog age also improves: the share of tickets older than 90 days drops from about a third of the backlog in quarter one to under a tenth by quarter four. Because both leading indicators are improving quarter over quarter, the reserved percentage stays at 20% rather than increasing; if either trend had reversed for two straight quarters, the governance review would have raised the reservation to 25% until it recovered.

Trade-offs and pitfalls

• A fixed percentage that's never revisited eventually becomes wrong in one direction or the other, wasteful once debt is under control, or inadequate if the organization is generating debt faster than the tax retires it; the quarterly review against real trend data is what keeps it calibrated.
• Rotating the toil-owner role avoids burnout concentrating on one person, but only if the rotation is genuinely enforced rather than perpetually landing on whoever is newest or most junior on the team.
• An incentive structure that only rewards feature shipping quietly undermines the capacity policy in practice even while it stays true on paper; the performance-review language has to actually reflect debt work as valued, or the reservation erodes as people self-select back into feature work to protect their standing.
• Leading indicators like page volume can be gamed by simply silencing a noisy-but-real alert rather than fixing its cause; pairing the page-volume trend with an independent incident-severity trend guards against optimizing the metric instead of the outcome.
• Overcorrecting the reserved percentage sharply after a single bad quarter creates its own whiplash and burnout; the two-consecutive-quarter rule before adjusting is there specifically to avoid reacting to noise.

Situation: I worked in a matrix organization on a customer data initiative where Product, Engineering, Legal, and Sales all had a stake.

Task: I had to keep the project moving even though I did not control everyone’s priorities or timelines.

Action: I set up a clear operating model. I defined the business outcome, named one accountable owner, and created a decision log so people could see what was agreed and what was still open. I also built a regular cadence with the key managers so issues surfaced early instead of at the end. When priorities conflicted, I tied the discussion back to business impact. For example, when Sales wanted a faster release and Legal needed more review time, I proposed a phased launch so we could move forward without increasing risk.

Result: The initiative stayed on track because decisions were made in the open and each manager understood how their concerns were being handled.

The main lesson was that in a matrix, progress depends less on direct authority and more on clarity, trust, and disciplined communication.

Direct answer
Ramping someone from day one to their first shipped work is a deliberate sequence, not a single onboarding checklist: assess what they actually already know, give them small real tasks with tight review loops before a full feature, gradually widen the scope of ownership, and define upfront what "shipped" and "done" mean so the finish line is unambiguous. The plan should look different depending on who's arriving, not just be a fixed template applied to everyone.

Structured elaboration

The default arc

• First few days: orient and assess. Don't assume a blank slate; find out what they already know so you're not re-teaching things or, worse, skipping things they actually need.
• Early tasks: small, real, low-blast-radius work with fast, close review. The goal here is confidence and calibration to the team's standards, not speed.
• Middle stretch: progressively larger scope with more independence, review shifting from "check everything" to "check the risky parts."
• First real shipped piece: something end-to-end they own, with you available but not doing it alongside them, and a clear definition of "done" agreed before they start, so success isn't a moving target.

Adapting the plan to who's actually arriving

This is where a generic checklist breaks down, and it's the part that separates a senior answer:

• A contractor under least-privilege or compliance constraints: access is scoped down from day one, so the plan has to work around what they legitimately can't see or touch, and documentation often needs to be more explicit since they can't casually ask around as easily as a full-time hire embedded in the org.
• A career-changer from an adjacent discipline (a backend engineer moving into data engineering, a research scientist moving into production ML): they're not a blank slate, they have real transferable skills. The plan should explicitly identify what carries over and target ramp-up specifically at the actual new-domain gaps, not restart from zero the way you would for someone with no relevant background.
• A cohort of remote interns rather than one hire: 1:1 pairing time doesn't scale to a group. The plan shifts toward a shared structured curriculum, peer learning between the interns, and scheduled office hours, with 1:1 time reserved for the things that genuinely need it.
• A remote hire versus a senior IC joining: a remote hire needs more of everything written down explicitly, since the informal hallway learning that fills gaps for an in-person hire doesn't happen by accident. A senior IC's gap is usually organizational context and relationships, not raw skill, so their plan should be lighter on procedural scaffolding and heavier on introductions, context on how decisions get made, and where the landmines are.

Worked example

Situation

I mentored someone joining as an individual contributor with solid general skills but no exposure to our specific stack or codebase, with a goal of them shipping one real, complete piece of work within their first several weeks.

Action

Week one was mostly orientation and a short assessment task to see where they actually stood, not a generic reading list. From there, I gave them a small real bug fix with a tight review loop so they got fast, specific feedback on our conventions early, before those habits calcified the wrong way. Over the following weeks the scope widened: a small self-contained feature with me reviewing closely, then a larger piece with me available but stepping back from line-by-line review, focusing instead on the riskiest parts of the design.

Result

They shipped a real, complete piece of work end-to-end within the target window, with a review pass that looked much closer to how we review any other team member's work by that point, which was the actual signal of readiness, not just that the calendar had passed.

Trade-offs & pitfalls

• Treating every new hire's plan as the same template. A junior mentor runs the same onboarding for a contractor, a career-changer, an intern cohort, and a senior IC. A senior mentor adapts the shape of the plan to who's actually arriving, because the actual gap being closed is different in each case.
• Under-scoping early tasks out of excessive caution, or over-scoping out of impatience. Both undermine the confidence-building purpose of the early stretch: too small and it's condescending or boring; too large too soon and the first review becomes overwhelming and demoralizing.
• Not defining "done" up front. Ambiguity about what counts as finished either causes needless rework or lets something ship that isn't actually ready, and both erode trust in the mentoring relationship.
• Ignoring the constraints a nontraditional hire is actually operating under. Applying a full-access, in-person, junior-IC plan to a least-privilege contractor or a remote hire sets them up to fail on logistics that have nothing to do with their actual skill.

Direct answer: For a categorical feature with millions of unique values, there's no single right encoding; the real decision is a trade-off between memory, collision risk, interpretability, and which model family will consume the feature, and the strongest production answers usually combine two of frequency/hashing/target-encoding/learned-embeddings rather than picking exactly one.

Structured elaboration:

• Frequency (count) encoding: replace each category with how often it appears. Cheap, fixed-size regardless of cardinality, but collapses distinct categories that happen to occur equally often into the same encoded value, and needs a defined behavior for unseen categories at serving time (typically the global or a smoothed default rate).
• The hashing trick: hash the category string into a fixed number of buckets. Memory is bounded and predictable (you choose the bucket count up front), and it naturally handles unseen categories (any new string still hashes somewhere), at the cost of collisions where two different categories share a bucket and become indistinguishable to the model. The collision rate is a direct, computable function of how many distinct categories you're hashing into how many buckets, which lets you size the hash space deliberately rather than guessing.
• Target encoding with smoothing: replace a category with a (regularized) estimate of the target given that category. Very informative, but must be computed out-of-fold or it leaks the label into the feature; also needs a smoothing/shrinkage term so rare categories don't get a noisy, over-confident estimate.
• Learned embeddings: a neural network learns a dense vector per category during training. Captures rich structure and similarity between categories, at the cost of needing a training loop, careful embedding-size choice, and a defined cold-start behavior for categories the embedding table has never seen.

Model-family fit matters: tree-based models handle raw high-cardinality categoricals reasonably natively (or via target/frequency encoding) without needing scaling; linear models need a numeric, roughly-comparable-scale representation (hashing or target encoding); deep models are the natural home for learned embeddings.

Worked example: Hashing 100,000 distinct category values into 2^18 (262,144) buckets: two related but distinct quantities are worth separating here. The expected number of occupied buckets (buckets holding at least one category) follows the standard balls-into-bins expectation,

$\text{expected occupied buckets} \approx n_{\text{buckets}} \times \left(1 - \left(\frac{n_{\text{buckets}}-1}{n_{\text{buckets}}}\right)^{n_{\text{categories}}}\right)$

which for these numbers gives about 83,100, matching an empirical run almost exactly. But that is a bucket count, not the collision rate over categories, since a bucket occupied by two or more categories still counts once. The quantity that actually determines the model's collision risk is the per-category collision probability, 1 - (1 - 1/n_buckets)^(n_categories - 1) ≈ 1 - e^(-n_categories / n_buckets), which for 100,000 categories into 262,144 buckets is about 31.7%: roughly 68,300 of the 100,000 categories land alone in their own bucket, and the remaining ~31,700 (about 32%, not 16.8%) share a bucket with at least one other category and become indistinguishable to the model. This also matches an empirical run closely. Doubling the bucket count to 2^19 (524,288) drops the collision rate to about 17-18%, which is the concrete lever you have if collisions are hurting accuracy: it's a memory-versus-accuracy dial, not a fixed property of hashing.

Trade-offs and pitfalls: Unseen-value handling differs sharply by method: frequency and target encoding need an explicit fallback (a default/global value) for categories never seen in training, while hashing handles unseen values "for free" (they just land in some bucket) but at the cost of always having some baseline collision rate even for previously-seen categories. Target encoding is the highest-leakage-risk of the four if it isn't computed out-of-fold with proper smoothing.

Direct answer

Feature engineering (windowing, event alignment), a resampling or weighting strategy validated inside cross-validation, and a time-aware CV setup all need to work together for a rare, time-ordered event like equipment failure, since getting any one piece wrong (leaking future information, or evaluating with a metric that doesn't reflect the true cost of a missed failure) undermines the whole approach.

Structured elaboration

Feature engineering: build windowed aggregates (rolling mean, rolling standard deviation, trend over the last N readings) leading up to each timestamp, and align events carefully so a feature never incorporates information from after the prediction point (a rolling window must be strictly trailing, never centered or forward-looking).

Resampling or weighting: given how rare failures typically are, a mix of moderate undersampling of the abundant "normal" periods plus class weighting on the remaining data is often more practical than heavy SMOTE, since interpolating between failure events across different equipment or time periods can produce physically implausible synthetic sensor readings.

Time-aware cross-validation: use walk-forward or rolling-origin folds so training always precedes validation in time, and additionally try to preserve a reasonable minority (failure) count per fold, which may mean widening some folds' training windows if a chronological cut would otherwise leave a fold with too few failure examples to evaluate meaningfully.

Prioritizing recall while managing false alarms: choose a metric and threshold explicitly tied to the operational cost of a missed failure (expensive, possibly safety-critical) versus a false alarm (an unnecessary maintenance check), typically meaning you accept a real reduction in precision to keep recall high, but validate that the resulting false-alarm RATE is something the maintenance team can actually absorb.

Worked example

For a fleet of 500 machines with roughly 2 failures per machine per year, a rolling 30-day feature window computed strictly from past sensor readings, walk-forward validation with monthly folds, and a threshold tuned to catch 90% of true failures might produce, say, 15 false alarms per true failure caught, a ratio the maintenance team needs to sign off on as workable before this goes to production.

Trade-offs and pitfalls

The recurring trap in rare-event time-series problems specifically is a feature that looks harmless but subtly uses future information (a "time since last similar reading" feature computed by scanning both forward and backward from the current point, or a rolling window that isn't strictly trailing), which inflates offline validation metrics in a way that silently collapses once deployed, since production never has access to the future the offline computation quietly used.

Problem formalization:

• Environment: M services i∈{1..M}, T decision rounds. Each round t we allocate a vector of discrete resources a_t = (a_{t,1},...,a_{t,M}) from a feasible budget set A (e.g., ∑i c_i a{t,i} ≤ C, per-host caps).
• Reward: stochastic QoS reward r_t = ∑i R_i(a{t,i}, s_{t,i}) where s_{t,i} are unknown service states (load, SLO attainment); we observe rewards (possibly delayed/noisy) after allocation. Objective: maximize cumulative expected reward E[∑_{t=1}^T r_t] (or minimize regret vs best fixed allocation/policy).
• Unknowns: response curves R_i(·) (monotone, possibly concave), cross-service interactions (contention).

Why this is a combinatorial bandit:

• Each action is a combinatorial object (vector allocation under constraints) drawn from an exponential-size set A. Reward decomposes (possibly approximately) over arms (services) but choice couples via budget constraints. Observations may be semi-bandit (per-service feedback) or bandit (only aggregate). This matches combinatorial multi-armed bandits (CMAB).

Algorithmic solutions:

1. Approximate Combinatorial UCB (CMUCB):

• Maintain per-service estimates μ̂_i(a) or parametric models (e.g., diminishing returns curve). Compute optimistic estimates μ̂_i + bonus_i(t) and solve an offline knapsack/ILP to pick allocation maximizing sum of optimistic rewards subject to constraints. Use approximation-aware regret bounds: if offline oracle is α-approx, regret scales O((poly)·T/α). Use structured models (GLM, parametric) to reduce sample complexity.

2. Greedy with submodular objectives:

• If ∑i R_i(a{i}) is monotone submodular in discrete resource units, use greedy hill-climbing per round: iteratively allocate marginal unit with largest estimated marginal gain. With (1-1/e) approximation and confidence bonuses, this gives provable approximate regret. Practical variant: Thompson Sampling + greedy sampling to handle uncertainty.

3. Contextual/parametric bandits:

• Use contextual CMAB: include features (current load, queues). Fit online Bayesian/linear models and use LinUCB/Thompson Sampling over allocations with efficient combinatorial optimization via knapsack solvers.

4. Handling interactions:

• If strong cross-coupling, treat joint arms for small groups or use low-rank factorization of interaction matrix and learn latent factors.

Practical monitoring and risk controls:

• Safety constraints: enforce hard SLO-preserving actions (never drop below reserve allocations), and clipping of exploratory allocations.
• Conservative bootstrapping: begin with off-line A/B tests or simulation to initialize priors; use decaying exploration rates.
• Canary & rollback: stage policy in canary cluster with traffic shadowing; automatic rollback on SLO degradation beyond thresholds.
• Observability: per-service QoS, latencies, error rates, resource usage, and reward attribution with high-cardinality logs; compute regret/expected reward delta vs baseline in real-time.
• Alarm & intervention: require human-in-loop for high-impact allocation changes, throttle exploratory updates when variance or delayed feedback increases.
• Explainability: surface marginal gains used for each allocation decision to operators.

Trade-offs:

• Exact CMAB algorithms give theoretical guarantees but need efficient oracles; greedy/submodular methods scale well and are robust if submodularity approx holds. Contextual models reduce samples but require reliable features.

This design balances provable algorithms with practical safety: start with parametric/contextual CMUCB + greedy allocation, strong monitoring and safety gates, then iterate as more data reduces uncertainty.

Direct answer

A plain stack already gives $O(1)$ push, pop, and top because those operations only ever touch the top element. The trick for $O(1)$ minimum retrieval is to keep a second, parallel stack that tracks what the minimum would be after each push: whenever you push a value onto the main stack, you also push the smaller of that value and the previous minimum onto the min-stack, so its top is always the correct current minimum, and popping both stacks together keeps them in sync without ever rescanning.

Approach

• Maintain two stacks of equal length at all times: stack holds the real values, min_stack holds, at each position, what the minimum was after that push.
• push(x): append x to stack. Append x to min_stack if min_stack is empty or x is less than or equal to its current top; otherwise append the current top again (repeating the still-current minimum).
• pop(): pop from both stacks together; the value from stack is returned, the value from min_stack is discarded.
• get_min(): return min_stack's top directly.

class MinStack:
    def __init__(self):
        self.stack: list[int] = []
        self.min_stack: list[int] = []

    def push(self, x: int) -> None:
        self.stack.append(x)
        if not self.min_stack or x <= self.min_stack[-1]:
            self.min_stack.append(x)
        else:
            self.min_stack.append(self.min_stack[-1])

    def pop(self) -> int:
        if not self.stack:
            raise IndexError("pop from empty stack")
        self.min_stack.pop()
        return self.stack.pop()

    def top(self) -> int:
        return self.stack[-1]

    def get_min(self) -> int:
        return self.min_stack[-1]


if __name__ == "__main__":
    s = MinStack()
    s.push(5)
    s.push(3)
    s.push(7)
    print(s.get_min())  # 3
    s.pop()
    print(s.get_min())  # 3
    s.pop()
    print(s.get_min())  # 5
    print(s.top())       # 5

Running this prints 3, 3, 5, 5: after pushing 5, 3, 7 the minimum is 3; popping 7 (the top) leaves the minimum still 3; popping 3 next leaves only 5, so both the minimum and the top become 5.

Key points

• Using <= (not strict <) when deciding whether to push a new minimum is what makes duplicate minimum values work correctly: if two entries tie for the minimum and you only recorded the first, popping it would incorrectly raise the recorded minimum before the still-present duplicate is gone.
• An alternative "encoded delta" trick stores a single stack, keeping only a running minimum variable, and pushes a value relative to that minimum instead of the raw value, updating the running minimum on push/pop as needed. It roughly halves auxiliary storage but is more error-prone to implement correctly, especially in fixed-width-integer languages (C++, Java) where the encoded delta itself can overflow if the gap between the pushed value and the previous minimum is large.

Complexity

Time: $O(1)$ for every operation (push, pop, top, get_min). Space: $O(n)$ auxiliary for $n$ elements (two stacks, each up to size $n$; a larger constant factor than a single stack, but still linear).

Edge cases

• pop or top on an empty stack should raise or otherwise signal an error rather than reading past the end.
• Duplicate values at the current minimum: handled correctly only if the min-stack push condition uses <=, not <.
• A single-element stack: get_min() must equal top().

Direct answer

When a model improves engagement but hurts a related revenue metric, the first step is confirming both effects are real and not artifacts of measurement, then deciding based on the actual net effect on the metric the business ultimately cares about, not on which single metric looks better in isolation.

Structured elaboration

1. Verify both effects with proper experimental rigor. Confirm the engagement gain and the click-through-rate drop are both measured from the same, valid experiment (not one from an A/B test and the other from a noisier before-after comparison), so you're comparing effects you actually trust.
2. Trace the revenue drop to its mechanism. Is the CTR decline because the new model is surfacing more organic content and pushing ads further down, or some other specific behavior change? Understanding the mechanism tells you whether it's fixable without giving up the engagement gain.
3. Compute the net business effect. Translate both the engagement gain and the revenue drop into the same unit (typically dollars, over a comparable time horizon), including a reasonable estimate of engagement's longer-term value (retention), not just its immediate effect.
4. Consider a modify option before a binary keep-or-rollback decision. If the mechanism is traceable (ads pushed down), a modification (reserve specific slots for ads regardless of the new ranking) might recover most of the revenue while keeping most of the engagement gain.
5. Communicate the recommendation with the mechanism and the net number, not just a verdict. Stakeholders need to understand WHY you're recommending keep, modify, or rollback, not just which option won.

Worked example

If the mechanism turns out to be that better organic ranking pushed sponsored placements further down the page, reserving a fixed number of above-the-fold slots for ads regardless of the ranking model's organic scores might recover most of the CTR loss while preserving the bulk of the engagement improvement, turning what looked like a binary trade-off into a modify-and-keep-both outcome.

Trade-offs and pitfalls

The common mistake is treating this as purely a math problem (whichever metric moved by a bigger relative percentage wins) without tracing WHY the trade-off is happening, which is often what reveals a modify option that a pure keep-or-rollback framing would miss. The other risk is under-valuing engagement's longer-term effect on retention because it's harder to quantify than an immediate revenue number, systematically biasing the decision toward the metric that's easier to measure.

Direct answer

Start with df.memory_usage(deep=True) to find out which columns are actually responsible for the memory, do not guess. Almost always the answer is one of three things: numeric columns stored wider than the data needs (int64/float64 when int8/float32 would hold every value), a text column that is really a small set of repeated categories, or, on current pandas, string columns still paying for the older, heavier text representation. Fix the numeric columns and the repeated-category columns first, since together they are usually the highest-impact, lowest-risk change, and they lose no information: every original value is still exactly recoverable.

Finding out where the memory is going

The two commands that answer this on any DataFrame are df.info(memory_usage="deep") (per-dtype totals and a grand total) and df.memory_usage(deep=True).sort_values(ascending=False) (the per-column breakdown, in bytes). deep=True matters on both: without it, pandas reports the size of the pointers/array structure, not the actual bytes of variable-length data like strings, which understates text-heavy columns dramatically. Concretely, on the DataFrame built below:

import pandas as pd
import numpy as np

n = 5_000
rng = np.random.RandomState(0)
df = pd.DataFrame({
    "user_id": np.arange(n, dtype="int64"),
    "score":   rng.randint(0, 100, n).astype("int64"),
    "country": rng.choice(["US", "UK", "CA", "DE", "FR"], n),
    "price":   rng.uniform(0, 1000, n).astype("float64"),
})
print("before:", df.memory_usage(deep=True).sum(), "bytes")

def downcast_numeric(frame):
    for col in frame.select_dtypes(include=["integer", "float"]).columns:
        if pd.api.types.is_integer_dtype(frame[col]):
            frame[col] = pd.to_numeric(frame[col], downcast="integer")
        else:
            frame[col] = pd.to_numeric(frame[col], downcast="float")
    return frame

def convert_to_category(frame, max_unique_ratio=0.5, max_unique=1000):
    # include both 'object' and 'string' so this catches text columns under
    # pandas 3.0's default string dtype as well as legacy object columns
    for col in frame.select_dtypes(include=["object", "string"]).columns:
        n_unique = frame[col].nunique(dropna=False)
        if (n_unique / len(frame) <= max_unique_ratio) or (n_unique <= max_unique):
            frame[col] = frame[col].astype("category")
    return frame

df2 = downcast_numeric(df.copy())
df2 = convert_to_category(df2)
print("after:", df2.memory_usage(deep=True).sum(), "bytes")
print(df2.dtypes)

Actual output:

before: 170132 bytes
after: 40183 bytes
user_id       int16
score          int8
country    category
price       float32
dtype: object

A roughly 76% reduction, with zero information loss: user_id values all fit in int16, score (0-99) fits in int8, country has only 5 distinct values across 5,000 rows so category stores each value once and a small integer code per row, and price loses only precision beyond what float32 carries, not the values themselves in any meaningful sense for typical analysis.

Currency note verified in sandbox: pandas 3.0 changed the default in-memory representation for plain text columns from the legacy object dtype to a dedicated, more compact string dtype. df.select_dtypes(include=["object"]) alone still catches the new string dtype today for backward compatibility, but pandas emits a Pandas4Warning saying that behavior will be removed in a future version, so include=["object", "string"] (as used above) is the version-robust way to write this, it works cleanly on both current and future pandas without a warning. This also means memory-reduction advice written before pandas 3.0 will understate how much a pure string column already costs today: the new default string dtype for a ~9-character id-like column measured at roughly 18 bytes/row deep, versus roughly 59 bytes/row for the same data forced into legacy object, a real ~3.3x difference before you even apply category.

Highest-impact fix to try first

For most real datasets, numeric downcasting is the highest-impact, lowest-effort fix: it is fully automatic (pd.to_numeric(..., downcast=...) inspects the actual min/max in the column and picks the smallest safe type), reversible, and touches every numeric column in one pass. Converting low-cardinality text to category is usually the second-highest-impact change and is worth doing right after, in the same pass.

Other techniques worth knowing, roughly in order of how often they apply

• Datetime parsing: a date stored as text (object/string) costs far more than the same value as datetime64[ns]; pd.to_datetime(col, errors="coerce") converts it and also enables fast datetime-specific operations afterward.
• Sparse dtypes: for a column that is mostly zeros or mostly one repeated value (common after one-hot encoding), pd.Series(data, dtype="Sparse[int]") stores only the non-default values plus their positions.
• Selective loading: if you know only some columns are needed downstream, usecols= at read_csv time avoids ever materializing the rest, cheaper than loading everything and dropping columns afterward.
• Drop intermediate columns as soon as they are no longer needed, rather than carrying every derived column for the life of the DataFrame.

Trade-offs and pitfalls

• Downcasting integers or floats can lose precision if you are not careful: downcast="integer" only picks a smaller type when every value in the column actually fits, so it is safe by construction, but downcast="float" to float32 does trade off precision, verify that the columns you downcast do not feed into calculations sensitive to that last few bits of precision (e.g., some financial aggregations).
• category is a net win when the number of distinct values is small relative to the row count and the column is used repeatedly in groupby, filtering, or joins (comparisons on the integer codes underneath are faster than comparing strings). It is a net loss on high-cardinality free text (ids, GUIDs, long descriptions) where the category-code table ends up almost as large as just storing the values, plus the overhead of the mapping itself.
• Sparse dtypes help only when the "default" value genuinely dominates; a column that is 40% zeros is not a good sparse candidate, one that is 99.9% zeros is.

Complexity and edge cases

Complexity: downcasting and categorical conversion are each a single O(n) pass per column examined; nunique() for the category heuristic is itself an O(n) pass per candidate column, so profiling many wide, high-cardinality columns before converting has a real up-front cost worth being aware of on very wide DataFrames.
Edge cases: an all-NaN numeric column cannot be safely downcast to most integer types (no integer type represents NaN) and will either stay float64 or need a nullable integer type (Int32) instead. A column with exactly one unique value converts to category trivially and near-optimally. An empty DataFrame (len(df) == 0) makes the n_unique / len(df) ratio a division by zero, guard for that explicitly before applying the category heuristic.