Lyft Machine Learning Engineer Interview Preparation Guide - Senior Level

Direct answer

When the sink itself cannot transact, move the "did this already happen" question out of the sink and into a small, dedicated record of what has been applied: assign each unit of work a stable, deterministic key, write to the sink, then record that the key has been applied, and have every retry check that record before writing again. The sink no longer needs to be transactional, because the safety property lives in the check-then-write pattern around it, not inside it.

Structured elaboration

The core pattern

Give every unit of work a deterministic key derived from its business identity, never a randomly generated per-attempt identifier, since two different attempts at the same logical work must produce the SAME key. Before writing to the sink, check an external record store for that key; if it is already marked applied, skip the write. After a successful write, mark the key as applied. Order matters here: the sink write happens first and the mark happens second, so if the process crashes in between, the worst case is one retried write plus a state check, not a lost write. Because the sink cannot transact, the record store around it only needs to guarantee a consistent "mark as applied" operation, which is a much smaller problem than making an arbitrary sink transactional.

Handling the gap between "sink write succeeded" and "record marked"

This is the one genuine edge case: if the process dies in that gap, a naive retry sees "not yet marked" and writes again, duplicating. Two workable responses, in order of preference. First, if the sink can absorb an idempotent write for that one operation on its own (an upsert keyed the same way), let it, and treat the external record store as an optimization that skips most redundant attempts rather than the last line of defense. Second, if the sink genuinely cannot deduplicate at all (pure append, no key), narrow the risk instead of eliminating it: keep the record-store update tightly coupled to the sink write, accept a bounded, rare duplicate-write risk in that specific crash window, and catch it later through reconciliation rather than trying to force an un-transactable sink into transactional behavior.

Folding in lineage for audit

Attach the same deterministic key, plus a timestamp and attempt count, to the record as it is written to the sink, not only to the external dedupe store. That turns the idempotency key into an audit trail as well: if a downstream consumer later asks whether a record was written once or might be a duplicate, the lineage metadata attached to the record answers it directly, instead of requiring someone to reconstruct the answer from a separate dedupe log that may have already expired.

flowchart LR
    Producer[Upstream producer] --> Proc[Processor]
    Proc --> Check{Dedupe key seen already?}
    Check -- No --> Write[Write to sink]
    Check -- Yes --> Skip[Skip: already applied]
    Write --> Mark[Record key as applied]
    Mark --> Sink[(Non-transactional sink)]

Worked example

A billing-events stage writes usage records to an append-only sink with no unique constraint. Redesign: assign each record a key built from the account identifier, billing period, and usage type. Before writing, check a small key-value record store for that key; if marked applied, skip. Write the record to the sink tagged with that same key as a lineage field. Mark the key applied in the record store. On retry after a mid-write crash, the check finds no record yet, so at most one duplicate line can appear in the sink for that key; a lightweight downstream step that keeps only the last-marked-applied row per key removes it before the data reaches reporting. This narrows the failure mode from unbounded duplication to one bounded, catchable case.

Trade-offs & pitfalls

• Putting all the safety into careful retries (backoff, fewer attempts) instead of into the write pattern reduces the odds of duplication without eliminating the mechanism that causes it.
• Keying on something that is not stable across retries, such as a freshly generated identifier per attempt, guarantees every retry looks like new work.
• The external record store adds a dependency and a bit of latency to every write; that is the cost of buying retry-safety for a sink that cannot provide it itself.
• Fully eliminating the crash-window duplicate is strictly better when the sink supports an idempotent absorb, but not every sink does; when it does not, the honest answer is a bounded, reconciled risk, not a claim of perfect exactly-once.
• Treating this as solved once retries stop producing visible duplicates in testing is a common wrong turn; the crash-window race is rare by construction and will not show up until production scale.

Direct answer

An average can barely move even when a small fraction of requests get much slower, because the mean is dominated by the large bulk of ordinary requests and only weakly pulled by a thin tail; the 99th percentile (p99), by definition, reports exactly that tail. Tracking down the cause means segmenting the slow requests specifically (by host, region, request type, time window) rather than reading an aggregate dashboard, then correlating with resource-level signals like garbage-collection pauses, contention, connection-pool exhaustion, or a downstream dependency's own tail.

Structured elaboration

Why average latency structurally misses this. If only a small percentage of requests are unusually slow, their contribution to the arithmetic mean is proportional to that small percentage, while the p99 statistic is defined precisely at that percentile and reports the slow tail directly. A metric built to summarize the "typical" request is, by construction, not built to surface a rare-but-severe one.

Diagnostic order:

1. Confirm the spike is real, not a monitoring artifact: check the tracing system's sampling rate and histogram bucket boundaries, since coarse buckets or low sampling can distort a percentile estimate even when nothing in production actually changed.
2. Segment the p99 by dimension: host, region, request type, time-of-day. A spike concentrated on one host or one downstream dependency points to a localized cause (one bad instance, one flaky dependency); a spike spread evenly across the fleet points to a systemic cause (a shared resource, or a change common to all instances).
3. Trace the SLOWEST requests specifically, not an average-sampled set of traces, since random 1% sampling can easily miss the exact 1% of requests that matter here.
4. Correlate with resource telemetry: garbage-collection pause logs (for managed runtimes), CPU steal time (a sign of a noisy neighbor on shared hardware), thread-pool or connection-pool queue depth, and the tail latency of any downstream service you call, since a downstream tail can propagate into your own tail on every request that happens to hit it.

Common root causes that produce exactly this signature (flat average, spiking p99): stop-the-world garbage-collection pauses that hit only some requests; lock contention that only manifests under a specific interleaving of concurrent requests; connection-pool exhaustion causing occasional queuing rather than constant slowness; cold-cache or cold-start stalls on a subset of replicas; and downstream timeouts or retries that only trigger intermittently.

Worked example

An illustrative distribution (not a measured benchmark, chosen to show the mechanism) makes the dilution concrete. Suppose out of 1,000 requests, 990 (99%) take 20 ms and 10 (1%) take 800 ms, perhaps due to an occasional garbage-collection pause:

Against a prior baseline average of 20 ms (when nothing was slow), that's only a 39% relative move in the average, easily inside a typical alert threshold. But the p99 of this same distribution sits right at the slow tail, around 800 ms, a roughly 40x jump from baseline. The average moved by less than half, while the metric that describes what one in a hundred users actually experienced moved by 40x. That gap is exactly why alerting must be defined on p95/p99 (95th/99th percentile), not the mean.

Trade-offs & pitfalls

Alerting purely on average latency will structurally miss this entire class of regression; service-level objectives (SLOs) need to be defined on the tail directly. A random trace-sampling strategy (say, 1% of all requests) can easily fail to capture the specific 1% that are slow; tail-biased sampling (always trace anything above a latency threshold) catches what uniform sampling misses. The common wrong turn is dismissing a p99 spike as "noise" because the average looks fine, when the p99 is precisely the number describing the unlucky tail of real users. Fixing the tail can also introduce new trade-offs worth stating explicitly: hedged requests (firing a duplicate request if the first is slow) reduce tail latency but increase total load, which can itself create a new bottleneck if applied too broadly.

Direct answer
Replace a vague quantifier with a specific number, a specific comparison point, or an explicit definition of what counts, so the reader can check the claim rather than just trust your impression of it.

Structured elaboration

• "Significant increase" is unfalsifiable on its own: significant compared to what, and by how much? Fix it by naming the actual number and the baseline it's compared against.
• "Large drop" has the same problem in the other direction; a reader can't tell if that means a 5% dip or a 50% collapse.
• The general pattern: replace a subjective adjective ("significant," "large," "modest") with either a number and a baseline, or, if the exact number genuinely isn't available, an explicit statement of the range and why it's uncertain, which is still more falsifiable than a bare adjective.
• A quick self-check: could someone else look at the underlying data and disagree with whether your adjective was the right one? If yes, the phrase is doing too much subjective work and needs a number behind it.

Worked example
Vague: "Revenue saw a significant increase this quarter."
Specific: "Revenue grew 18% quarter-over-quarter, from $4.2M to $5.0M."

Vague: "There was a large drop in signups after the pricing change."
Specific: "Signups fell 34% in the two weeks after the pricing change, from roughly 1,400/week to about 920/week."

Vague: "Customer satisfaction scores showed a modest improvement."
Specific: "Our NPS (Net Promoter Score, a customer-loyalty survey metric typically scored from -100 to 100, based on how likely customers are to recommend you) moved from 32 to 38, a 6-point increase, over the last two survey cycles."

Each rewrite keeps the same claim but replaces the reader's guesswork with a number and a comparison point they can independently evaluate.

Trade-offs and pitfalls

• If you genuinely don't have the precise number, don't invent a specific-sounding one to appear rigorous; say "we don't have an exact figure yet, but early signals suggest an increase" rather than fabricating false precision.
• Numbers without a baseline can still mislead ("revenue grew 18%" sounds good until you learn it grew from a very small base); include enough context that the number is honestly interpretable, not just numeric.
• Overloading every sentence with numbers can make a document harder to read, not easier; reserve the rigor for the claims that are actually load-bearing for a decision.

Direct answer

I weigh a short list of signals against each other: what the team or product genuinely needs next, where I'm personally the bottleneck, how durable the skill is versus how much of its appeal is short-lived hype, how long it'll take to become useful, and how it fits where I want to grow longer-term, then I deliberately resist just picking whatever happens to be most interesting that week.

Structured elaboration

The signals, roughly in the order I actually weigh them: what's genuinely needed next (not hypothetically useful, but blocking something soon); where I am the bottleneck versus where someone else already covers it; durability, since a skill built on something likely to be replaced in a year pays off less than one that generalizes; time to first usefulness, since a skill that takes six months to pay off is a different bet than one that pays off in a week; and longer-term direction, since some choices compound toward where I want to be in a few years and some don't.

If I use anything like a scoring approach across those signals, I keep it as a judgment aid, not a formal weighted-matrix exercise. Reducing this to a spreadsheet score tends to manufacture false confidence in what's actually a judgment call.

There are times the right answer is to learn nothing new and go deeper on current work instead, particularly when the team's actual bottleneck is depth in something I already do, and picking up something new would just be more comfortable than admitting that.

Worked example

Recently I had to choose between going deeper on Airflow, the batch-orchestration tool I already ran our nightly pipelines on, or picking up event-driven stream processing, an adjacent area I'd never worked in that a few upcoming projects seemed likely to lean on. I weighed it using the signals above: streaming wasn't blocking anything yet, so it scored low on "genuinely needed next," but it scored high on durability and on long-term direction, since it was a skill I expected to matter regardless of which specific project used it. I chose to learn streaming. In hindsight, my durability read was mostly right, but I underestimated how long it would take to become useful: I expected a project to need it within a couple of months, but it was closer to eight months before a fraud-detection feature actually required near-real-time signals instead of our usual nightly batch, so it paid off later than I expected, which is worth reporting honestly rather than pretending the choice was cleanly validated on schedule.

Trade-offs and pitfalls

The common failure mode is turning this into a rigid scoring exercise that produces a false sense of objectivity about what's ultimately a judgment call. The opposite failure is always chasing whatever's currently getting the most attention under the label of "future-proofing," without actually checking it against need or durability.

Direct answer

When assignment happens per-device (a cookie on web, a device id on the app) but the real unit of interest is the person, users who touch the product on multiple devices get assigned independently on each device, so some of them land in both control and treatment at once. That breaks the assumption that each experimental unit receives exactly one arm: it dilutes the measured treatment effect, since a "contaminated" user's behavior is influenced by both arms, and it can double count outcomes if the same person's actions are logged and analyzed once per device-identity rather than once per person. The fix is to randomize and log at the most persistent identity you actually have, then dedupe and correct for the identities you had to resolve after the fact rather than at assignment time.

Structured elaboration

Where the bias enters

• Assignment stage: a user with a web cookie and a mobile device id gets two independent coin flips. If both land the same way, no harm; if they split, that person is exposed to treatment and control simultaneously, a violation of SUTVA (the assumption that one unit's outcome does not depend on another instance of its own assignment).
• Analysis stage: if the analysis unit is "device" rather than "resolved person," a contaminated person's activity appears once in each arm's numerator, and the two rows are not independent observations even though the analysis code treats them as such, which understates the true variance.
• Coverage bias: identity resolution itself makes mistakes (false merges linking two different people, false splits treating one person as two); if those errors are not random with respect to treatment, they introduce their own bias on top of the contamination.

Practical strategies

1. Randomize at the most stable identity you have. Prefer a logged-in account id over a device id or cookie whenever the user is authenticated; fall back to a deterministic device hash only for logged-out traffic, and treat that population as a separate, lower-confidence stratum in reporting.
2. Log everything needed to resolve identity after the fact. Persist device id, cookie id, and account id (hashed, respecting privacy) on every assignment and every outcome event, even when the assignment itself was made at device level, so contamination can be measured and corrected during analysis rather than discovered too late.
3. Define the primary analysis on the resolved (canonical) identity, not the raw assignment record: after backend identity merge, collapse a contaminated user into a single row and apply an explicit, pre-registered rule for what arm they count as, for example "any-device-treatment counts as treated," reported alongside a stricter "single-device-only" rule as a sensitivity check.
4. Quantify and bound the bias rather than ignore it. Report the primary result plus at least two sensitivity analyses: one restricted to users seen on exactly one device (removes contamination but shrinks the sample), and one using the any-device-treatment rule (keeps the full sample but is a diluted estimate of the true per-exposure effect).
5. Use cluster-robust standard errors at the resolved-identity level so that outcomes from the same person are not treated as independent observations even after collapsing to one row per person, since a person can still contribute multiple sessions or events.

Worked example

Assume 20% of users are active on exactly two devices (web and app) and 80% are single-device (an illustrative, stated split). Assignment happens independently per device with probability 0.5 to treatment. For a two-device user, the four equally likely device-pair outcomes are (control, control), (control, treatment), (treatment, control), (treatment, treatment), each with probability 0.25:
$P(\text{both control}) = P(\text{both treatment}) = 0.25, \qquad P(\text{split, i.e. contaminated}) = 0.5$
So among the 20% of users who are two-device, half get split across arms, which is 10% of the total user base. If a contaminated user's outcome is counted in both the treatment and control totals rather than resolved to one arm, then 10% of the treatment-arm numerator and 10% of the control-arm numerator are contributed by the exact same set of people, which both understates the between-arm difference and violates the independence assumption behind the standard error calculation. Restricting the primary analysis to the 90% of users who are single-device or resolve cleanly to one arm removes the contamination at the cost of 10% of the sample, which should be reflected directly in the power calculation for the corrected analysis.

Trade-offs and pitfalls

• The "any-device-treatment" rule is conservative and interpretable but structurally dilutes the estimated effect toward zero for contaminated users, since they experienced a mix of both arms; don't present it as an unbiased estimate of the pure per-exposure effect.
• Restricting to single-device users is cleaner statistically but changes who the estimate applies to: if multi-device users differ systematically (often more engaged, higher-value), the single-device estimate may not generalize to the full user base.
• Identity resolution is itself a model with false-merge and false-split error rates; a resolution pipeline retrained or changed mid-experiment can shift the contamination rate over time and should be monitored, not assumed constant.
• Logging every identifier needed for later resolution has real privacy and storage cost; agree on hashing and retention policy with the privacy function before instrumenting, not after a contamination investigation is already underway.

Direct answer

A parameter-server architecture for deep-network training centers on a set of server processes holding authoritative parameter shards, a group of worker processes computing gradients, and an explicit protocol for how updates are ordered and made consistent across servers. The two central design decisions are (1) whether updates are synchronous or asynchronous, and (2) how a server (or the server group) survives failure.

Structured elaboration

• Leader election / server topology: for a sharded parameter server, each shard typically has a primary that accepts writes and one or more replicas kept consistent via a simple replication protocol (e.g. leader forwards updates to followers and only acknowledges the worker after a quorum of replicas apply the update). Leader election (choosing a new primary if one fails) is usually handled by an external coordination service (etcd/ZooKeeper-style) rather than hand-rolled, since getting leader election right under network partitions is famously hard.
• Consistency of updates: synchronous mode has the server wait for all (or a fixed quorum of) workers' gradients for a step before applying them and advancing the parameter version, giving every worker a consistent view but limited by the slowest worker (a straggler). Asynchronous mode applies each worker's update as it arrives, which keeps workers from ever blocking on each other but means different workers can pull different, mutually inconsistent parameter snapshots within the same "round," introducing staleness bias.
• Sharding: parameters are partitioned (e.g. by hashing parameter/embedding keys) across many server processes so no single node's network or memory becomes the bottleneck; a worker's gradient for a given mini-batch typically needs to be split and routed to the shards owning the touched parameters.
• Failure handling: a failed server shard needs either replication-based failover (a replica is promoted) or checkpoint-based recovery (restore the shard's last checkpoint and replay recent updates from a log); a failed worker simply stops contributing gradients until it restarts and re-pulls current parameters.

Worked example

For a model with 200M parameters split across 8 server shards (25M parameters each), a worker computing a mini-batch gradient partitions its gradient tensor by the same hash function used to shard the parameters, sends each of the 8 slices to its owning shard, and each shard applies its slice of the update independently and in parallel: total server-side work per step is unchanged, but the eight shards can process it concurrently rather than one server serializing all 200M parameter updates.

CAP theorem and network partitions

A sharded, replicated parameter server is a distributed data store under the hood, so CAP theorem trade-offs apply directly: under a network partition (a shard's primary can't reach some of its replicas, or a subset of workers can't reach a given shard), you cannot have both full Consistency and full Availability for that shard, only one. Choosing quorum-based synchronous replication (a write is acknowledged only after a majority of replicas apply it) favors Consistency: parameters stay correct but writes/reads on the minority side of a partition block or fail, which mirrors synchronous training's willingness to have a straggler block progress in exchange for a single consistent version of the parameters. Choosing to keep serving reads/writes on both sides of a partition favors Availability: workers on the minority side keep training against a possibly stale or diverging parameter snapshot, which is the same trade asynchronous PS already makes for staleness reasons, just now imposed by a network fault rather than by design. In practice most PS deployments pick CP (consistency-favoring) for the coordination layer that elects primaries (etcd/ZooKeeper are themselves CP systems) while allowing the data plane (asynchronous gradient push/pull) to behave AP-like during a partition, accepting bounded staleness rather than blocking training entirely; this hybrid is why the earlier synchronous-vs-asynchronous choice and the CAP choice are really the same underlying decision applied at two different layers of the system.

Trade-offs & pitfalls

Synchronous parameter-server training gives you the same statistical guarantees as synchronous AllReduce (each step uses a consistent gradient) but adds server infrastructure complexity that AllReduce avoids for dense models; the parameter-server pattern earns its keep specifically when access is sparse (each worker only needs a subset of shards) or extremely large (embeddings that don't fit replicated on every worker).

Direct answer

Keep a stalled dependency peer-to-peer as long as direct conversation is still making progress. Escalate when you hit a concrete trigger: a scope change that neither side can unilaterally absorb, genuinely conflicting priorities that only someone with visibility into both roadmaps can arbitrate, or a hard deadline-driven blocker where peer-to-peer conversation has already stalled.

Framework

Default: work it peer-to-peer. Most stalls are under-communication or unclear ownership, and a direct conversation or a short written proposal usually unsticks them without anyone else getting involved.

Concrete triggers to escalate.

• Scope change: the fix now requires work neither team budgeted for, and only a manager can reprioritize that.
• Conflicting priorities: both sides are acting rationally from their own team's goals, and the trade-off needs someone with visibility into both roadmaps to arbitrate.
• Hard blocker with a deadline: a fixed external date is genuinely at risk, and peer-to-peer conversation has already stalled past a reasonable window, for example no movement after two direct attempts over several days.
• Repeated pattern: the same kind of stall keeps recurring with the same team, which means the real issue is the working relationship or process, not this one dependency.

What to bring when you escalate. A short brief: what's blocked, what you've already tried peer-to-peer, the realistic options and their trade-offs, and the specific decision you need.

Worked example (applying the criteria)

Situation: your team's deliverable needs a schema change from another team that they've deprioritized for two weeks despite two direct requests.
Applying the criteria: this isn't just a communication gap, direct conversation was already tried twice with no movement. It's a conflicting-priorities case, the other team's roadmap has no room for this without reprioritizing something else, combined with a hard blocker, a fixed external deadline in three weeks that this schema change sits on the critical path for (meaning if this dependency slips, the final deadline slips by the same amount, unlike a dependency with buffer to absorb delay).
Action: escalated to the shared manager with a one-page brief covering what's blocked, the two peer-to-peer attempts and their outcome, and two options: the other team reprioritizes one sprint of work, or your team ships a temporary workaround with known limitations, along with the deadline risk if neither happens within the week.
Result: the shared manager reprioritized one sprint item, unblocking the schema change with two weeks to spare before the deadline. Both teams also agreed to flag scope-affecting asks earlier next time, so the same dependency doesn't reach this point again.

Trade-offs and pitfalls

• Escalating too early over normal friction burns trust and reads as an inability to work horizontally.
• Escalating too late, repeatedly trying peer-to-peer past the point it's actually working, puts the deadline at real risk and looks like poor judgment in hindsight.
• A vague escalation with no options and no specific ask wastes the leader's time compared with a brief that names the decision needed.

Requirements & constraints (clarify): 100M users, 5B daily inferences (~57k RPS (requests per second) globally), p95 <100ms per region, EU data residency (GDPR), continuous model updates with zero downtime and safe rollback.

High-level architecture:

• Global Traffic Manager (GTM) / DNS-based geo-routing → regional API gateways
• Per-region inference clusters (K8s/autoscaling groups) colocated with regional feature caches and online feature store replicas
• Central training/CI pipeline (global) that writes model artifacts and metadata to an artifact registry; model shards pushed to regions
• Feature pipelines: streaming feature ingestion (Kafka/Cloud PubSub) → regional feature materialization (Feast-like) + offline feature store for training
• Observability & control plane: metrics, tracing, model registry, rollout manager, audit logs

Dataflow:

1. Event/feature ingestion globally; user EU-origin events stored and processed in EU region only (via ingestion routing and partitioning).
2. Online request: client → GTM → nearest regional gateway → auth & feature fetch (local cache/Redis or regional feature store) → model inference (TF-Serving/onnxruntime/gRPC) → response.
3. Offline: raw events replicated to central training only if permitted; EU raw data stays in EU storage; aggregated/anonymous cross-region metrics permitted under compliance rules.

Replication & storage choices:

• Model artifacts: pushed to regional read-optimized stores (S3/gcs regions or object storage) via encrypted replication; registry records versions and allowed regions.
• Feature stores: primary-per-region for low-latency; cross-region async replication for non-PII aggregated features. Use strong locality for user-personalization features.
• Metadata, audit logs centrally aggregated but PII-filtered; for GDPR, store user-identifiable logs in-region and expose deletion APIs.

Continuous updates, zero downtime, safe rollback:

• CI/CD: automated training → validation (A/B offline metrics + shadow testing) → canary rollout.
• Deployment strategy: blue-green or canary with traffic shaping at the gateway. Start with shadow inference in region, then 1% traffic canary, monitor SLA (service-level agreement)/metrics for 30-60 mins, ramp to 100%.
• Model versioning: immutable artifacts with semantic versions and expected data schema; automatic health checks and automatic rollback if p95 latency or error rate regression thresholds exceeded.
• Feature compat checks before deploy.

Operational controls & SLOs:

• Per-region SLOs: p95 latency <100ms, error rate <0.1%, throughput headroom 2x.
• Monitoring: real-time metrics (latency, error rate, skew between online/offline predictions), distributed tracing, data drift detection, input distribution alerts.
• Safety: kill-switch at gateway for emergency rollback; circuit breakers; rate limiting.
• Compliance: consent management service, in-region data deletion APIs, encryption at rest/in-transit (KMS), regular audits, DPIA documented.
• DR & testing: periodic failover drills, chaos tests, load tests at 2x expected peak, replay shadow traffic for post-deploy validation.

Trade-offs:

• Strong regional locality increases operational cost but required for p95 and GDPR. Async replication preserves privacy but adds complexity for global aggregated analytics.
• Canary/blue-green increases storage and orchestration complexity but provides safe zero-downtime updates.

This design balances low-latency regional serving, strict GDPR residency, continuous deployment safety, and observability for safe operation at 5B daily inferences.

Also covers (folded from merged near-duplicates): ba398a72 folds the fraud-detection-specific global-failover requirements; 31d3ed48 folds the hybrid centralized-batch-plus-regional-online serving combination; b2a35dba folds the multimodal, geo-DNS/edge routing angle; 7b228da2 folds the cross-region MODEL-VERSION-CONSISTENCY requirement (avoiding stale/mismatched models across regions during rollout), re-homed from the ineligible 'Software Engineer' bucket.

Direct answer: Estimating a feature store's monthly cost from concrete parameters (users, feature count, vector size, query volume, batch cadence) requires breaking the estimate into its distinct cost DRIVERS, since online storage, offline storage, and compute each scale with a different dimension of the workload, and conflating them produces a misleading total.

Structured elaboration:
Breaking down the estimate: online storage scales with (users × features materialized online × average vector size), giving a total resident data volume, priced per the chosen key-value store's storage rate; offline storage scales similarly but is typically far cheaper per byte (warehouse/object storage rates are usually a fraction of a fast key-value store's), though it may retain more historical depth (many snapshots over time) rather than just the current value; batch compute scales with how much data the daily recompute job actually touches, driven by feature count and update frequency, not directly by query volume; streaming compute (if applicable) scales with event throughput, largely independent of the batch job's cost; network egress scales with query volume and average response size, mattering especially for cross-region serving; caching reduces the effective load on the persistent online store, at an added cost for the cache layer itself, netting out to a cost REDUCTION if the cache hit rate is high enough, since it avoids the more expensive underlying store's cost for the majority of reads.

Worked example: For the concrete figures given (100M users, 2,000 online-materialized features, 1KB average vector size, 200k queries per second, plus daily batch recompute): the online store's resident data volume is on the order of 100M × 2,000 × 1KB ≈ 200 terabytes if every feature were held per user (in practice a smaller "hot" working set dominates real query volume, which is exactly where caching earns its cost back); at 200k queries per second sustained, the online store's provisioned throughput capacity, not just its storage volume, becomes the other major cost driver, and these two dimensions (storage volume and query throughput) need to be priced and reasoned about separately, since a store sized generously for storage but under-provisioned for throughput (or vice versa) would either overspend or fail its latency target.

Trade-offs and pitfalls: A cost estimate presented as one single number without stating its key assumptions (cache hit rate, actual hot-working-set size versus total addressable feature volume, provisioned-versus-actual throughput) invites false confidence; presenting a SENSITIVITY range (what the cost looks like at a pessimistic versus optimistic cache hit rate, for instance) is far more useful to a decision-maker than a falsely-precise single figure.

Concrete counterexample:

• Denominations: {1, 3, 4}
• Amount: 6

Greedy (choose largest coin first):

• Pick 4 → remaining 2 → pick 1, 1 → total coins = 3 (4 + 1 + 1)

Optimal solution:

• Pick 3 + 3 → total coins = 2

Why greedy fails here:

• Greedy relies on the greedy-choice property: a locally optimal (largest-possible) choice leads to a global optimum. That property does not hold for this coin set. Choosing 4 looks best locally but forces using two 1s later; a different first choice (3) enables a better global combination (two 3s).
• Put another way, the greedy algorithm ignores how the first choice affects the space of remaining combinations. The problem lacks the necessary canonical coin-system property (like U.S. coins for many amounts) that guarantees greedy optimality.

Why dynamic programming fixes it:

• DP uses optimal substructure correctly: the optimal number of coins for amount A equals 1 + min_{c in coins, c ≤ A} optimal_coins(A - c). DP computes optimal_coins for all subamounts (bottom-up or memoized top-down), so it explores combinations rather than enforcing a single local choice.
• For amount 6 with {1,3,4}, DP evaluates:
  optimal_coins(6) = 1 + min(optimal_coins(5), optimal_coins(3), optimal_coins(2))
  which leads to choosing two 3s (optimal_coins(3) = 1), giving optimal_coins(6) = 2.

Takeaway:

• Greedy can be incorrect unless the coin system satisfies special properties. DP is safe because it examines all subproblems and enforces global optimality via recurrence.