Spotify Solutions Architect (Mid-Level) Interview Preparation Guide

Direct answer

Facing write contention at 20,000 requests per second, I would sequence the fix by how much architecture change it costs: read replicas first (days to weeks, no schema change, relieves the primary of read load so it can dedicate more capacity to writes), then partitioning of the hottest write tables (weeks to months, splits one table into many within the same database), then full sharding (months, splits the database itself across independent nodes) only once partitioning alone can no longer keep write throughput ahead of demand. Each step is validated against the projected load, not just the current one, since the roadmap has to hold as the write rate keeps climbing.

Structured elaboration

flowchart LR
    A["Today: 20,000 req/s,\nwrite contention on one primary"] --> B["Short-term:\nread replicas"]
    B --> C["Medium-term:\ntable partitioning"]
    C --> D["Long-term:\nsharding"]
    D --> E["Sustains projected\n100,000 writes/s,\n4,000,000 reads/s"]

Short-term: read replicas. Write contention is often partly caused by reads competing with writes for the same primary's CPU, memory, and I/O, even though replicas cannot scale write throughput directly. Deploying asynchronous read replicas and routing read-heavy, non-latency-critical queries away from the primary frees up headroom the primary can spend on writes. This is purely a read-scaling move (a read replica applies an asynchronously-copied stream of the primary's writes and serves reads from that copy, but never accepts writes itself, so adding more replicas relieves the primary of read load without ever adding write capacity); it buys time, it does not solve write contention at its root.

Medium-term: partitioning. Within the same database, split the hottest write tables so each write touches a smaller table with less lock contention and smaller indexes to maintain:

• Range partitioning (by time, for example) suits append-heavy, time-ordered writes and lets old partitions be archived or dropped cheaply.
• Hash or list partitioning suits writes that need to spread evenly rather than cluster by time.
• Implementation: analyze which tables and rows are actually hot, add partitioning without blocking writes (most modern databases support online partitioning operations), backfill in batches, and update queries to include the partition key so the database can prune irrelevant partitions instead of scanning all of them.
• This buys another step of headroom without yet paying the operational cost of a fully distributed database, but it does not remove the ceiling of a single physical database instance.

Long-term: sharding. Once partitioning within one database can no longer keep pace, split the data across independent database instances (shards), each an independent primary handling its own slice of writes in parallel. This is the only step of the three that removes the single-primary write ceiling entirely, and it is the most expensive: it needs a shard-key choice that spreads writes evenly (a hash of a stable, high-cardinality identifier, for example hash(user_id), is the usual default: hashing maps each write to a pseudo-random shard regardless of which user or time period is currently most active, so no single shard absorbs a disproportionate share of the write stream the way it would if writes were assigned by a correlated attribute like signup date), a routing layer, and a migration plan, plus a strategy for the rare cross-shard transaction. Shard-key selection at this stage is also where hotspot risk gets decided: a poorly chosen key concentrates writes on one shard no matter how many shards exist, so the same hotspot-rebalancing discipline used to fix a hot shard after the fact (detect via per-shard load metrics, then move the offending key ranges onto their own shard or split them further) should inform the key choice up front rather than being treated as a problem to solve only once it appears.

Rebalancing, migration, and consistency, applied to this roadmap

• Each transition (single database to partitioned, partitioned to sharded) uses the same safe pattern: backfill the new structure while the old one keeps serving, dual-write once backfill catches up, verify with checksums or row counts, then cut over in stages (a percentage of traffic or a cohort of users at a time) rather than all at once.
• Consistency: keep transactions single-partition or single-shard wherever possible; where a write must touch two shards, prefer an idempotent, retryable pattern (such as a saga: a sequence of local transactions with compensating steps) over a synchronous distributed transaction, since the latter adds coordination latency exactly where you are trying to remove contention.
• Monitoring during migration: replication/backfill lag, row-count or checksum parity between old and new structures, per-partition or per-shard write latency and error rate, and a rollback trigger if divergence exceeds a set tolerance.

Sizing the roadmap against projected load

Capacity planning for this service projects growth to 100,000 writes per second and 4,000,000 reads per second. For concreteness, assume (explicitly, as a planning input rather than a measured fact) that a single post-partitioning shard primary can sustain roughly 10,000 writes per second, and a single read replica can sustain roughly 40,000 reads per second. The number of write shards needed at the projected load:

The number of read replicas needed at the projected load:

Ten shards is a normal, operable sharded topology. One hundred read replicas is not: at that replica count, the coordination and infrastructure cost of pure replication stops being the right lever, and the roadmap needs a caching layer in front of the database, for example an in-memory read cache keyed by the same primary lookup the application already uses, to absorb the bulk of that 4,000,000 reads/second before it ever reaches a replica, rather than trying to solve all of it with replica count alone. As an illustrative planning input, not a measured fact: if that cache achieves roughly a 90% hit ratio on read traffic, it absorbs $0.90 \times 4{,}000{,}000 = 3{,}600{,}000$ reads/second, leaving $4{,}000{,}000 - 3{,}600{,}000 = 400{,}000$ reads/second that still need to reach a replica. At the same 40,000-reads-per-second-per-replica planning assumption used above, that works out to $400{,}000 / 40{,}000 = 10$ replicas, an operable count in the same range as the 10 write shards, instead of the 100 replicas a cache-free design would require.

Worked example

A checkout-adjacent online transaction processing (OLTP) service sits at 20,000 requests/second today, with write contention visible as rising lock-wait time on the primary. Applying the roadmap: read replicas ship in week 2, immediately reducing primary CPU spent on reads and giving writes more headroom without touching the schema. Table partitioning on the hottest write table ships by month 2, cutting average write latency on that table because each write now touches a smaller partition and a smaller index. By month 8, write volume has grown enough that partitioning alone is no longer sufficient (write latency is climbing again despite partitioning), triggering the sharding project, sized using the ceiling formulas above against the 100,000 writes/second and 4,000,000 reads/second planning target, landing on 10 shards plus a caching layer to keep the read-replica count out of the triple digits.

Trade-offs & pitfalls

• Read replicas are frequently reached for first because they are the cheapest step, but shipping them and declaring the write-contention problem solved is a common wrong turn: they relieve pressure, they do not add write capacity.
• Partitioning without first identifying the actually-hot tables wastes the migration effort on tables that were never the bottleneck; profile before partitioning.
• Sizing a roadmap only against today's load (20,000 requests/second) instead of the projected target undercounts the shard and replica counts needed and forces a second migration soon after the first; size against the planning horizon, as shown above.
• Cross-shard transactions are the main correctness risk introduced by the long-term step; a design that assumes they will stay rare, without an explicit compensating-transaction pattern for when they are not, tends to produce silent inconsistency under load rather than a clean failure.

Requirements (clarify): 100M global users, 95th-percentile end-to-end latency <200ms, durable messaging, high availability (>=99.95%), geo-aware delivery, cost-conscious.

High-level architecture:

• Clients → Edge POPs (CDN-like) → Global API Gateway → Regional Ingress Brokers → Partitioned Message Stores + Replication → Delivery Workers → Edge Egress POPs → Clients
• Control plane: Metadata service (catalog of partition ownership), Presence service, Routing fabric, Monitoring & autoscaling.

Partitioning / sharding:

• Topic-based then key-hash partitioning: messages partitioned by (topic, user_id % P). Choose P large (e.g., 100k) to allow fine-grained scaling.
• Hot keys handled via dynamic splitting and sticky routing to dedicated partitions.

Replication model:

• Multi-master per region for local writes; async cross-region replication with per-message causal metadata (vector clocks or CRDT-like sequence) and conflict resolution rules. For critical ordered channels, use leader-per-partition with leader in region of majority of subscribers.

Cross-region routing:

• Clients connect to nearest POP. If a subscriber in Region B subscribes to publisher in Region A, the regional broker consults metadata; if data not local, proxy fetches from remote replica (fast-path: recent writes pushed via async tailing to other regions), else read-from-leader via geo-replication. Use WAN-optimized gRPC streams and compression, edge caching for fan-out.

Capacity assumptions (order-of-magnitude):

• 100M users, 10% active/day → 10M DAU; peak concurrent 1M. Avg messages/user/day 50 → 500M msgs/day ~5.8k msgs/sec avg, peak 10x → 58k msgs/sec. Average msg size 1KB → peak throughput ~58 MB/s.
• Storage retention: 7 days → ~3.5 TB/day → ~24.5 TB total. Add replication factor 3 → ~74 TB.

Failure modes & mitigations:

• Region outage: failover via metadata to other regions, client reconnection to nearest healthy POP. Data loss window minimized by tail-based async replication and write-ahead logs.
• Network partitions: degrade to read-local/write-local; reconcile using causal metadata; rate-limit to avoid overload.
• Hot partitions: detect and split, migrate replicas; throttle or backpressure publishers.
• Broker overload: autoscale brokers, circuit breakers, client backpressure.

Three trade-offs:

1. Consistency vs latency: chosen eventual/causal consistency across regions with leader-for-ordered-partitions. Rationale: strict global consensus would break 200ms SLA; causal + leader for critical flows gives practical balance.
2. Replication style: async cross-region replication reduces write latency but risks small window of divergence. Acceptable given conflict resolution and short window; synchronous replication would increase latency and cost.
3. Partition granularity: many small partitions (high P) for elasticity and fine-grained scaling at cost of more metadata and routing complexity. Chosen because expected skew and need to isolate hot keys outweigh metadata overhead.

Observability & ops:

• End-to-end tracing, per-partition metrics, SLA alerting, chaos testing. Automated partition rebalancer, blue/green deploys, and DR runbooks.

Requirements & constraints:

• Ingest 10B events/day (~116k events/sec average, peak 10x).
• Near‑real‑time metrics with ≤1 min latency for critical dashboards.
• Large-scale batch analytics (hours latency).
• Multi‑region with cross‑region aggregation.
• Data residency: EU and India events must stay/replicated per local laws.
• Strong query correctness for some reports; eventual consistency acceptable for others.
• Failure isolation per region.

High-level architecture:

• Edge ingest: regional fleet of API gateways + Kafka (or Confluent Cloud) clusters per region. Use load‑balanced, autoscaled HTTP collectors and client SDK buffering + TLS.
• Regional streaming: each region writes to local durable partitioned Kafka topics. Use compacted topics for metadata, time‑windowed topics for metrics.
• Processing:
  • Near‑real‑time path: stream processors (Flink/KSStreams) in each region computing minute windows, SLAs, anomaly detection. Results written to regional time‑series DB (Druid/ClickHouse/Timescale) tuned for low‑latency queries.
  • Batch path: mirror topics to object storage (S3/GCS or region‑equivalent) as Parquet; run daily/hourly Spark jobs for global analytics in data lake.
• Cross‑region aggregation:
  • For dashboards that can tolerate bounded staleness: periodically (e.g., 1min) export regional aggregates (pre-aggregated rollups, provenance metadata, watermark timestamps) to a global aggregation service via async replication over secure links. Global aggregator merges using deterministic idempotent reducers and watermarking.
  • For correctness‑critical queries (billing, compliance), route query to federated query engine that reads only local region data or runs a cross‑region coordinated query with two‑phase read/validate (see consistency).
• Residency & compliance:
  • Tag events with residency metadata at ingest. Enforce policy via routing: EU/India events must remain in-region; only aggregated, non-identifying rollups can be replicated out after anonymization and legal review.
  • Maintain per-region encryption keys (KMS) and audit logs. Provide export controls and data erasure workflows.
• Replication & durability:
  • Kafka configured with replication factor within region; cross‑region replication via MirrorMaker 2 or tiered storage snapshots to object storage. Use immutable daily segments to simplify replays.
• Consistency & correctness trade-offs:
  • Eventual consistency: use for high‑throughput dashboards where 1–2 min staleness is acceptable; benefits: lower latency, higher availability, simpler failure isolation.
  • Stronger consistency: required for billing/SLAs/compliance. Achieve by reading from authoritative local store plus cross‑region coordination (quorum reads/writes) or by routing queries to single region owning canonicals. Trade-offs: higher latency, more complex cross‑region coordination, lower availability during partitions.
  • Use hybrid model: annotate data/queries with consistency level. Pre-aggregations guarantee monotonic merges using commutative/associative functions; include watermarks and provenance so global aggregator can bound error.
• Failure isolation & operations:
  • Region failure: continue serving unaffected regions; degrade global dashboards to show partial data with clear provenance and confidence intervals.
  • Backfills: replay from immutable object storage into stream processors.
  • Observability: instrument pipeline with metrics, alerting, lineage, SLA dashboards.
• Security & cost:
  • S3 lifecycle + compacted Kafka to control storage costs.
  • Use autoscaling, resource quotas per tenant, and partitioning scheme to avoid hotspots.

Why this design:

• Local ingestion and processing satisfy low latency and residency.
• Pre-aggregation + async cross‑region merge balances scale vs correctness.
• Explicit consistency levels let product owners choose correctness vs availability per use case.
• Immutable storage and deterministic aggregation simplify replays, audits, and compliance.

Framework: split the 90 minutes into rapid evidence-gathering (30m), deep probes (45m), synthesis & scorecard (15m). Use a weighted checklist across six categories; score each item 0–3 (0 = absent/critical issue, 1 = inadequate, 2 = acceptable, 3 = strong), multiply by weight to get weighted score. Provide red/yellow/green thresholds.

Weights (example; totals 100):

• Technical strategy & roadmap: 15
• Architecture quality & scalability: 25
• Talent & team risks: 15
• Security & compliance posture: 20
• Operational readiness & SRE: 15
• Integration complexity & migration effort: 10

90-minute agenda:

• 0–30m: Review key documents (roadmap, infra diagram, inventory, org chart, incident postmortems, audit reports) and ask clarifying Qs.
• 30–75m: Deep probes per category (see checklist).
• 75–90m: Compute scores, highlight top risks, remediation asks.

Checklist (sample probes)

1. Technical strategy (15)

• Product/tech alignment, roadmap clarity, dependency on single tech/vendor, M&A fit. (0–3)

2. Architecture (25)

• System diagrams, modularity, scalability, single points of failure, data model, observability hooks, cloud infra IaC presence. (0–3)

3. Talent & team (15)

• Org chart, key-person risk, hiring pipeline, engineering processes, velocity metrics, attrition. (0–3)

4. Security & compliance (20)

• Recent pen-test, vulnerability lifecycle, IAM, encryption, incident history, regulatory gaps. (0–3)

5. Operational readiness (15)

• CI/CD maturity, automated tests, runbooks, SLOs/SLIs, incident response, backups. (0–3)

6. Integration complexity (10)

• APIs, data formats, auth compatibility, coupling level, migration effort estimate. (0–3)

Scoring rubric & thresholds:

• For each category: weighted_score = (raw_score/3) * category_weight.
• Total possible = 100. Color bands:
  • Green: ≥ 75 — acquisition-ready; only low effort remediation.
  • Yellow: 50–74 — manageable risks; require targeted remediation plan (3–6 months).
  • Red: < 50 — material technical/legal risk; recommend hold or significant contingencies.

Example red flags (immediate escalation):

• Single-developer-owned critical services with no docs
• Unpatched critical vulnerabilities or failed compliance audits
• No CI/CD, manual releases > weekly and high MTTR
• Monolithic, heavily-customized legacy system with no migration path

Deliverable from 90m: one-page scorecard (scores + colors), top 5 risks with severity, quick remediation recommendations and required follow-ups (deep audits, code review, security assessment, org interviews).

Direct answer

Before committing to a plan, spend the first days mapping every team your work actually depends on, get an explicit, dated commitment from each one on what they will deliver, and track those commitments in one visible place so a slip surfaces the moment it happens instead of at the deadline.

Structured elaboration

Map the dependency graph early, not incidentally

Run a short cross-functional session at kickoff specifically to list what you need from other teams: what, by when, and in what form. Treat this as a deliverable of the kickoff, not a side conversation that happens if someone remembers to ask.

Get commitments, not assumptions

"They know we need this" is not a commitment. A commitment has an owner, a date, and an explicit acceptance criterion, meaning what "done" looks like from your side, not just theirs. Ambiguous handoffs are where dependencies quietly slip.

Make status visible continuously, not just at standups

A shared dependency tracker, checked weekly at minimum, with a clear ready, at risk, or blocked status per item, turns a hidden slip into a visible one while there is still time to react.

If you are joining an initiative already in motion

The mapping happens differently. Your first days are spent finding out who currently owns each piece, which may not match the org chart or what the original plan assumed, and estimating the time-to-impact for each dependency, meaning how long before a slip there would actually hit your own critical path (the specific chain of dependent tasks whose delay would directly delay your own delivery date, unlike a dependency that has slack to spare), before you commit to a timeline of your own. Committing to a date before doing this is committing to someone else's assumptions.

Worked example

A project depends on three other teams: one providing a new data feed, one exposing an API endpoint, and one delivering a design system component. At kickoff, the team runs a short dependency-mapping session and gets each provider to commit to a specific date and a specific definition of ready, for the API that means a documented contract and a staging environment, not just "the code exists." These commitments go into a shared tracker with a status column, reviewed weekly.

In week two, the API team's status moves to at risk because their own upstream dependency slipped. Because the tracker surfaced this immediately rather than at the original deadline, there is still time to either help unblock the API team or replan the timeline around a slower path, instead of discovering the problem in the final week when no good options remain.

For the joining-in-progress case: an engineer joins a multi-team initiative already underway. In the first few days, instead of accepting the existing plan at face value, they interview each team named in the plan to confirm who currently owns each dependency, since ownership has quietly shifted since the plan was written, and estimate the time-to-impact of each one: the API dependency would only hurt the timeline if it slipped more than two weeks, while the data-feed dependency has almost no buffer at all. Only after that mapping do they commit to a delivery date of their own, rather than inheriting the original plan's assumptions unchecked.

Trade-offs and pitfalls

A heavy dependency-tracking process on a small, low-risk project wastes more time than it saves; scale the rigor to the size and risk of the dependency rather than applying it uniformly everywhere.

The most common failure is treating the mapping as a one-time kickoff exercise instead of a living tracker. A dependency list that is accurate on day one and never updated again is exactly as useless as never having made one, because the whole point is catching drift as it happens.

Direct answer

An entity-relationship (ER) model has four building blocks: entities (the things you track, like Customer or Order), attributes (their properties), relationships (how entities associate with each other), and cardinality (how many instances of one entity associate with how many of another). A weak entity is one that cannot be uniquely identified by its own attributes alone and borrows part of its identity from an owning entity.

Structured elaboration

• Entities: independent objects or concepts in the domain (Customer, Order, Product). Drawn as rectangles; each instance is a row once the model becomes a table.
• Attributes: properties of an entity or relationship. They can be simple (atomic, like email), composite (address split into street/city/zip), multivalued (phone_numbers), or derived (age computed from birth_date). Exactly one attribute (or a combination) is chosen as the identifying key.
• Relationships: associations between entity instances (a Customer places an Order). Relationships can be unary (an Employee manages another Employee), binary (the common case), or n-ary (rarer, involving three or more entities at once, such as Supplier-Part-Warehouse).
• Cardinality and participation: cardinality states the maximum number of instances on each side of a relationship (one-to-one, one-to-many, many-to-many); participation states whether that side is mandatory (every instance must participate) or optional.
• Weak entities: an entity with no candidate key of its own. It is existence-dependent on an owning ("strong") entity and its identifier is a composite of the owner's key plus a partial key of its own. Example: OrderLine cannot be identified without knowing which Order it belongs to, so its real key is (order_id, line_number).

Worked example

Consider a simple domain: Customer, Order, and OrderLine.

erDiagram
    CUSTOMER ||--o{ ORDER : places
    ORDER ||--|{ ORDER_LINE : contains
    CUSTOMER {
        int customer_id PK
        string email
        string name
    }
    ORDER {
        int order_id PK
        int customer_id FK
        date placed_at
    }
    ORDER_LINE {
        int order_id FK
        int line_number
        int product_id
        int quantity
    }

Mapping this ER model to a normalized relational schema is mechanical:

1. Every strong entity becomes a table; its identifying attribute becomes the primary key. Customer and Order are strong entities here.
2. A one-to-many relationship (Customer places Order) is realized by putting the "one" side's key as a foreign key on the "many" side: customer_id on orders.
3. A weak entity (OrderLine, which cannot exist or be identified without an Order) becomes a table whose primary key is the composite of the owning entity's key plus the weak entity's own partial key: (order_id, line_number), with order_id also carrying an ON DELETE CASCADE foreign key, since a line item has no meaning once its order is gone.
4. Many-to-many relationships (not shown above, but common, like Product-to-Category) become their own junction table with a composite key of the two participants' foreign keys.

Trade-offs and pitfalls

• A common mistake is treating a weak entity's partial key as if it were globally unique on its own (line_number alone) instead of scoping it to its owner (order_id, line_number); this silently allows line 1 of order A to collide with line 1 of order B if the composite key isn't enforced.
• Mandatory-vs-optional participation is easy to skip during modeling but directly decides whether a foreign-key column is nullable, which in turn affects every downstream query that joins on it.
• Over-eager n-ary relationships are rare in practice; most "three-way" associations decompose cleanly into two binary relationships once you look closely, and forcing a single n-ary relationship where two binary ones would do usually loses information about which pairing caused which fact.

Direct answer

I treat learning time as scheduled, protected work rather than whatever's left over after everything else, and I lean toward topics adjacent to what I'm already delivering, so practice and delivery reinforce each other instead of competing for the same hours.

Structured elaboration

Protecting the time honestly: I keep a short, fixed block a few mornings a week, and I'm upfront, including with myself, that an incident-heavy week will eat into it; pretending the block is untouchable just sets up a plan that quietly fails the first time reality intrudes.

Choosing adjacent topics: picking something close to active work means reading directly feeds a task already on the plan, rather than living in parallel to delivery and never getting reinforced, which is usually how learning quietly evaporates.

Learning through the work, not just around it: where possible, I'd rather pick up something new by applying it to a real, if small, piece of committed work than by studying it in isolation first.

Making the trade-off visible: I state it explicitly, in planning or in a one-on-one, that a specific block of time is going toward this, rather than absorbing it as invisible unpaid effort that nobody accounted for and that quietly gets deprioritized under pressure.

A realistic weekly allocation: most weeks it's a modest, fixed slice of time split across delivery, reactive or on-call work, and study, and I say so plainly rather than implying I've found extra hours nobody else has.

Closing the loop: the test that it actually worked is a specific, nameable change in how I do the day job within a defined window afterward, not just a feeling of having learned something.

Worked example

During a stretch with unusually heavy delivery load, I kept two short mornings a week protected for structured logging and observability practices, a topic adjacent to the backend feature work I was already shipping. In an incident-heavy week, that block got sacrificed, and I said so in my next one-on-one rather than pretending it hadn't happened. Because the topic was chosen to reinforce active work, the reading fed directly into a task already on my plan, and within about a month I had changed one specific thing about how I approached that class of work: I started adding structured, searchable log fields to every new endpoint by default, instead of only adding ad hoc debug statements after something broke. The next two incidents on my services got diagnosed from those logs alone, without needing a live debugging session, which is the concrete result, not just a vague sense of having grown.

Trade-offs and pitfalls

The most common failure is scheduling learning time that never survives contact with the first busy week, because it was never actually protected or visible to anyone else who could help defend it. The other is picking topics so disconnected from current work that they never get reinforced by anything real, and quietly evaporate within a few weeks.

Definition & framing:

• SLOs are measurable targets for key customer-facing reliability attributes aligned to the customer's risk appetite (tolerance for failed/slow payments). An error budget = 1 − SLO and represents allowable unreliability that can be “spent” on changes, features, or risk-taking.
• For a payment gateway the risk appetite is typically low for failed/duplicated payments and moderate for short, rare latency spikes. Define SLOs per customer tier (e.g., enterprise vs. SMB).

Key metrics to measure:

• Successful transaction rate (end-to-end payment success): percent of transactions that complete without user-visible error.
• Authorization latency (P50/P95/P99): time from request submitted to auth result returned.
• Checkout availability / uptime: percent time API endpoints are responsive (healthy 200s).
• Transaction durability / consistency: percent of settled transactions without reconciliation discrepancies.
• Error rate by class: transient (timeouts), permanent (declines vs system errors), duplicates.
• Mean time to detect (MTTD) and mean time to repair (MTTR).

Suggested SLO values (examples tied to risk appetite):

• Conservative (low risk appetite, e.g., major merchants): success rate 99.999% (error budget 5.26 minutes/month). Authorization P99 < 500 ms. Availability 99.999%.
• Moderate (most customers): success rate 99.99% (error budget ~52.6 minutes/month). Authorization P99 < 1s. Availability 99.99%.
• Higher-risk-tolerant / experimental: success rate 99.9% (error budget ~8.76 hours/month). Authorization P99 < 2s.

Error budget policy & consequences of breaches:

• Track error budget burn rate in real time. Define burn-rate tiers:
  • Normal: burn within expected rate — continue feature deploy cadence.
  • Elevated (e.g., >2x expected): require gating — temporary freeze on non-critical deployments, add monitoring and runbook readiness.
  • Critical (e.g., full budget consumed): immediate halt to feature releases, roll back recent risky changes, trigger incident postmortem, allocate engineering time to reliability remediation until budget is restored.
• Customer-visible consequences: tiered SLAs (credits) if SLO breaches persist; communication templates and transparency dashboard.
• Remediation obligations: prioritized reliability engineering tickets, increased capacity/auto-scaling, backfill redundancy, or failover activation.

How SLOs should influence prioritization (feature vs reliability):

• Use error budget-based prioritization: if error budget healthy, allow a controlled rate of feature change; if burning fast, shift sprint capacity to reliability until budget recovers.
• Embed SLO impact in planning: every feature PR must include an SLO impact assessment (risk, expected burn). Quantify expected incremental error budget consumption.
• Prioritize work by ROI on error budget: fixes that reduce P99 latency spikes, eliminate class of transient failures, or improve observability rank high. Use a simple scoring: (user/business impact × likelihood reduction) / engineering effort.
• Run SLO-gated release pipelines: automated checks for canary burn, synthetic transaction health, and progressive rollout tied to observed SLOs.

Example operational workflow:

• Define SLO: Transaction success 99.99% monthly.
• Monitor: real-time dashboards + alerts at 99.995% (early) and 99.99% (critical).
• When alert fires: open incident, throttle risky changes, runbook actions (circuit breaker to gateway A, scale consumers), track spent budget, run postmortem with RCA and code/infra fixes prioritized until SLO restored.

Why this approach:

• Ties technical decisions to business risk appetite, makes trade-offs explicit, enables predictable reliability investments, and protects revenue/brand in payments where failures are costly.

Direct answer

For objects larger than about 1MB (megabyte), caching the object's bytes directly in a general-purpose in-memory cache is usually the wrong default; instead, offload the bytes to object storage and cache only a reference (and small, frequently-needed metadata), reserving direct in-memory caching for cases where the object is both large AND accessed with very low latency requirements that object storage cannot meet.

Structured elaboration

• Chunking: for objects that are read partially (a video seek, a large document's specific section), splitting into smaller chunks lets you cache only the actively-accessed portions rather than the whole object, and lets a partial cache hit still provide value.
• Compression: reduces both the memory footprint per cached object and network transfer time, at the cost of CPU for compress/decompress; worth it when memory or network bandwidth is the tighter constraint relative to available CPU.
• Offloading to object storage with cached references: store the actual bytes in a system built for large-object storage (durable, cheap per gigabyte) and cache only a reference (a URL or storage key) plus small metadata (size, content type, a short-lived signed access token if needed); this keeps the fast in-memory cache's precious capacity for many small, hot items rather than a few large ones crowding it out.
• Memory fragmentation considerations: storing objects of widely varying sizes in the same cache (a mix of small config values and occasional large blobs) can cause memory fragmentation in some cache implementations, reducing effective usable capacity below the raw configured limit; segregating large objects into a separate cache instance or tier, sized and tuned differently, avoids this.
• Balancing latency versus cost: caching large object bytes directly in memory gives the lowest possible latency but at high memory cost per item (crowding out many smaller, possibly more valuable cached items); offloading to object storage with a content delivery network (CDN) in front trades a small amount of latency (still fast, just not "already in local memory" fast) for much lower cost per byte and no fragmentation risk to the primary cache.

Worked example

A service serving large generated reports (5 to 50 MB each): rather than caching the full report bytes in the same Redis instance used for small, frequently-accessed session and configuration data, generated reports are written to object storage with a content-addressed key, and only that key (plus small metadata) is cached in Redis; clients fetch the actual report bytes from object storage (behind a CDN for repeat access) using the cached reference, keeping Redis's memory dedicated to the many small, hot items it is well-suited for.

Trade-offs and pitfalls

Caching large object bytes directly "because it's simple" can quietly degrade the whole cache's effectiveness for everything else sharing that cache instance, by consuming a disproportionate share of memory and contributing to fragmentation; segregate large and small objects into different tiers by default rather than only after noticing a problem. Offloading to object storage adds a small amount of latency and a second system to operate (object storage plus, often, a CDN in front of it); this is the right trade for genuinely large objects, but do not apply it reflexively to moderately-sized objects where direct in-memory caching remains the simpler, faster choice.

Direct answer

Shared infrastructure with strong logical isolation (separate namespaces, per-tenant auth tokens, tenant tagging, resource quotas) is usually the right default, since it pools GPU utilization across customers whose peaks rarely align, cutting cost significantly. Dedicated infrastructure per tenant is worth the extra cost when a customer's contractual or regulatory requirements demand a hard blast-radius boundary, where their data or model weights must never be reachable from another tenant's compute, even in a bug scenario.

Structured elaboration

Shared, logical isolation: pooled GPU utilization means one customer's idle hours cover another's peak, most of the cost saving in multi-tenant serving; the security posture depends entirely on the isolation layer (auth, routing, process isolation) being bug-free, since one flaw there is a cross-tenant leak.

Dedicated per-tenant: no pooling benefit, meaningfully more expensive at the same load; a compromise of the isolation layer cannot cross tenant boundaries since there's no shared compute to cross into; more fleets to patch, but any incident is contained to one tenant.

Worked example

20 customers, each needing a peak of 4 GPUs for 2 hours a day, spread through the day. Dedicated:

$\text{dedicated GPUs} = 20 \times 4 = 80$

Shared, sized to the busiest overlap window (at most 6 customers overlapping at once):

$\text{shared GPUs} = 6 \times 4 = 24$

a little over 3x fewer GPUs. That gap is exactly what a customer with hard isolation requirements, like a bank, is asking you to give up when it demands dedicated infrastructure.

Trade-offs and pitfalls

"Logical isolation" is a spectrum: container-level is weaker than VM-level, which is weaker than physically separate hardware. The mistake is treating isolation as binary instead of naming exactly which layer, network, compute, storage, or model weights, needs separation, since compliance often only demands one specific layer.

What the interviewer probes next

Tenant-scoped quotas to stop a noisy tenant from starving others, whether you'd offer a middle tier of dedicated compute with a shared control plane, and how incident response differs between the two designs.