Amazon Cloud Architect (Entry Level) - Comprehensive Interview Preparation Guide

Direct answer
A shared database accessed by many services is operationally simple (one place to back up, one place to query for a report, easy cross-table joins) but structurally couples every service that touches it: a schema change by one team can silently break another team's queries, and there's no way to enforce that only one service writes a given piece of data. Database-per-service removes that coupling, giving each service full control over its own schema and enforcing that it's the only writer of its data, at the cost of needing an explicit strategy for anything that used to be a simple SQL join across tables now owned by different services.

Structured elaboration
With a shared database: cross-service joins are trivial (a single SQL query spanning tables owned by different logical services), transactions across what should be service boundaries are easy (a single atomic, consistent, isolated, durable (ACID) transaction updating two tables), and reporting/analytics can query the whole dataset directly without needing to assemble it from multiple sources. The cost is that these are exactly the things that make it hard to deploy services independently: a schema migration has to consider every service that touches the affected tables, not just the service that conceptually owns them, and there's no enforced boundary preventing one service from reaching into data it doesn't really own.

With database-per-service: each service's schema can evolve on its own release schedule, because no other service can be broken by a change to tables it never had direct access to. The costs show up wherever data used to be joined or transacted across what are now separate services: a query that used to be one SQL join becomes either a replicated read model, an API composition call, or a purpose-built reporting/analytics pipeline that consolidates data from every service (commonly by consuming each service's event stream into a data warehouse); a transaction that used to be a single ACID commit across two tables becomes a saga or another cross-service coordination pattern, accepting eventual consistency where strict atomicity used to be free; and data duplication becomes a deliberate, managed trade-off (a service caching a read-only copy of data it needs from another service) rather than an accident.

Worked example
For reporting and analytics specifically, database-per-service usually means building a separate analytical store (a warehouse or lake) fed by each service's change-data-capture stream or event log, rather than pointing a business-intelligence (BI) tool directly at production service databases; this avoids both the tight coupling of shared production tables and the load a heavy analytical query would otherwise put on a service's transactional database. The operational implications differ sharply by pattern: with a shared database, one team's bad migration or one runaway analytical query can degrade every service at once (a single failure domain); with database-per-service, a failure or slow query in one service's database is isolated to that service, at the cost of needing per-service backup, monitoring, and on-call rather than one shared setup.

Direct answer. Ownership has to be a named role tied to the business function or system the runbook covers, not "whoever wrote it last," with versioning that ties each revision to a specific trigger, not just a date. Catching drift before a real event means testing the runbook during a planned exercise instead of discovering it's wrong while executing it under pressure, paired with a lightweight review trigger whenever something the runbook depends on changes.

1. Ownership

Assign a named owner to every runbook, ideally the person or role accountable for the business function or system it covers, not a shared team inbox and not necessarily the original author, who may move on. Ownership means being responsible for the runbook's accuracy, not personally executing every step during a real event. Make it visible on the document itself (name, role, last-reviewed date) so anyone opening it during an incident can see who to ask if something looks wrong, and so a stale "owner" who's left the organization is an obvious red flag rather than a hidden one. When ownership changes hands, require an explicit handoff with a fresh review as part of the transition, not a silent reassignment.

2. Versioning

Treat runbooks as controlled documents: every change gets a version number, a date, who made it, and why, kept in the document's own change history rather than relying on file metadata or institutional memory. Tie versions to triggers, not just a calendar. A scheduled review is a reasonable baseline, but the more reliable trigger is linking review to the events that actually make a runbook wrong: the system it depends on changes, the org structure around it changes, or a real exercise or incident surfaces a gap. A runbook untouched for eighteen months right after the system it describes was rebuilt is far more suspect than one reviewed on schedule six months ago. Keep prior versions accessible, not just the latest, so a post-event review can tell whether a gap that showed up was already known and fixed but hadn't propagated to whoever was executing, or was genuinely new.

3. Catching drift before it matters

The strongest mechanism is using the runbook for real, on a schedule, inside an exercise (a tabletop discussion or a functional exercise that actually executes some steps), before a live event forces the discovery. A runbook nobody has walked through since it was written is a hypothesis, not a tested procedure. Build a lightweight checklist step into whatever process generates the kind of change that breaks runbooks: when a system change, a supplier change, or an org change is being planned, that process flags which runbooks might be affected, so the review happens close to the change instead of being discovered much later. Track a simple staleness signal across the whole runbook set (time since last review, time since the system or team it covers last changed) and report on it the way you'd report any other risk metric, so an accumulating backlog of unreviewed runbooks is visible to whoever owns the continuity program overall, not discovered runbook by runbook.

Worked example

A payments-processing runbook is owned by the payments platform lead, currently at version 4: v2 added a fallback provider, v3 corrected an escalation contact list after a reorg, v4 followed a functional exercise that found the documented rollback step no longer matched how the system actually rolled back. Six months after v4, the team migrates the payments platform to new infrastructure as part of an unrelated project. Because "infrastructure migration for a system with an owned runbook" sits on that project's change-review checklist, the payments platform lead is flagged automatically and confirms the runbook needs an update before the migration ships, catching the drift as part of the planned change rather than during the next incident. At the next scheduled functional exercise three months later, the team walks through the updated runbook end to end and finds one further step, a monitoring dashboard link, still pointing at the decommissioned infrastructure. That becomes a logged finding and v5.

Trade-offs & pitfalls. A purely calendar-based review cadence catches drift too slowly for a fast-changing system and too often for a stable one; tying review to real change triggers is more work to set up but matches effort to actual risk. Ownership without accountability for staleness (a name on a document nobody ever checks) is barely better than no ownership; the staleness signal has to surface to someone, not just exist. And testing a runbook only during a real event is the worst possible time to discover it's wrong: the exercise programme is what lets a team find the gap on an ordinary afternoon instead of during an actual outage.

RTO (Recovery Time Objective) is how long you're allowed to be down: the maximum acceptable gap between an outage starting and service being restored. RPO (Recovery Point Objective) is how much data you're allowed to lose: the maximum acceptable gap, measured in time, between the last durably captured write and the moment of failure. The requirement "payments must be restored within 30 minutes with no more than 5 minutes of data loss" is literally RTO = 30 min and RPO = 5 min stated in plain language, and each number drives a different part of the design.

What each number drives

RPO = 5 minutes drives replication and backup frequency. A nightly or even hourly backup can't meet this: if the outage happens 4 hours after the last backup, you'd lose 4 hours of transactions, not 5 minutes. A 5-minute RPO effectively requires continuous replication (near-synchronous in-region, or streaming WAL (write-ahead log: a durable, ordered record of every change, written before it's considered applied) or CDC (change-data-capture: a stream of those same row-level changes read off that log) shipping to the DR site) with replication lag actively monitored and alarmed well below the 5-minute budget, plus point-in-time recovery for protection against logical corruption that replication alone would just copy.

RTO = 30 minutes drives standby readiness and failover automation. A cold-standby DR site that has to be provisioned from scratch after the fact will blow past 30 minutes just on infrastructure boot time. A 30-minute RTO points toward a warm standby (already running, sized down, kept current via the same replication that satisfies the RPO) with an automated failover runbook: health-check detection, automated promotion, and DNS/routing cutover, because a manual, human-paged process realistically eats 10-15 minutes just in detection and decision-making before any recovery action starts.

Worked example: what these numbers cost against an annual SLA

A useful way to make the 30-minute number concrete is to check it against annual downtime budgets at standard availability tiers, using $525{,}600$ minutes per year (365 $\times$ 24 $\times$ 60):

A single incident with a 30-minute RTO, if the service is held to a 99.99% SLA, consumes:

of the entire year's downtime budget in one event. That reframes "30 minutes sounds generous" into "this design can absorb roughly one such incident a year and still hit four nines," which is exactly the kind of number that should drive whether the DR design gets warm-standby automation now or gets revisited after the first real incident eats most of the annual budget.

Trade-offs and pitfalls

The most common mix-up is treating RTO and RPO as interchangeable "how bad was it" numbers instead of two independent design constraints: a system can have a great RTO (back up in 2 minutes) and a terrible RPO (lost the last hour of writes) if it fails over to a backup instead of a live replica, or the reverse (RPO≈0 via synchronous replication, but a slow, manual promotion process blows the RTO). Both have to be solved, and usually by different mechanisms: RPO is a replication/backup-cadence problem, RTO is an automation/standby-readiness problem. A second pitfall specific to payments: RPO=0 sounds like the obviously "safer" number to chase, but strict synchronous replication that blocks writes during a replica outage can turn a replication hiccup into an availability incident, trading a data-loss risk you might never hit for a downtime risk you're now taking on every day.

Direct answer

"Zero downtime in two weeks" is two demands that may conflict. Here the migration means moving a live payment service to new infrastructure, and the cutover is the moment traffic switches from the old system to the new one. Availability is the share of time the service works for users. I would first find out what zero downtime has to protect (which users, which functions, how long an interruption is tolerable), then what two weeks is tied to, then lay out the options with their risk and cost. I would take it back as a choice, not a refusal: "Here is what is achievable by that date and what it costs to go further."

Exploring what it has to mean

• "Which services and users must stay up? Is read-only access during cutover acceptable?"
• "What is the longest interruption nobody would notice or complain about?"
• "What happens at 2 a.m. versus at peak? Are there blackout periods (times when changes are forbidden, such as month-end)?"
• "Why two weeks? A contract, an event, a licence expiry, a cost?"
• "What is the rollback requirement (the ability to switch back to the old system) if something fails?"

Feasibility, with the arithmetic

Availability numbers translate into allowed downtime. For a 30-day month (43,200 minutes):

A 99.999% payment cutover across clouds leaves about 26 seconds in a month. 99.9% is often called "three nines" and 99.999% "five nines". A single failed deployment can use that up, so the plan needs a staged cutover (moving traffic in steps rather than all at once), rollback rehearsed, and real-time monitoring.

Dependencies cap the target. A migration adds dependencies: the new cloud's services, a network link, a partner API. Suppose a client expects 99.99% overall but a dependency fails in 5% of hours (an exaggerated, illustrative figure chosen so the effect is easy to see). If every request needs that dependency, the parts are "in series": the whole works only when every part works, so availabilities multiply. 0.9999 x 0.95 = 0.949905, about 94.99%, so the dependency alone caps the system near 95%. Two independent copies reduce the failure to 5% x 5% = 0.25%, giving 99.75% (this assumes the failures are independent). So the answer is design change (redundancy, fallback, retries), not a promise.

Options I would bring back

1. Phased migration in cohorts (separate groups of users moved in turn; for 100,000 users: first 1,000, then the next 10,000, then the next 50,000, then the remaining 39,000), with the ability to roll back each cohort.
2. Read-only window (a few minutes when users can view but not change data), announced in advance.
3. Full dual-running (old and new systems both live and kept in sync, so traffic can move with no gap), which needs more time and budget.

My recommendation, for the stated two weeks: option 1 or 2 with a fixed agreed maximum interruption, and option 3 only if the VP funds more time. What would change my call: a hard legal or contractual reason for no interruption at all.

Capacity or growth uncertainty

If traffic may grow suddenly (viral growth), a capacity shortfall during the cutover is downtime too. Ask for plausible bounds, not one number. Illustrative: today 200 requests per second, low case 300, expected case 600, high case 2,000. Then agree what degrades first if the high one occurs, for example new signups are queued before payments slow down.

Taking it back to the VP

Use a one-page summary: what "zero downtime" is interpreted as, three options with date, risk and cost, my recommendation, and the decision needed by a given day. Lead with what we can deliver, then what each extra increment costs.

Trade-offs and pitfalls

• Saying "impossible", which ends the conversation.
• Promising the number without a rollback plan.
• Forgetting that dependencies set the ceiling.
• Different people reading "downtime" differently: always agree the measurement point.

Direct answer

With heavy Postgres extensions and custom operational tooling already in place, the question is not whether managed is simpler in the abstract, it is whether the managed service actually supports every extension already depended on. If it does, managed usually wins on total cost of ownership (TCO) once operational labor is counted; if even one required extension is not supported, that alone can force self-managed regardless of the TCO math.

Structured elaboration

Decision dimensions

A quantitative TCO method

For a 3-year horizon: TCO equals infrastructure cost times 36 months, plus operational labor hours per month times fully-loaded hourly cost times 36 months, plus expected incident cost, the probability of a serious incident times its average cost, over 3 years.

Worked with illustrative but internally consistent numbers: managed costs $1,200/month infrastructure, roughly 4 hours/month of team time, mostly monitoring and minor tuning, at a fully-loaded rate of $100/hour. TCO equals ($1,200 x 36) plus (4 x $100 x 36), equals $43,200 plus $14,400, equals $57,600 over 3 years, plus a small incident-cost term since the provider absorbs most operational failure modes. Self-managed costs $600/month infrastructure, cheaper raw compute and storage, but roughly 20 hours/month of operational time, patching, backup verification, tuning, on-call for database issues, at the same $100/hour rate, plus an estimated one serious incident over 3 years, a botched failover or a missed backup verification, costing an estimated $15,000 in downtime and recovery labor. TCO equals ($600 x 36) plus (20 x 100 x 36) plus $15,000, equals $21,600 plus $72,000 plus $15,000, equals $108,600 over 3 years.

In this worked scenario, self-managed's lower infrastructure line, $21,600 versus $43,200, is completely swamped by its operational labor line, $72,000 versus $14,400, making managed nearly half the 3-year TCO despite its higher sticker price. The crossover would move toward self-managed only if the team's operational hours per month were far lower, an already-expert dedicated database function running it as a small fraction of their time, or if infrastructure cost dominated at a scale where the managed markup per unit becomes very large.

Decision criteria for when each wins

• Managed wins when every required extension is supported, the team has no dedicated database operations depth, and the honest operational-hours estimate is more than a token amount per month.
• Self-managed wins when a required extension is genuinely unsupported by every viable managed option, a hard blocker rather than a preference, the team already has deep Postgres operations expertise as a sunk cost, so the labor line shrinks toward the managed team's numbers, or the scale is large enough that the managed markup, not the labor, is the dominant cost line.

Worked example

See the quantitative TCO comparison above: given the stated inputs, managed comes out at roughly $57,600 and self-managed at roughly $108,600 over 3 years, entirely because of how much the operational-labor line dominates once honestly estimated.

Trade-offs and pitfalls

• The most common mistake is comparing sticker prices only, $600 versus $1,200, and concluding self-managed is cheaper, without ever pricing the labor line honestly; the TCO method above exists specifically to force that number onto the page.
• Underestimating incident probability for self-managed is a second common mistake: a team that has never had an incident often means it has not been running long enough yet, not that the risk is zero.
• A hard extension requirement should be verified against the managed provider's current supported-extension list before ruling it out, not assumed from an old version of the documentation, since managed providers add extension support over time.

Direct answer

For a platform with spiky, unpredictable traffic, I would run the CPU-bound image-processing workload on serverless functions (FaaS) first, with a small amount of provisioned concurrency for the baseline, behind a queue wherever the SLA (service-level agreement) allows asynchronous processing. The reason is arithmetic: to meet a strict SLA through spikes on Kubernetes, you must keep peak capacity running or accept queueing while nodes boot, and at a 20x peak-to-baseline ratio that is roughly 7x the cost of the serverless option in the example below. I would plan the roadmap so the workload can move to managed Kubernetes when measured average utilization makes containers cheaper (about 48% on the prices below) or when a hard limit such as GPUs or 15-minute runtimes forces it.

Terms used throughout:

• FaaS (functions as a service): the provider runs your function on demand and bills per request and per GB-second (memory size times run time). AWS Lambda is the example.
• Managed Kubernetes: a cloud-run Kubernetes control plane (for example EKS, Amazon Elastic Kubernetes Service) that runs your containers as pods (one or more containers scheduled and scaled together) on servers (nodes) you pay for while they run; a pod autoscaler adds or removes pod copies to match load, and if no node has room, a cluster autoscaler adds or removes nodes themselves.
• Cold start: the delay when the platform creates a new execution environment (download code, start the runtime, run initialization) before it can serve a request.
• Provisioned concurrency: paying Lambda to keep a set number of environments initialized so those requests skip the cold start.
• p99: the 99th percentile latency, the time within which 99% of requests finish.

Assumptions for the worked example

Lambda allocates CPU in proportion to memory, reaching one full vCPU at 1,769 MB, so a CPU-bound function should be sized at 1,769 MB or more, not at the 128 MB default.

LAMBDA_GBS, LAMBDA_REQ = 0.0000166667, 0.20/1e6
PC_ALLOC, PC_DUR = 0.0000041667, 0.0000097222
FG_VCPU_S, FG_GB_S = 0.000011244, 0.000001235
SEC_MONTH = 730*3600
mem_gb = 1769/1024                       # 1 vCPU equivalent
cpu_s = 0.300                            # ASSUMED: 300 ms of CPU per image at 1 vCPU
baseline, spike, spike_min_per_day = 20, 400, 60   # img/s; spikes total 60 min/day
days = 730/24
imgs = (baseline*(24*60-spike_min_per_day)*60 + spike*spike_min_per_day*60)*days
avg_rate = imgs/SEC_MONTH
# Lambda on-demand only
lam = imgs*(cpu_s*mem_gb*LAMBDA_GBS + LAMBDA_REQ)
# Lambda with provisioned concurrency covering baseline (conc = rate*duration, +50%)
pc = int(round(baseline*cpu_s*1.5))
pc_alloc = pc*mem_gb*SEC_MONTH*PC_ALLOC
base_imgs = baseline*SEC_MONTH
pc_dur = base_imgs*cpu_s*mem_gb*PC_DUR
od_imgs = imgs-base_imgs
lam_pc = pc_alloc + pc_dur + od_imgs*cpu_s*mem_gb*LAMBDA_GBS + imgs*LAMBDA_REQ
print(f"images/month={imgs/1e6:.2f}M avg={avg_rate:.1f}/s peak concurrency={spike*cpu_s:.0f} vCPU")
print(f"Lambda on-demand: ${lam:,.0f}/month")
print(f"Lambda + {pc} provisioned envs for baseline: ${lam_pc:,.0f}/month")
def fargate(vcpus): return vcpus*(FG_VCPU_S+2*FG_GB_S)*SEC_MONTH
peak_vcpu = spike*cpu_s/0.7   # run at 70% target utilization
print(f"containers sized for peak ({peak_vcpu:.0f} vCPU always on): ${fargate(peak_vcpu):,.0f}/month")
avg_vcpu = avg_rate*cpu_s/0.7
print(f"containers if autoscaling were perfect ({avg_vcpu:.1f} vCPU avg): ${fargate(avg_vcpu):,.0f}/month")
lam_vcpu_s = mem_gb*LAMBDA_GBS
fg_vcpu_s = FG_VCPU_S+2*FG_GB_S
print(f"$ per busy vCPU-second: Lambda={lam_vcpu_s:.3e} Fargate(1vCPU,2GB)={fg_vcpu_s:.3e} ratio={lam_vcpu_s/fg_vcpu_s:.2f}")
print(f"containers win on compute price above {fg_vcpu_s/lam_vcpu_s:.0%} average utilization")

images/month=94.17M avg=35.8/s peak concurrency=120 vCPU
Lambda on-demand: $832/month
Lambda + 9 provisioned envs for baseline: $813/month
containers sized for peak (171 vCPU always on): $6,178/month
containers if autoscaling were perfect (15.4 vCPU avg): $553/month
$ per busy vCPU-second: Lambda=2.879e-05 Fargate(1vCPU,2GB)=1.371e-05 ratio=2.10
containers win on compute price above 48% average utilization

Reading those numbers before the takeaway: the provisioned count of 9 and the peak concurrency of 120 both come from the same idea (conc = rate x duration in the code). Keeping up with the 20 images/s baseline without queueing needs 20 x 0.3 s = 6 environments busy at once; add 50% headroom for the noise around that average and you provision 9 (pc). The 400 images/s spike needs 400 x 0.3 s = 120 concurrent environments to absorb instantly (peak concurrency=120 vCPU), and sizing containers for that peak at a 70% target utilization (headroom so a real traffic wobble does not immediately throttle) needs 120 / 0.7 ≈ 171 vCPU held always-on.

Provisioned concurrency has two prices, not one: a reservation rate ($0.0000041667/GB-s, PC_ALLOC) charged for every second an environment sits ready whether or not it is handling a request, and a lower execution rate ($0.0000097222/GB-s, PC_DUR, versus $0.0000166667 on-demand) charged only while it is actually running code, because you already paid to reserve it. The baseline's 52.56M images a month (base_imgs, 56% of the 94.17M total) run at that discounted execution rate, about $170 to hold 9 environments ready plus $265 while they are busy; every image above baseline (od_imgs, about 41.6M) still pays the full on-demand rate (about $359), plus the flat per-request fee on all 94.17M images (about $19). That totals the printed $813, a little under the $832 of running every image at the plain on-demand rate, because the discount on the baseline's share of images outweighs the cost of holding 9 environments ready around the clock.

Why 48%, specifically: Lambda's price per unit of actual work is fixed no matter how busy the account is, so its cost per busy vCPU-second is the constant $2.879e-05 printed above. A container's price is fixed per second it runs whether or not it has work, so its cost per unit of actual work falls as utilization rises, a container running at half utilization effectively pays double its sticker price for the work it does. Setting the container's cost per busy second at utilization U equal to Lambda's fixed rate gives the crossover:

Below about 48% utilization the container is paying for idle time Lambda never bills; above it, the container's busy-time price undercuts Lambda's fixed one.

Container cost is priced at Fargate (per-second container) rates for a like-for-like unit price; EC2 virtual-machine nodes under EKS would be somewhat cheaper per vCPU but add the $0.10/hour cluster fee and node management.

The key reading: the realistic Kubernetes number lies between $553 and $6,178, and where it lands depends on how much capacity you must hold warm to survive a spike you cannot predict. Scaling a Kubernetes deployment means the pod autoscaler adds pods, and if no node has room, the cluster autoscaler must launch new nodes, which takes time you must measure in your own environment. Unpredictable spikes of 20x therefore push you toward the $6,178 end. Serverless costs about $830 whichever way the spikes fall. Provisioned concurrency for the baseline is roughly cost-neutral here ($813 versus $832) because the baseline keeps those environments busy, and it removes cold starts from the steady part of the traffic.

Comparison across the six named dimensions

Design specifics for the image workload on FaaS

• Pass references, not bytes. Clients upload to object storage (S3) with a pre-signed URL (a time-limited upload link that needs no credentials); the function receives the object key. Lambda's synchronous payload limit is 6 MB per request and response, too small for many images.
• Asynchronous by default. Where the product allows it, an upload event goes onto a queue (SQS, Amazon Simple Queue Service, AWS's managed message queue) that triggers the function; the queue absorbs spikes and a reserved-concurrency cap (a hard ceiling on how many concurrent environments this function may use, protecting shared account capacity and downstream systems from being overrun; distinct from provisioned concurrency, which keeps environments pre-warmed) protects anything downstream. Only truly interactive paths call synchronously.
• Deterministic output keys (for example thumbnails/{image_id}/{size}.webp) so a retried event overwrites the same object instead of creating duplicates.
• Memory sized for CPU: benchmark at 1,769 MB and above. Extra memory adds more vCPU, but only helps if the image library uses multiple threads.
• Container-image packaging with the native imaging library baked in; keep the image small because image size affects cold starts.

Phased roadmap

Assume the current state is image processing inside an existing monolith on virtual machines.

1. Phase 0: measure (1-2 weeks). Instrument the current path: CPU time per image by size, image-size distribution, real peak-to-baseline ratio, and SLA misses. Replace my assumed 300 ms with the measured number and rerun the cost model.
2. Phase 1: extract behind an interface. Put image processing behind an internal API and an upload event, with the business logic in a plain library that does not know whether it runs in Lambda or a container. This is the lock-in insurance.
3. Phase 2: shadow, then canary, on FaaS. First shadow it: send a copy of production traffic to the function and compare outputs byte for byte and latency at p99, without serving its results to users. Then canary it: route 5%, 25%, 100% of real traffic, with a flag that routes back to the old path instantly. Load-test a synthetic 20x spike before 100%, including a cold-start-heavy run.
4. Phase 3: harden. Provisioned concurrency sized to the measured baseline, reserved concurrency as a ceiling, a DLQ (dead-letter queue, where messages that fail repeatedly are parked for inspection) on the async path, alarms on throttles (invocations rejected because a concurrency limit was hit), errors, duration p99 and queue age (how long a message waits before being processed, a sign the consumer is falling behind).
5. Phase 4: re-evaluate quarterly. Track average utilization (busy vCPU-seconds divided by provisioned vCPU-seconds had it run on containers). If it stays above roughly 50%, or the workload needs GPUs or more than 15 minutes per task, move it to managed Kubernetes. Because Phase 1 kept the logic in a portable container image, that move is a deployment change, not a rewrite.

Migration considerations: run old and new paths in parallel until output parity is proven; keep the old path deployable for rollback through at least one full traffic cycle (for example a month-end peak); make every processing step idempotent (running it twice has the same effect as running it once) so a replay during cut-over is harmless; and budget for the database or storage behind the function, which sees the same 20x spikes once the compute stops being the bottleneck.

Trade-offs and pitfalls

• Comparing FaaS with a perfectly autoscaled cluster. That comparison favors Kubernetes ($553 versus $832) but assumes capacity appears the instant a spike starts, which is the one thing unpredictable traffic denies you.
• Under-sizing Lambda memory for CPU-bound work. At low memory the function gets a fraction of a vCPU and runs proportionally slower, so it can cost about the same while blowing the SLA.
• Ignoring the 2.1x unit-price gap. It is why serverless is a phase for this workload's current traffic shape, not necessarily its permanent home.
• Choosing Kubernetes for portability alone when nobody on the team runs it. The operational burden is real on day one; lock-in cost is only paid if you actually move.

Direct answer

Measuring tail latency across six hops means separating two questions: which hop is actually responsible for a given slow request (attribution), and is the tail getting worse over time (detection). Attribution needs per-hop distributed tracing with sampling that preserves slow traces even when it drops fast ones; detection needs P95/P99 (95th- and 99th-percentile latency, the response times only the slowest 5% and 1% of requests exceed) tracked as their own alertable series, since a stable median (this is the common trap: P99 spikes while the median looks completely healthy) hides exactly this class of problem. Once a hop is identified, the fix is rarely "make everything faster" but a targeted mitigation such as hedged requests, request prioritization, resource partitioning, or admission control aimed at that specific hop, each of which trades some cost or complexity for the latency it buys back.

Measurement and attribution

• Instrumentation: every one of the six services emits a span per request with start/end timestamps, propagated trace context, and enough metadata (host, downstream call outcome, queue wait time) to distinguish "this hop was slow" from "this hop was waiting on the next one."
• Sampling strategy: pure random sampling at low rates (say 1%) will almost never happen to capture a P99 request, since by definition only 1% of requests qualify and the sample and the tail rarely overlap. Use tail-preserving (tail-based) sampling: buffer a trace briefly and only decide to keep it once you know whether any span exceeded a latency threshold, so slow traces are captured close to 100% of the time while typical traces are still sampled cheaply.
• Attribution: once a slow trace is captured, break its total duration into a waterfall of per-hop contributions to see which hop consumed the largest share.
• Spike detection: alert on the rate of change of P95/P99 against a rolling baseline (for example, a sustained jump relative to the trailing window), not on a single fixed threshold, since normal traffic variation would otherwise cause constant false alarms.

Worked example: attributing a P99 spike across six hops

As an illustrative example, not measured data, suppose a captured slow trace shows an 800 ms end-to-end duration split across the six hops as follows:

Hop B accounts for $\frac{300}{800} = 0.375 = 37.5\%$ of the total, the single largest share, so it is the first place to investigate and the first place a mitigation should target, rather than spreading effort evenly across all six services.

Mitigation techniques, their overhead, and their risk

Trade-offs and pitfalls

• Chasing every hop at once instead of attributing first. Without the waterfall breakdown, teams tend to optimize the hop that's easiest to touch rather than the one actually driving the P99.
• Sampling uniformly at a low rate and concluding tail latency "looks fine" because slow traces were simply never captured. The sampling strategy has to be tail-aware, not just cheap.
• Applying hedging without a cap. It is the mitigation most likely to backfire under genuine overload, since it adds load exactly when the system can least afford it.
• Treating any single mitigation here as free. Every row in the table above buys latency at the cost of either infrastructure efficiency, code complexity, or operational risk; picking one should follow from what the attribution step actually showed, not from familiarity with the technique.

Direct answer

This is really two problems on one ticket. The latency-sensitive service's tail latency is a cold-start problem, which Provisioned Concurrency or, where the runtime supports it, SnapStart can often fix without leaving Lambda at all. The background job exceeding Lambda's maximum execution duration is a hard platform ceiling Lambda cannot solve regardless of tuning, so that workload has to move. I'd keep the latency-sensitive path on Lambda with mitigations first, move only the long-running job to Amazon Elastic Container Service (ECS) running on Fargate, AWS's serverless option for running containers without managing servers (or dedicated EC2 if it needs specialized hardware or very high sustained throughput), and re-evaluate the latency-sensitive service's home once real numbers come back from the mitigations.

Structured elaboration

• Lambda's two relevant hard limits: a 15-minute maximum execution duration per invocation, with no configuration to raise it, and cold-start cost that scales with runtime and initialization complexity.
• SnapStart initializes the execution environment once, snapshots it, and resumes from that snapshot on later cold starts instead of re-running full initialization every time. It requires idempotent init code, since the same snapshot can be resumed repeatedly: anything that captures a unique connection, credential, or random value at init time needs an explicit re-initialize-after-resume hook.
• If the latency-sensitive runtime isn't SnapStart-eligible, Provisioned Concurrency is the fallback: size a baseline to typical concurrent executions and scale it with Application Auto Scaling against a schedule or utilization metric, rather than pinning it permanently at peak.
• The background job doesn't get a "mitigate cold starts" decision at all, since Lambda's 15-minute ceiling isn't tunable. ECS/Fargate is the default move for a stateless, containerizable batch job; dedicated EC2 (possibly Spot for cost) is worth it specifically when the job needs sustained throughput at a scale where per-vCPU Fargate pricing loses to Reserved or Spot EC2 pricing, or needs hardware Fargate doesn't offer, such as a GPU.

Worked example

The latency-sensitive service is a Java Lambda function. Enabling SnapStart (available for Java 11 and later, with no code change beyond making static initializers idempotent) is the first thing worth trying, since it's close to free compared to Provisioned Concurrency's ongoing cost. The background job currently runs long enough that it's chained across multiple self-invocations to dodge the 15-minute ceiling, a known anti-pattern: each self-invocation adds its own cold-start and orchestration overhead and is fragile under partial failure, since a failure partway through now has to be handled as "which invocation in the chain failed," not "did the job fail." Moving it to a single Fargate task removes the chaining entirely and lets it run to completion in one execution context.

Trade-offs & pitfalls

• Defaulting straight to "move everything to ECS/Fargate" throws away Lambda's zero-idle-cost and native event integrations for the latency-sensitive service before cheaper, lower-risk mitigations like SnapStart or Provisioned Concurrency have even been tried.
• SnapStart isn't free of gotchas: any state captured in the snapshot (open connections, cached credentials, RNG seeds) needs an explicit re-initialization hook, or you get subtly wrong behavior that only shows up under real load, not in a quick smoke test.
• Chaining Lambda invocations to work around the 15-minute limit changes the failure and retry semantics of "one job" into the failure and retry semantics of "N separate invocations," which usually isn't what anyone intended when the job was first written.
• Splitting the two workloads onto different compute targets is the right call architecturally, but it means maintaining two deployment pipelines and two operational playbooks instead of one, a real ongoing cost worth naming rather than treating as a footnote.

Direct answer

The requirement, an item added must be visible the instant the same user fetches their cart, is a read-your-writes guarantee, not a demand for global strong consistency at all times. Read-your-writes only needs to hold for a user relative to their own prior write, so the right design is not a binary pick between one strongly consistent relational database and one globally replicated eventually consistent store. It is: route each user's cart traffic to a single home region, get strong consistency cheaply within that region, and replicate asynchronously across regions for durability and for the rare case the user changes location.

Structured elaboration

Why a single-region strong RDBMS alone falls short. It trivially satisfies read-your-writes for users near that one region, but every other user pays cross-region latency on every cart operation, and that one region becomes a single point of failure for carts everywhere, which defeats the point of "globally replicated" in the first place.

Why a globally replicated NoSQL store's default mode alone falls short. A globally replicated key-value store's default multi-region mode typically replicates asynchronously with eventual consistency across regions: a write accepted in one region is not guaranteed to be visible to a read served from a different region within any bounded time. If a user's cart read happens to land on a different regional replica than the one that accepted their add-to-cart write, for example because a load balancer failed over between edge points of presence without session affinity, they can see a cart missing the item they just added. That is the exact correctness bug the question is pointing at.

The hybrid design.

1. Home-region routing: assign each user or session a home region (nearest to their first-touch location) and pin both their cart reads and writes to it at the edge, via a routing header or cookie the load balancer inspects, rather than sending requests to whichever region happens to be least loaded.
2. Same-region strong consistency for the common path: within the home region, either a normal relational transaction, or a strongly consistent read against the home region's replica of a key-value store, gives a real, cheap read-your-writes guarantee with no cross-region hop.
3. Asynchronous cross-region replication for durability and travel: replicate cart data outward so it survives a regional outage and is still available, slightly stale, if the user changes location, at which point the system re-homes them to the new nearest region.
4. For the narrow case where a user's device genuinely switches regions mid-session and home-region routing cannot be guaranteed, such as a server-to-server checkout integration without a sticky client, a synchronously replicated, strongly consistent write mode that some managed global key-value services now offer as an opt-in configuration can guarantee any-region reads see the latest write, at the cost of a real cross-region round trip on every write. That cost is acceptable selectively, not as the default for every add-to-cart click globally.

Worked example

Contrast the two failure paths concretely. A user in Sydney adds a jacket. If the client's next read is served by whatever region a stateless load balancer happens to pick, say Frankfurt because Sydney reported a momentary health blip, and Frankfurt has not yet received the asynchronous replica of that write, the fetch returns an empty cart. That is the literal bug. With home-region routing pinned to the Asia-Pacific region plus health-based regional failover, rather than per-request round robin across all regions, that scenario now only occurs during an actual regional outage of the user's home region, a much rarer and more acceptable failure mode than every single request carrying a chance of landing on a stale replica.

Trade-offs and pitfalls

A common pitfall is treating "eventual consistency" as a database property that can be swapped out later; the routing and session-affinity layer is doing the real work here, and changing the underlying database engine alone, relational to NoSQL or the reverse, does not fix a routing bug.

Another pitfall is defaulting to a synchronous, any-region-strong write mode everywhere because it sounds like the safest option. It taxes every single cart write with a real cross-region round trip, commonly tens of milliseconds depending on geography, for a guarantee the overwhelming majority of requests do not need, since most users never switch home regions mid-session.

What would flip the recommendation: the checkout and payment step, not the cart itself, needs a stronger guarantee independent of the cart's consistency model. Preventing a double charge when two payment attempts race from two devices calls for a genuinely serializable transaction (the strictest isolation level: the database guarantees the result is the same as if every transaction ran one at a time, with none interleaved) and an idempotency key at the payment boundary, regardless of how the cart's own reads and writes are routed.

Direct answer

Shared infrastructure with strong logical isolation (separate namespaces, per-tenant auth tokens, tenant tagging, resource quotas) is usually the right default, since it pools GPU utilization across customers whose peaks rarely align, cutting cost significantly. Dedicated infrastructure per tenant is worth the extra cost when a customer's contractual or regulatory requirements demand a hard blast-radius boundary, where their data or model weights must never be reachable from another tenant's compute, even in a bug scenario.

Structured elaboration

Shared, logical isolation: pooled GPU utilization means one customer's idle hours cover another's peak, most of the cost saving in multi-tenant serving; the security posture depends entirely on the isolation layer (auth, routing, process isolation) being bug-free, since one flaw there is a cross-tenant leak.

Dedicated per-tenant: no pooling benefit, meaningfully more expensive at the same load; a compromise of the isolation layer cannot cross tenant boundaries since there's no shared compute to cross into; more fleets to patch, but any incident is contained to one tenant.

Worked example

20 customers, each needing a peak of 4 GPUs for 2 hours a day, spread through the day. Dedicated:

$\text{dedicated GPUs} = 20 \times 4 = 80$

Shared, sized to the busiest overlap window (at most 6 customers overlapping at once):

$\text{shared GPUs} = 6 \times 4 = 24$

a little over 3x fewer GPUs. That gap is exactly what a customer with hard isolation requirements, like a bank, is asking you to give up when it demands dedicated infrastructure.

Trade-offs and pitfalls

"Logical isolation" is a spectrum: container-level is weaker than VM-level, which is weaker than physically separate hardware. The mistake is treating isolation as binary instead of naming exactly which layer, network, compute, storage, or model weights, needs separation, since compliance often only demands one specific layer.

What the interviewer probes next

Tenant-scoped quotas to stop a noisy tenant from starving others, whether you'd offer a middle tier of dedicated compute with a shared control plane, and how incident response differs between the two designs.