DoorDash Cloud Architect (Mid-Level) Interview Preparation Guide

Overview (6‑month timeline)
Month 0–1: assess maturity, define baseline controls. Months 2–3: pilot with 3 teams (low/med/high maturity). Months 4–5: phased rollout to remaining teams. Month 6: enforcement and metrics gate; continuous improvement.

Onboarding & Training

• Role-based curriculum: executive briefing, architects deep-dive, devs/practitioners hands-on labs.
• Delivery: 2 half-day instructor-led workshops + self-paced modules (Well‑Architected Pillars, cost ops, security, reliability, observability).
• Hands-on: 1-week “Well‑Architected sprint” per team with checklist and remediation playbooks.

Tooling & Automation

• Deploy AWS Well‑Architected Tool + guardrails: AWS Config rules, SCPs, IAM guardrails, cost allocation tags.
• Templates: IaC (Terraform/CloudFormation) starter patterns embedded with best practices.
• Integrations: CI/CD scans, Security Hub findings mapped to W‑A pillars.

Enforcement & Governance

• Tiered policy: advisory (pilot), recommended (after month 3), mandatory (month 6).
• Gate: PR/CD pipeline checks and monthly architecture review board sign-off for production launches.
• Exceptions process: time-boxed risk acceptance with mitigation and owner.

Support Model

• Central Cloud Center of Excellence (CCoE): 2 architects, 1 SRE, 1 security SME as concierge.
• Office hours, Slack channel, templates, remediation squads for critical findings.

KPIs (measurable)

• % of teams completing training within 2 months.
• % workloads reviewed by Well‑Architected Tool. Target 90% by month 6.
• Average number of critical/major findings per workload (decrease 50% vs baseline).
• Time to remediate critical findings (target <30 days).
• % of infra covered by IaC templates and guardrails (target 80%).

This plan balances education, automation, and governance with measurable gates and a support model to accelerate adoption while minimizing disruption.

Design a fixed schema (not a free-for-all), enforce it through a shared logging library rather than convention, control verbosity by sampling low-value log levels on hot paths while always keeping errors, and roll it out by dual-emitting the old and new formats side by side until every dashboard and script that greps the old format has migrated.

Framework

What a log entry should capture. At minimum, every structured log line needs:

• timestamp (ISO 8601, UTC), level, service, env
• correlation_id / trace_id (and span_id if tracing is wired up), so a log line can be tied back to the request it came from
• message (still free text, but now one field among many, not the whole line)
• Context relevant to what's being logged: route, status, duration_ms for a request-handling log; error.type / error.message for a failure

Sensitive fields (user identifiers, emails, anything PII) should never be logged raw. Either omit them, or hash/redact them at the point of emission, enforced by the shared logging library so it isn't up to each call site to remember.

Keep field values bounded, not just field names fixed. A stable schema still has a cardinality trap: a field like route should be a small set of route templates (/orders/{id}), not the raw URL with the real ID interpolated in, and free-text fields like stack traces should be capped in length rather than allowed to grow unbounded. Uncontrolled cardinality is what makes a downstream index expensive, so the schema review has to look at value shape, not just field names.

Why this pays off beyond "logs are searchable now." Once every service emits the same fixed fields, three concrete things get easier that were hard with free-text logs: alert rules can key off a stable field (error.type = "TimeoutError") instead of a regex over a message whose wording changes every release; a post-incident timeline can be reconstructed by filtering and sorting on correlation_id and timestamp across services instead of manually eyeballing each service's raw output in turn; and a legacy service that's risky to touch can be instrumented with this schema before any behavioral refactor, giving you a baseline of real production behavior (error rates, latency, call patterns) to diff the refactor against, turning "did the refactor change behavior" from a guess into something you can actually check.

Verbosity on high-traffic endpoints. Logs and metrics are not interchangeable: a hot endpoint doesn't need a log line per request just to know it's up, that's what a request-count metric is for. Reserve full logging for what metrics can't tell you:

• Always log ERROR and WARN at 100%, since those are rare and high-value.
• Sample INFO-level access logs on high-traffic endpoints (a small, fixed percentage), and keep the sample rate configurable per endpoint rather than global.
• Keep DEBUG out of production by default, gated behind a feature flag or short-lived toggle for active investigations.

Rollout without breaking existing tooling. The riskiest part of this kind of change is not the schema, it's cutting over a service whose logs some existing dashboard or on-call script already greps. A safe sequence:

1. Introduce the shared logging library and have it emit the new structured fields alongside the existing free-text message, so old tooling that parses the raw message keeps working unchanged.
2. Stand up new dashboards/queries against the structured fields in parallel with the old ones, and validate they agree.
3. Once consumers (dashboards, alert rules, on-call runbooks) have migrated to the structured fields, deprecate the old free-text-only path.
4. Roll out service by service, starting with a low-traffic one, not as a single cutover across the whole fleet.

flowchart LR
  A[Service code] --> B[Shared logging library]
  B --> C{Migration phase}
  C -->|during rollout| D["Emit: old free-text message + new structured fields"]
  C -->|after rollout| E[Emit: structured fields only]
  D --> F[Log shipper]
  E --> F
  F --> G["Old dashboards (parse message field)"]
  F --> H["New dashboards (query structured fields)"]

Worked example

Take one high-traffic endpoint doing 5,000 requests/sec. Suppose the team sets a per-endpoint logging budget of 200 INFO-level access-log events/sec for it, to keep total ingestion cost bounded. The required sample rate is:

Meanwhile, if this endpoint's baseline error rate is 0.3%, that's $0.003 \times 5{,}000 = 15$ error events/sec, which stays comfortably inside the budget even at 100% logging, so ERROR/WARN can stay unsampled while INFO gets sampled down to 4%. This is the concrete reasoning a strong candidate walks through: log level and sample rate aren't picked in the abstract, they come out of (a) an ingestion budget you've set and (b) the request/error volume you actually have.

Trade-offs and pitfalls

• Sampling INFO logs means a specific request that had no error and wasn't in the sampled 4% has no log trail. That's an acceptable trade for routine traffic, but pair it with always-log-on-error and, ideally, trace-based sampling that always keeps traces for anything flagged interesting (slow, retried, or touching a specific customer), so the rare interesting case isn't lost to the sample.
• A schema that isn't enforced by a shared library drifts fast: one team adds userId, another adds user_id, and six months later nothing joins cleanly. Put schema validation in the library, not in a style guide.
• Rolling out too fast (cutting the whole fleet to the new format in one release) is the single biggest risk here: any on-call script, saved dashboard, or SIEM rule (a Security Information and Event Management rule: a saved detection query that watches log data for suspicious patterns) that still parses the old free-text line breaks silently until someone notices during an incident, which is the worst possible time to notice. The dual-emit window exists specifically to avoid that.
• Over-redacting can be its own failure mode: hashing or dropping a field that turns out to be needed for debugging (say, an order ID that isn't actually PII) makes incidents harder to resolve. Decide field by field, don't blanket-redact everything that looks user-related.

Direct answer

I would not move the stack wholesale. I would classify every workload against a small set of criteria, move the ones serverless fits, leave the ones it does not, and migrate one workload at a time behind a traffic switch that makes rollback a configuration change. Model servers usually go to serverless containers or a managed inference service (not plain FaaS), feature caches go to a managed cache, and batch jobs go to an orchestrated workflow. The plan fails more often on organization (ownership, on-call, cost accountability) than on technology, so that is a workstream of its own.

1. Assessment criteria: what moves where

Score each workload on these questions:

Applied to this stack:

2. Cutover strategy

Use the strangler-fig pattern: put a routing layer in front, move one workload at a time, and let the old system shrink.

flowchart LR
  A[Assess and score] --> B[Build serverless version]
  B --> C[Shadow traffic]
  C --> D[Canary 5 percent]
  D --> E[Ramp 25 to 100 percent]
  E --> F[Soak 2 weeks]
  F --> G[Decommission K8s workload]
  D -.rollback.-> H[Weight back to K8s]
  E -.rollback.-> H

(K8s above is the common abbreviation for Kubernetes, used to keep the diagram's node labels short.)

• Order: start with a low-risk, stateless, bursty workload (a batch scoring job or a small CPU model) to build the paved road (the standard, supported path other teams then follow: the defaults and tooling that make the safe way the easy way); do GPU serving and anything on the checkout path last.
• Shadow first: mirror live requests to the new path, compare outputs, discard results.
• Canary and ramp: a canary release sends a small slice of real traffic to the new version first, so a bug is caught while it can only hurt a few users. Weighted routing at the gateway or load balancer (5%, 25%, 50%, 100%), each step held through a full daily traffic cycle.
• Data: the feature cache is derived data, so warm the managed cache from the source of truth before the canary rather than dual-writing (writing every update to both the old and new stores in parallel to keep them in sync) forever.

3. Rollback plan

• Keep the Kubernetes deployment running at reduced replicas during the ramp and for a two-week soak (running at full production traffic without incident, long enough to catch issues that only show up over days, such as a slow memory leak or a cache that gradually fills up) after 100%. Rollback is flipping the routing weight, measured in minutes.
• Define rollback triggers in advance: SLO burn rate (how fast you are consuming your error budget, the allowed amount of SLO violation, relative to the rate that would exhaust it before the period ends) above threshold, error rate above baseline, prediction mismatch above tolerance, cost per request above budget.
• Make it possible: no one-way schema or data-format changes during the migration window; if the batch pipeline changes its output format, the old consumers must still read it.
• Rehearse: execute one rollback during the canary on purpose, so the first real one is not the first one.

4. Load-testing before cutover

Serverless fails differently under load, so test the serverless-specific limits, not just throughput:

• Concurrency quota. The default Lambda quota is 1,000 concurrent executions per Region, shared by all functions in the account. Size it with Little's law (concurrency equals arrival rate multiplied by time per request): 800 requests per second at 250 ms average is 800 × 0.25 = 200 concurrent; check the peak, not the average, and request increases ahead of time.
• Scaling rate. Each function can add at most 1,000 execution environments every 10 seconds. A step change larger than that is throttled with HTTP 429 (Too Many Requests) until it catches up.
• Downstream protection. 200 concurrent environments means up to 200 database connections; put a connection proxy in front (a small pooling layer, such as RDS Proxy, that sits between the many function environments and the database and multiplexes their connections into a much smaller, stable pool) or cap the function with reserved concurrency (a per-function ceiling on how many concurrent executions it may run, guaranteed and capped, so it cannot consume the whole account's quota or overwhelm a downstream dependency).
• Cold-start share at burst. Ramp from idle to peak and measure the fraction of requests that hit new environments.
• GPU capacity. For scale-to-zero GPU services, test acquiring GPUs at burst time in the target region.
• Cost under load. Record billed duration and memory during the test to project the monthly bill against the Kubernetes baseline.

5. Organizational and process changes

Pitfalls

• Moving the always-busy workloads. They get more expensive and gain nothing.
• Forcing GPU serving into FaaS or splitting a stateful cache across functions.
• Big-bang cutover, which makes rollback a second migration.
• Treating it as an infrastructure project. Without ownership and cost accountability, a hundred functions become a distributed mess no one owns.

Direct answer

Before you delete anything, you need proof the migration actually succeeded (not just that the new system is live), a defined retention and archival plan for whatever legal or audit obligations outlive the system itself, a rollback path in case something surfaces after decommission that the pre-cutover testing missed, and a communication plan that reaches every stakeholder who might still depend on the old system, including the ones you do not already know about.

Structured elaboration

A decommissioning plan has four parts, and skipping any of them is how "the migration is done" turns into "we deleted data we needed":

1. Verification that the migration is actually complete. Not "the new system works," but "nothing depends on the old one anymore." This means auditing traffic and access logs on the legacy system for weeks after the cutover, not just at the moment of cutover, because low-frequency dependencies (a monthly batch job, a quarterly report) will not show up in a one-week traffic sample.
2. Legal and audit retention requirements. Many systems have data that has to remain queryable for years after the system itself is gone, for regulatory or contractual reasons. That means an archival strategy decided before decommission, not scrambled together after someone asks for five-year-old records the week after you deleted the database.
3. A rollback plan for the decommission itself, distinct from the rollback plan for the original migration. If something surfaces after you have shut the legacy system down (a caller nobody knew about, a data discrepancy only visible under a rare condition), you need a defined path back, even if that path is "restore from the last verified backup and re-enable the legacy code path for a bounded window," not "we have no idea, we deleted it."
4. Third-party integrations and stakeholder communication. External partners often integrate with systems in ways your internal traffic logs cannot see (a partner polling an API you exposed to them specifically). The communication plan needs to reach them with enough lead time to migrate on their side, not just notify internal teams.

Worked example

A team decommissioning a legacy order-management system after a successful migration:

• They keep the legacy system read-only (not deleted) for 90 days post-cutover, monitoring access logs the whole time. In week six, they find a quarterly compliance report job still reading directly from the legacy database, which nobody had flagged as a dependency because it only runs four times a year.
• They export the full historical dataset to a queryable archive with a retention period matching the company's seven-year audit requirement, and verify a sample of archived records against the live system before the live system goes away, since an archive nobody has tested is not actually a safety net.
• They notify the three external partners who integrate with the legacy system's API directly, giving them 60 days' notice and a migration guide, rather than assuming internal migration alone covers everyone with a dependency.
• Only after all of this, and after a final confirmed zero-traffic week on the legacy system, do they actually shut it down, with the archived data and a documented restore procedure kept in case something surfaces later.

Trade-offs and pitfalls

The tempting shortcut is to declare victory the moment the new system handles 100% of live traffic and decommission immediately, but "no traffic this week" is not the same as "no dependencies," and the cost of being wrong (deleted data you needed, a partner integration silently broken) is far higher than the cost of a monitored grace period before deletion. The other common mistake is treating archival as a technical afterthought rather than a compliance requirement with its own sign-off, which is how companies end up unable to produce records a regulator or auditor asks for.

Direct answer

Give every local Prometheus shard its own durable write-ahead-log buffer sized to absorb hours of outage, keep alert-rule evaluation running against local, short-retention storage rather than depending on the remote-write target being healthy, and add a query router that transparently serves recent-window dashboard queries from local TSDBs when the long-term storage backend is unavailable. The design goal is that a remote-write outage degrades only long-range historical queries, not real-time alerting or recent dashboards.

Local, independent alert evaluation

The single biggest design decision is that alert-rule evaluation happens locally, against each Prometheus shard's own TSDB, not against the federated/remote long-term store. That means an LTS outage cannot cause a missed critical alert, since the evaluation path never depended on LTS being up in the first place. Longer-window or cross-service rules that genuinely need federated data are treated as a separate, best-effort class, explicitly allowed to degrade during an LTS outage rather than blocking critical alerting.

Local buffering and backpressure

• The WAL (write-ahead log) already retains recent samples before they're shipped via remote_write; the key design choice is sizing that buffer, and local disk headroom, to survive a realistically long backend outage rather than the default few minutes.
• Export buffer-health as its own first-class metric set: buffer size, oldest-unsent-sample age, and remote-write error/retry rate. These get their own alert ("telemetry pipeline degraded") so a backend outage is itself visible, not silently absorbed until the buffer overflows.
• On recovery, the buffered WAL segments backfill into the LTS backend; this needs to be rate-limited so the backfill burst doesn't overload the just-recovered backend.

Fallback query path

A query router sits in front of dashboards and rule evaluation. For queries within the local retention window, it serves directly from the local Prometheus TSDB; for queries reaching further back, it prefers the federated LTS backend but falls back to "data unavailable for this range" rather than blocking, when LTS is down. This is what keeps dashboards functional (for the recent window that matters most during an active incident) through an extended outage.

Worked example

Buffer sizing. Assume a local Prometheus shard ingests $50{,}000$ samples/sec, and (a stated modeling assumption, in line with typical Gorilla-style TSDB compression, a time-series compression scheme, originally from Facebook's Gorilla paper, that stores timestamps as delta-of-delta and each value as the XOR of its bit pattern against the previous value instead of the raw number) roughly $2$ bytes/sample on disk after compression.

For a target outage tolerance of $D = 4$ hours ($14{,}400$s):

Across a fleet of 20 shards:

For a more aggressive $D = 24$ hours ($86{,}400$s):

Even provisioning for a full day of LTS outage tolerance costs well under 200 GB of local disk across a 20-shard fleet, which is why "buffer generously, backfill on recovery" is the right default rather than trying to minimize buffer size aggressively.

flowchart LR
  Scrape[Local Scrape] --> LocalTSDB[Local TSDB, short retention]
  LocalTSDB --> WAL[WAL Buffer]
  WAL --> RemoteWrite[remote_write Queue]
  RemoteWrite --> LTS[Long-term Storage backend]
  LocalTSDB --> RuleEval[Local Rule Evaluation]
  RuleEval --> Alertmanager[Alertmanager HA Cluster]
  QueryRouter[Query Router] --> LocalTSDB
  QueryRouter --> LTS
  Dashboard[Dashboards] --> QueryRouter

Trade-offs and pitfalls

• Local buffering trades disk cost for resilience; the worked example shows that cost is genuinely small relative to the protection it buys, so under-provisioning the buffer to save a modest amount of disk is a bad trade in nearly every real deployment.
• Local-only alert evaluation means a rule that legitimately needs a cross-shard or cross-region view (e.g. "is this error happening everywhere, not just here") cannot run purely locally; those rules have to be explicitly designed as the best-effort, LTS-dependent class, and their owners need to know that upfront rather than discovering it during an outage.
• Backfilling a large buffered backlog into a freshly-recovered LTS backend without rate limiting can re-trigger the same kind of overload that caused problems in the first place; the backfill path needs its own throttle, separate from the live remote_write path.
• A query router that silently falls back to "no data" for out-of-window queries is better than blocking, but it needs to surface that degradation clearly in the UI; a dashboard that quietly shows an empty long-range chart during an outage, with no indication why, will get misread as "there's no data" rather than "the backend is down."
• The buffer sizing model above assumes a roughly steady ingestion rate; a real deployment should size for peak, not average, ingestion rate, since outages don't reliably happen during quiet periods.

Direct answer
Of the three, I'd respond to the payment failures first, then the public API errors, then the batch job. The payment failures affect a small percentage but include high-value customers and touch money directly, which makes the per-incident cost disproportionately high even at low volume. The API errors affect a much larger share of users (a fifth), which makes them close behind despite lower per-user stakes. The batch job failing on non-critical reports has the lowest urgency because nothing customer-facing depends on it in the next few minutes.

Structured elaboration
Prioritization among simultaneous alerts should weigh three factors together, not any one alone:

• Impact per affected user. A failed payment is a much sharper failure than a slow page load; money and trust are on the line in a way that compounds if it isn't caught quickly.
• Scope. How many users or how much revenue is actually affected right now, not how loud the alert is. A noisy alert with low real impact should never outrank a quiet one with high impact.
• Business criticality and reversibility. Some failures get worse the longer they run (money moving incorrectly, data corrupting further); others are simply delayed (a report that's a few hours late is annoying but recoverable).
  A good way to communicate this under pressure: state the ranking and the one-line reason out loud or in the incident channel immediately, then explicitly assign or acknowledge who is covering the lower-priority items so they aren't silently dropped, even while you focus on the top one.

Worked example
For the three alerts given: payment failures affecting 1% of transactions, including high-value customers, go first, because even a small volume of incorrect financial outcomes needs immediate first-action (typically confirming whether payments are failing safely, i.e., not charging without fulfilling, versus failing unsafely). Second, the public API 5xx errors affecting 20% of users across regions, because the sheer scope makes it the next highest-impact item even though each individual failure is less severe than a payment error. Third, the internal nightly batch job, which is deprioritized to 'someone will pick this up after the first two are stable' since it affects no external users right now. First concrete action on the payment issue: check whether failures are being safely rejected (no charge, clear error to the user) or unsafely processed (charge succeeds but order fails), since that distinction determines whether this needs an urgent stop-the-bleeding action or can tolerate a few more minutes of investigation.

Trade-offs and pitfalls
A common pitfall is anchoring on whichever alert is loudest or has the highest raw volume, rather than the one with the highest actual business impact; a batch job failing nightly can generate more page noise over time than a rare but severe payment issue. Another pitfall specific to this pattern: sometimes a single root cause produces what looks like several simultaneous alerts (a bad deploy triggering a flood of unrelated-looking alerts across a fleet), and prioritizing them as independent problems wastes time that a single fix would resolve; it's worth a quick check for a shared root cause before treating three alerts as three separate incidents. At a higher level, a manager juggling multiple simultaneous incidents across teams faces the same decision with an added dimension: whether the aggregate is severe enough to formally declare a larger, cross-team incident rather than three parallel smaller ones.

Direct answer: For 10PB of historical data and live ETL with zero data loss and minimal downtime, split the problem in two: bulk-transfer the HISTORICAL data using a physical or high-throughput parallel transfer method on its own timeline (it's static, so there's no downtime pressure on it), while migrating the LIVE ETL pipeline via change-data-capture (CDC)-based continuous sync with a short final cutover, and only cut over application traffic once both are verified caught up and consistent.

Structured elaboration. Initial bulk transfer approach: at 10PB, pure network transfer is almost always too slow and too costly (egress and time) compared to a physical transfer appliance (or, if network capacity genuinely supports it, a scheduled high-throughput parallel-transfer job run over weeks); this decision is a straightforward cost/time trade-off calculation once the org's actual available bandwidth is known. Live-sync strategy (CDC): once bulk transfer of the historical baseline completes, start CDC replication from the point the bulk transfer's snapshot was taken, so the target catches up to "now" and then stays current; this requires the source system to retain enough change-log history to bridge the gap between when the bulk snapshot was taken and when CDC starts consuming from it. Validation checks: row/record-count and checksum comparison at the PARTITION level (not the whole 10PB as one unit, which is both slow and gives poor failure localization), so a discrepancy can be traced to a specific partition/time-range rather than triggering a full re-transfer. Cutover plan: once CDC lag reaches near-zero and partition-level validation passes across the full dataset, pause the live ETL write path briefly, drain final lag, do one last validation pass, and cut the application/ETL over to the new lakehouse. Incremental testing phases: validate correctness incrementally as bulk-transfer batches complete (don't wait for all 10PB before testing anything), so schema or format incompatibilities are caught on batch 3 rather than batch 300. Cost estimate: physical appliance/transfer cost plus cloud storage cost for the target (accounting for storage-class strategy: cold-tier for genuinely historical/rarely-accessed data, not uniformly hot storage for all 10PB) plus the engineering cost of the CDC pipeline and validation tooling. Mitigation for data drift/format incompatibilities: run a schema-compatibility check as an early, cheap gate before committing to the full bulk transfer (validate against a representative sample first), and build the CDC pipeline to explicitly reject and quarantine (not silently drop or silently coerce) any record that doesn't match the expected schema, surfacing it for manual review. Metadata/catalog migration: for a Hadoop-based source, the Hive metastore (the catalog mapping table/partition names to their actual file locations and schemas) has to migrate alongside the raw data, not as an afterthought; a lakehouse target needs that same table-to-file mapping rebuilt or imported, or every downstream query breaks even though the underlying files transferred correctly. Source-specific tuning: if the source is a petabyte-scale system like Oracle Exadata, the target lakehouse layout should be designed around the ACCESS PATTERNS the source's own indexing and partitioning reveal (which columns Exadata's indexes were built on, and which partitioning scheme kept its hot queries fast) rather than a generic flat migration, since those patterns are the cheapest available evidence for how to lay out partitioning and file sizing in the new lakehouse. Business impact: track and report the business consequence of the migration in progress, not just its technical status, e.g. the specific reports or SLAs that depend on data freshness during the transition, so stakeholders see a business-relevant view (data as fresh as before vs. degraded) rather than only "batch N of 300 complete."

Worked example. A pragmatic 3-month incremental phasing: month 1, transfer and validate the metastore/catalog mapping plus the coldest, lowest-business-risk partitions first (proves the pipeline and the Hive-metastore rebuild work before anything business-critical rides on it); month 2, transfer the bulk of the remaining historical data in parallel batches by partition/time-range, each validated via checksum before being marked complete, while CDC replication starts catching up live ETL changes since the month-1 baseline snapshot; month 3, close the remaining gap, run the final partition-level validation pass across the full 10PB, and execute the brief write-pause cutover. Concretely, in calendar terms: weeks 1-8 cover the bulk historical transfer described above, week 9 onward is CDC catch-up running in parallel with the tail of the transfer, and the final validation-and-cutover window lands in month 3 once CDC lag is confirmed near-zero.

Trade-offs & pitfalls. Attempting to network-transfer all 10PB directly without first modeling the actual achievable throughput against available bandwidth is a common and expensive mistake; the appliance-vs-network decision should be made from a real bandwidth-and-cost calculation, not a default assumption that network transfer is always simpler.

Direct answer

A cost-cutting initiative should be scoped as an owned delivery project, not a one-time cleanup: assess where the actual waste sits before touching anything, sequence the cheapest, lowest-risk cuts first to build credibility and buy time for the slower structural fixes, assign one accountable owner per workstream, and put a monitoring guardrail in place from day one so the savings don't quietly erode once everyone's attention moves elsewhere.

Structured elaboration

• Assessment approach. Start from actual usage and billing data, not intuition: pull a cost breakdown by service and by team, and separate genuine waste (idle or oversized resources, no lifecycle policy on old data, missed commitment discounts) from cost that's simply the price of real, needed capacity. Only the first category is a legitimate savings target.
• Sequencing the quick wins first. Rank the identified waste by how fast and how safely it can be captured. Rightsizing and cleanup of clearly idle resources are almost always the fastest, lowest-risk wins and should go first, both because they compound and because an early, visible result buys credibility for the slower changes that follow.
• Owner per piece. Split the initiative into workstreams (for example, compute rightsizing, storage cleanup, longer-term commitment purchases) and assign a single accountable owner to each, with the initiative owner tracking the roll-up and any cross-workstream dependency, since an unowned savings target reliably slips.
• Measuring and sustaining savings. Define the savings metric before starting (a specific dollar or percentage reduction against a clear baseline), and, separately, build the guardrail that keeps it from creeping back: tagging enforcement so new resources are attributable, a recurring budget review, and an automated anomaly alert, so a new source of waste is caught within weeks rather than rediscovered a year later in the same state.

Worked example

Monthly cloud spend is $100,000. An assessment against usage data finds about $20,000 a month (20 percent) is attributable waste: roughly $8,000 in idle or oversized compute, $5,000 in storage with no lifecycle or deletion policy, and $7,000 in on-demand spend that should be covered by a reserved commitment. Leadership commits to cutting $15,000 a month (15 percent of total spend, 75 percent of identified waste, leaving a margin since some categories are harder to fully capture) within 10 weeks.

Sequencing:

• Weeks 1 to 2 (quick win, owner: infrastructure engineer): rightsize and shut down clearly idle compute. Saves $7,000 a month.
• Weeks 2 to 4 (owner: platform engineer): apply lifecycle and deletion policies to stale storage. Saves $3,000 a month, cumulative $10,000.
• Weeks 4 to 8 (owner: self, since it needs a finance sign-off on a multi-year commitment): purchase reserved capacity or savings plans covering the remaining steady-state usage. Saves $5,000 a month, cumulative $15,000, hitting the target two weeks ahead of the 10-week deadline.

Sustaining it: tagging is enforced at resource creation so every cost rolls up to an owning team, a monthly budget review compares actual spend against the new $85,000 baseline, and a billing anomaly alert fires if any team's spend jumps more than 10 percent week over week, so a new waste source shows up in days, not at the next audit.

The same shape compresses just as well to a narrower scope, for example a 6-week effort focused on three specific services instead of an entire cloud bill: the same four steps (assess, sequence quick wins first, assign an owner per service, and add the guardrail), just fewer workstreams and a shorter clock.

Trade-offs and pitfalls

The most common failure is announcing a savings target before finishing the assessment, which locks in a number that either turns out to be unreachable or leaves easy money on the table. A second failure is treating the initiative as done once the target is hit: without the tagging and alerting guardrail, teams naturally re-acquire capacity for convenience and the savings erode within a couple of quarters. Watch also for chasing the largest-looking waste category first purely because the dollar figure is biggest. If it is also the slowest and riskiest to fix (like a reserved commitment that locks you in for a year), sequencing it first burns time you needed for the fast wins that actually build momentum and credibility with stakeholders.

Direct answer

A repeatable Total Cost of Ownership (TCO) approach for a 3-year cloud migration is: fix a small set of cost categories, build a low, base, and high estimate per category per year rather than a single number, sum to a 3-year total, and present the range together with the one or two assumptions that actually drive it, instead of a single, falsely precise figure.

Cost categories

• Cloud: compute and storage run rate.
• Networking: data-transfer and egress cost, easy to under-budget because it scales with usage in a way that's less visible than compute.
• Staff: both steady-state operations after migration and the migration labor itself, which is usually front-loaded into year one and often underestimated.
• Licensing: existing software licenses that may be retired or kept during a transition period, plus any new cloud-native tooling now paid for separately.
• Migration: one-time cost of assessment, refactoring, data transfer, and a dual-run period where both the old and new environments are paid for simultaneously.

Handling uncertainty and sensitivity

Treat each category as a range, not a point estimate, and weight the range asymmetrically toward the high side for anything without direct historical data, since migrations of this shape run over more often than under. Rather than sensitivity-testing every category equally, identify the one or two most likely to actually swing the total, usually the one-time migration cost and the ongoing infrastructure run rate, and spend the analysis effort there.

Worked example

For a hypothetical mid-sized application migration (illustrative figures, not derived from any real deal), assume a steady-state post-migration run rate of $180k a year in cloud compute and storage plus $20k a year in networking, $90k a year in steady-state operations staff, and $40k a year in licensing. Year one additionally carries a $120k one-time migration cost and elevated staff cost of $150k, migration labor plus ramping operations, while cloud and networking only reach about half the steady-state rate as workloads cut over gradually through the year.

Base-case 3-year TCO: $410k + $330k + $330k = $1,070k (about $1.07 million).

For sensitivity, apply a downside of minus 10 percent to the one-time migration cost and minus 15 percent to the combined cloud-and-networking line (the two most estimate-sensitive items), and an asymmetric upside of plus 30 percent to migration and plus 15 percent to cloud-and-networking, reflecting that migrations more often overrun than underrun.

Low case: migration $120k times 0.9 is $108k; cloud-and-networking $100k times 0.85 is $85k in year one and $200k times 0.85 is $170k in years two and three. Year 1 low = 108 + 85 + 150 + 40 = $383k; Year 2 and Year 3 low = 170 + 90 + 40 = $300k each. 3-year low = 383 + 300 + 300 = $983k.

High case: migration $120k times 1.3 is $156k; cloud-and-networking times 1.15 is $115k in year one and $230k in years two and three. Year 1 high = 156 + 115 + 150 + 40 = $461k; Year 2 and Year 3 high = 230 + 90 + 40 = $360k each. 3-year high = 461 + 360 + 360 = $1,181k.

The presented range is roughly $983k to $1,181k around a base case of $1,070k, about plus or minus 10 percent, and the two line items that drive nearly all of the spread are the migration cost and the cloud-and-networking run rate.

Trade-offs and pitfalls

The most common mistake is budgeting only the recurring run rate and forgetting the one-time migration and dual-run overlap cost, which routinely turns out to be one of the largest single-year line items. A second mistake is ignoring networking, since data-transfer and egress cost is often billed separately and can quietly become a meaningful recurring cost that a "compute and storage" estimate alone misses. A third is presenting a single-point TCO number as if it were precise; a 3-year infrastructure estimate carries enough real uncertainty, demand growth, price changes, migration scope creep, that a range with named drivers is both more honest and more useful for a budget decision than false precision.

Direct answer

I would implement this as a choreographed or orchestrated saga: a sequence of local transactions, each in its own cloud/service, where every step either succeeds and moves the workflow forward or triggers a compensating action that undoes the effect of prior steps, all keyed by a single saga ID so retries are safe. Since ACID (atomicity, consistency, isolation, durability) transactions cannot span three separately-hosted services, correctness comes from designing every step to be idempotent and every failure to have a defined compensation, not from pretending a distributed transaction exists when it doesn't.

Structured elaboration

Saga pattern, orchestrated vs choreographed: an orchestrated saga has a central coordinator that calls each service in sequence and explicitly triggers compensations on failure, easier to reason about and to test since the whole workflow's logic lives in one place, at the cost of that coordinator being a component every step depends on. A choreographed saga has each service react to the previous step's event and publish its own completion or failure event, no central coordinator, which avoids that single dependency but makes the overall workflow's state harder to observe and debug since it's implicit in the sum of published events rather than explicit in one place. For a supply chain workflow spanning three clouds, I'd lean orchestrated: the cross-cloud network path already adds enough indirection that keeping the workflow logic centralized and inspectable is worth the coordinator dependency.

Idempotency: every step must be safe to retry, since a network partition between clouds means you frequently cannot tell whether a prior call actually succeeded or the response was merely lost. Each request carries an idempotency key (the saga ID plus a step identifier); the receiving service checks whether it has already processed that exact key before applying the operation again, so a retried ReserveStock call after a timeout either finds the reservation already made (no-op, return the prior result) or genuinely makes it once, never double-reserving stock.

Compensation: for every forward step, define its inverse explicitly, upfront, as part of the same design, not bolted on after a failure is discovered in production. Reserving stock compensates with releasing stock; scheduling a shipment compensates with cancelling it. Compensations must themselves be idempotent for the same reason forward steps are: a compensation might also need to be retried.

Eventual correctness: the workflow does not guarantee that inventory and shipping are consistent at every instant (there's a real window where stock is reserved but shipment hasn't been scheduled yet), it guarantees that the workflow always reaches one of two final states: fully completed, or fully compensated back to the starting state, with no state where inventory is decremented but the order was actually cancelled and nobody ever released it.

Testing these patterns: this is not something to reason about only on paper. Concretely: unit test each step's idempotency directly (call the same operation twice with the same idempotency key, assert the second call is a no-op with the same result as the first); integration test the full saga's happy path and every single-step-failure path explicitly (inject a failure at each step in turn and assert the correct compensation chain fires and the system ends in the fully-compensated state); and run chaos-style tests that inject network partitions or timeouts between the specific service calls to confirm retries and idempotency checks behave correctly under real failure conditions, not just under a mocked "the call failed" branch that never actually exercises the retry path.

Worked example

sequenceDiagram
  participant OrderSvc as Order Service (Cloud A)
  participant InvSvc as Inventory Service (Cloud B)
  participant ShipSvc as Shipping Service (Cloud C)
  participant Ledger as Saga Log (event store)
  OrderSvc->>Ledger: append OrderCreated (sagaId, idempotencyKey)
  OrderSvc->>InvSvc: ReserveStock(sagaId, sku, qty)
  InvSvc->>Ledger: append StockReserved or StockReserveFailed
  alt stock reserved
    InvSvc->>ShipSvc: ScheduleShipment(sagaId)
    ShipSvc->>Ledger: append ShipmentScheduled or ShipmentFailed
    alt shipment scheduled
      ShipSvc-->>OrderSvc: SagaCompleted
    else shipment failed
      ShipSvc->>InvSvc: CompensateReleaseStock(sagaId)
      InvSvc->>Ledger: append StockReleased
      InvSvc-->>OrderSvc: SagaCompensated
    end
  else stock reserve failed
    InvSvc-->>OrderSvc: SagaAborted
  end

An order triggers OrderCreated, logged to the saga's event store with a unique sagaId and idempotency key. The orchestrator calls Inventory Service's ReserveStock; if inventory reserves successfully, the orchestrator calls Shipping Service's ScheduleShipment. If shipment scheduling fails (say, no carrier capacity in that region), the orchestrator calls Inventory Service's CompensateReleaseStock, using the same sagaId so Inventory can look up exactly what it reserved for this saga and release precisely that, not just decrement a counter that might now be wrong if another saga has run in between. If a network timeout occurs between the orchestrator and Shipping Service (not a definite failure response, an unknown outcome), the orchestrator retries ScheduleShipment with the same idempotency key; Shipping Service checks whether it already has a shipment recorded for that key, and if so returns the existing result rather than scheduling a second shipment for the same order.

Trade-offs and pitfalls

The main cost of the saga pattern is that it trades strong consistency for availability and cross-cloud feasibility: there is a real window of partial completion visible to anything reading state mid-saga (an inventory dashboard might briefly show reduced stock for an order that ultimately gets cancelled), which needs to be an accepted, designed-for behavior, not a bug someone discovers later. The most common implementation mistake is defining the happy path and forgetting that a compensation can itself fail: what happens if CompensateReleaseStock times out? The answer has to be the same idempotent-retry discipline applied to compensations as to forward steps, plus a dead-letter path and alerting for a saga that's been stuck mid-compensation longer than expected, since an un-monitored stuck saga is a silent data-integrity problem waiting to be discovered by an angry customer rather than by the system itself.