Google Solutions Architect (Entry Level) - Comprehensive Interview Preparation Guide

Direct answer

Treat pushback as signal, not an attack: restate the alternative back to the interviewer to confirm you understood it, name the assumption your original choice actually depends on, and check whether the pushback introduces a genuinely new constraint or is just testing your conviction. If it changes a load-bearing assumption, revise the design and say so plainly. If it does not, hold the decision and explain why the alternative loses on the axis that matters here, without getting defensive or repeating yourself louder.

Structured elaboration

Separate what kind of decision is being challenged

A useful first move, often invisible to the interviewer but doing real work for you, is classifying the decision itself:

• A reversible decision (a cache eviction policy, an index choice, a queue's retry backoff) can be tried, measured, and changed later at low cost. It is fine to say "I'd start with X, and revisit once we have real traffic data" and mean it.
• A largely irreversible decision (the primary datastore for a dataset that will grow to hold years of production data, a data-residency architecture with legal constraints attached) is expensive to unwind once built. These deserve a firmer defense, because "we'll just change it later" is not actually true for them.
  A candidate who signals which category their choice falls into is showing exactly the judgment this kind of pushback is designed to probe.

The actual steps, in order

1. Paraphrase the alternative back ("so the question is why not do X instead of what I proposed"). This confirms you understood the objection rather than reacting to a version of it you invented, and buys you a beat to think.
2. State the assumption or constraint your original choice depended on, out loud. This is the load-bearing piece: if that assumption is still true, your choice still holds; if the interviewer's follow-up just knocked it down, you now know exactly what to revise.
3. Ask, explicitly if needed, whether the pushback is introducing new information (a constraint you did not have, or did not weight correctly) or is testing whether you actually understand your own trade-off. Those call for different responses.
4. Decide: hold, revise, or partially revise (keep the core choice, adjust a parameter). Say which one you are doing and why, in one sentence.
5. Move on. Do not keep re-litigating a decision you already reopened and closed; that reads as insecurity, not thoroughness.

A worked dialogue skeleton

Interviewer: "Why would you use a queue here instead of just calling the downstream service directly?"

Candidate: "So the question is whether the extra moving part, the queue, is worth it compared to a direct synchronous call. My choice assumes the downstream service is slower and less reliable than the caller can afford to block on, so decoupling protects the caller's own latency and gives us a retry point if the downstream service is briefly unavailable."

Interviewer: "What if that downstream service is actually one of the most reliable and fast services we operate?"

Candidate: "That changes the assumption I was leaning on. If it is genuinely fast and reliable, the resilience argument for a queue weakens a lot, and a direct call with a short timeout and a couple of retries might be simpler and just as safe. I would want to know its actual latency and error behavior before committing either way, but I would not stubbornly keep the queue just because that is what I said first."

Interviewer: "And if it were the flakiest service in the system instead?"

Candidate: "Then I would hold the original call. A flaky downstream dependency is exactly the case the queue protects against, buffering the caller from its failures and giving us retry and backpressure without cascading the failure upstream."

Notice the candidate did not fold immediately in the second exchange, and did not dig in reflexively in the third; the answer changed only where the underlying assumption actually changed.

Trade-offs & pitfalls

• Caving on every objection is the most common failure mode: treating any pushback as proof you were wrong signals you did not have real conviction in the first place, and an interviewer who sees you reverse instantly on a restated version of your own design will keep pushing to find the floor.
• Stonewalling is the opposite failure and just as damaging: repeating your original justification louder, or refusing to update even when the interviewer has handed you a genuinely new constraint, reads as an inability to incorporate new information, which is the exact skill system-design interviews are trying to probe.
• Relitigating from scratch instead of anchoring on the specific new point wastes time and often talks yourself into a worse answer than the one you started with; stay anchored to the one assumption that was actually challenged.
• Treating every decision as equally reversible is a subtler pitfall: defending a cache TTL choice and defending your core datastore choice with the same intensity misses that one of them is cheap to revisit later and one is not. Senior candidates spend their conviction where it is actually load-bearing.
• The strongest signal is not being right on the first guess, it is showing a clear, repeatable process for deciding whether to hold or revise, and being transparent in the moment about which one you are doing.

Direct answer

Detect PII with a fast deterministic layer (regex/pattern rules for structured PII like emails, SSNs, phone numbers) combined with an ML classifier for unstructured or contextual PII (names, addresses in free text), and treat neither as sufficient alone. Enforce redaction at the collector, not the agent or the storage layer alone: agent-side redaction is best for privacy but hardest to update and audit across a fleet, storage-side is the last line of defense but means raw PII already crossed the network. Use deterministic, salted tokenization (not plain masking) so debugging can still correlate the same underlying value across events without ever storing or logging the raw value, and produce an immutable audit record for every redaction decision so you have concrete evidence for an auditor, not just a policy document claiming it happens.

Structured elaboration

Detection techniques:

• Regex/pattern rules: fast, deterministic, good for structured PII with a known shape (email, SSN, credit card, IP). Cheap enough to run on every event inline.
• ML classifiers: needed for unstructured, contextual PII (a name or address embedded in free-text log messages) that no fixed pattern reliably catches. Higher recall on the hard cases, but adds latency and needs periodic retraining/evaluation as it can drift.
• Hybrid: run regex first (catches the bulk of structured PII cheaply); route ambiguous or high-risk fields to the ML classifier as a second pass.

Masking versus tokenization:

• Deterministic tokenization (HMAC, a one-way keyed hash function, of the value with a per-tenant secret salt): the same raw value always maps to the same token, which preserves the ability to correlate "this is the same user across these events" for debugging, without the token being reversible back to the raw value.
• Reversible tokenization (format-preserving encryption, encryption whose output keeps the same shape and length as the input, via a KMS-backed vault): needed only when an authorized party may legitimately need the original value back, gated behind strict access control and its own audit trail, separate from the deterministic path used for ordinary debugging.
• Partial masking (j***e, a****@domain.com): preserves a debugging hint about shape/length without preserving correlatable identity; use where correlation isn't needed, just "this field had a name-shaped value."

Enforcement point, and why it's a layered decision, not a single choice:

Auditability: every redaction decision writes an immutable audit record: input hash (never the raw value), detection method, confidence score, rule/model version, timestamp, event ID. This is what turns "we redact PII" into something an auditor can actually verify happened, on a specific event, using a specific rule version, rather than a claim about the system's design.

flowchart LR
    A[Agent] -- light regex mask --> C[Collector]
    C --> RX[Regex pass]
    RX -- ambiguous field --> ML[ML classifier pass]
    RX -- clear match --> TOK[Deterministic tokenization]
    ML --> TOK
    TOK --> ST[(Redacted storage)]
    TOK --> AUD[(Immutable audit log)]
    ST -- storage-level guard --> ST

Worked example

Combined detection recall. Assume the regex layer alone catches 92% of true PII instances (illustrative assumption), and the ML secondary pass catches 70% of what the regex layer missed:

At an assumed $10{,}000{,}000$ PII-bearing fields/day across the fleet:

This is the number that justifies the storage-level guard as a real requirement rather than a nice-to-have: even a well-tuned two-stage detector with 97.6% combined recall still lets 240,000 fields/day through in this example, which is why enforcement can't stop at the collector alone for a regulated customer.

Audit log storage overhead. Using the pipeline's ingestion baseline of 200,000 events/sec, assume 15% of events trip at least one detector:

At 120 bytes/audit record (hash + method + confidence + rule/model version + event ID):

Three hundred gigabytes a day of audit trail alone is a real storage line item, not a footnote, worth surfacing explicitly when scoping this design rather than discovering it after the audit log has been running unmanaged for a month.

Trade-offs & pitfalls

Common wrong turns: treating any single detection layer as sufficient and skipping the residual-leakage math above, which is what a regulated auditor will actually ask for ("what's your false-negative rate, and what's the backstop"); using reversible tokenization everywhere by default because it's more flexible, when most debugging only needs deterministic correlation and every reversible token is a standing risk that needs its own access control and audit trail; and logging the audit record with the raw value "just in case," which defeats the entire point of redaction by creating a second place PII lives, this time inside the system meant to prove PII doesn't land anywhere.

Direct answer. For a warehouse ingesting 5 TB/day raw with 3 years of raw retention plus 5 years of monthly aggregated snapshots, the storage estimate is dominated by the raw tier (roughly 1.4 PB compressed) with the aggregated tier adding a comparatively small amount (tens of TB); compute sizing for 100 concurrent medium-complexity users is a separate calculation driven by concurrency, not by the storage volume.

Structured elaboration and calculation. Stated assumptions, since none of these numbers exist without them: a 4:1 columnar compression ratio (a reasonable mid-range assumption for typical structured event/log data in Parquet-family formats; verify against actual data if available rather than trusting this blindly) and that a monthly aggregated snapshot is roughly 2% the volume of one day's raw ingest (a placeholder for 'far smaller than raw' that should be replaced with a real number once the aggregation's actual cardinality is known).

daily_raw_tb = 5
raw_retention_years = 3
compression_ratio = 4
raw_retained_tb = daily_raw_tb * 365 * raw_retention_years / compression_ratio
# = 5 * 365 * 3 / 4 = 1368.75 TB

agg_pct_of_raw = 0.02
agg_retention_years = 5
monthly_snapshots = agg_retention_years * 12  # = 60
agg_snapshot_tb = (daily_raw_tb * 30 * agg_pct_of_raw) / compression_ratio
# = (5 * 30 * 0.02) / 4 = 0.75 TB per snapshot
agg_total_tb = agg_snapshot_tb * monthly_snapshots
# = 0.75 * 60 = 45 TB

total_tb = raw_retained_tb + agg_total_tb  # = 1413.75 TB

Run (verified): raw retained storage = 1,368.8 TB, aggregated snapshot storage = 45.0 TB, total = ~1,413.8 TB (~1.4 PB) compressed.

Compute sizing for concurrency (rough calculation). Storage volume and query compute are largely independent concerns: 100 interactive users running medium-complexity reports is primarily a CONCURRENCY sizing problem (how many queries can run simultaneously without contending for the same compute slots), not a volume problem -- a warehouse holding 1.4 PB can still serve 100 concurrent users fine if each query only scans a well-pruned slice (recent partitions, relevant columns) rather than the full historical dataset. The rough calculation needs its own stated assumption: a 'medium-complexity' report occupies roughly 2 vCPU-equivalent and 4GB RAM of compute-slot footprint while running (a placeholder for the warehouse's own documented per-slot sizing, to be replaced with a real number once available).

concurrent_users = 100
vcpu_per_query = 2
ram_gb_per_query = 4

# Worst case: size for ALL 100 users querying at the exact same instant, zero queueing tolerance
peak_vcpu = concurrent_users * vcpu_per_query
peak_ram_gb = concurrent_users * ram_gb_per_query
# = 200 vCPU-equivalent, 400 GB RAM at peak concurrency

Run (verified): sizing for zero-queueing peak concurrency requires roughly 200 vCPU-equivalent and 400 GB RAM of query-serving compute. This is a conservative upper bound, not a final number: if a brief queueing delay (a few seconds) is acceptable rather than requiring every one of the 100 users to get an instant, uncontended slot, the warehouse's own concurrency/queueing model (most cloud warehouses queue excess concurrent queries rather than rejecting them) can size meaningfully below this peak-simultaneous figure -- that queueing-aware refinement is warehouse-vendor-specific and should be validated against the actual platform's documented concurrency-scaling behavior rather than assumed generically here.

Cost considerations. The two calculations above translate directly into the warehouse's two largest cost lines, and each responds to a different lever. Storage cost scales with the 1.4 PB figure and is the more forgiving of the two to optimize after the fact: raw data that ages past its most-queried window is a natural candidate for a cheaper storage tier (most warehouses offer a discounted tier for infrequently-scanned partitions), and the 4:1 compression assumption is itself a direct cost lever, since a real compression ratio measured at 6:1 or 8:1 on the actual data (columnar formats often compress better than this mid-range assumption, depending on column cardinality) would proportionally cut storage cost with no architecture change. Compute cost for the 100-concurrent-user workload is the less forgiving of the two: warehouse compute is typically billed per query-second or per reserved compute-slot-hour, so provisioning for the zero-queueing peak (the 200 vCPU-equivalent figure above) around the clock is materially more expensive than sizing to a lower steady-state baseline and letting the warehouse's own autoscaling or query queue absorb short bursts above it -- the queueing-tolerant sizing discussed above is therefore a real cost decision, not just a technical nicety: it trades a few seconds of occasional query queueing for a meaningfully smaller reserved compute footprint.

Trade-offs & pitfalls. Every number here traces back to a STATED assumption (compression ratio, aggregation size ratio, per-query compute footprint) that should be flagged explicitly to whoever consumes this estimate, and revisited once real data is available -- presenting 1.4 PB or 200 vCPU as hard facts rather than 'computed estimates under these assumptions' is how capacity plans quietly become wrong. Buffer/cache sizing for the 100-concurrent-user compute layer is a separate, non-trivial follow-up estimate (working-set size for commonly-queried recent partitions) that should not be conflated with the raw+aggregate storage total computed here.

Direct answer

HIPAA (the Health Insurance Portability and Accountability Act, the US law governing protected health information, PHI) is not something Google certifies you compliant with by using a service. Google signs a Business Associate Agreement (BAA) covering in-scope services, and Cloud Storage, BigQuery, and GKE (Google Kubernetes Engine) are all BAA-eligible, but the customer remains the party responsible for how those services are actually configured and used. So the architecture has to make the compliant configuration the only reachable one, not merely a possible one.

Architecture

flowchart LR
    U[Clinician application] -->|TLS| LB[HTTPS load balancer]
    LB --> APP[GKE workload: private cluster]
    APP -->|CMEK encrypted| GCS[Cloud Storage: PHI documents]
    APP --> DLP[Sensitive Data Protection: de-identify]
    DLP -->|CMEK encrypted| BQ[BigQuery: de-identified analytics]
    APP --> LOG[Cloud Audit Logs]
    GCS --> LOG
    BQ --> LOG

• Encryption at rest and in transit. All three services encrypt at rest by default, but PHI-holding buckets and BigQuery datasets get CMEK (customer-managed encryption keys, via Cloud KMS, Cloud Key Management Service) so the customer controls the key's lifecycle and can revoke access to the underlying data by disabling the key, something a HIPAA audit typically expects specifically for PHI. In transit, TLS end to end, terminated at the load balancer and, if a service mesh is in play, mutual TLS between GKE pods internally.
• Audit logging. Cloud Audit Logs' Data Access logs on Cloud Storage and BigQuery are off by default and have to be explicitly enabled; this is the single most commonly missed HIPAA control on this stack. Export them to a separate, access-restricted logging project so the audit trail cannot be altered by whoever has access to the workload itself.
• VPC (Virtual Private Cloud) Service Controls. A perimeter drawn around the Cloud Storage, BigQuery, and GKE-adjacent APIs that handle PHI, so that even a correctly-authenticated request headed somewhere it should not go, copying data out to a personal bucket outside the organization, for instance, is blocked at the network and API boundary, not left to IAM (Identity and Access Management) alone to catch.
• Sensitive Data Protection (formerly known as Cloud DLP, Data Loss Prevention). Used two ways here: to classify what data actually is PHI on ingest, and to de-identify or tokenize it before it lands in any lower-sensitivity copy, such as an analytics dataset a broader set of staff can query.

Documentation and attestation to expect from the customer

The signed BAA itself, and Google's published compliance documentation showing why the specific in-scope services are permissible for PHI. On the customer's side: a completed risk analysis or security risk assessment, which the HIPAA Security Rule requires independent of which cloud provider is used; an inventory of exactly which systems and data flows actually touch PHI, since that inventory is what justifies where the VPC Service Controls perimeter boundary is drawn; and access review records showing least-privilege IAM grants are actually audited on an ongoing basis, not merely configured once and left alone.

Trade-offs and pitfalls

VPC Service Controls perimeters are disruptive if drawn too tightly, legitimate cross-project data flows fail silently with denials that look like generic permission errors rather than perimeter violations; start in dry-run mode and widen the perimeter's ingress (inbound) and egress (outbound) rules deliberately, rather than iterating against a fully enforced perimeter. CMEK adds an availability dependency: if the key is disabled or its IAM revoked, every read against that data fails immediately, so changes to a CMEK key's IAM need the same change-control rigor as a production deploy, not an ad hoc edit. The most common wrong turn is treating "HIPAA compliant" as a property of which services were chosen rather than how they are configured: the exact same three services can be either compliant or a serious violation depending entirely on whether Data Access logging, CMEK, and the VPC Service Controls perimeter are actually turned on and correctly scoped.

Direct answer

The decision comes down to how expensive an interruption actually is for this specific service. A reserved instance or Savings Plan (SP) covers a guaranteed baseline at a discount with zero interruption risk; spot buys a deeper discount in exchange for eviction risk. For a moderately stateful, throughput-oriented service, the right structure is usually a reserved or Savings Plan floor sized to the steady minimum load, so core capacity never carries interruption risk, plus spot covering the elastic portion above that floor, with on-demand as the fallback when spot capacity isn't available, validated incrementally rather than committed to at full scale on day one.

Structured elaboration

What you need to know before deciding

• Baseline and peak utilization, so you know how much load is genuinely steady versus how much is elastic.
• Historical interruption rate for the specific instance family and region you'd run spot on; this is workload-specific and should be measured, not assumed from a generic industry figure.
• Mean time to recovery (MTTR): how long it takes the service to recover from an interruption, and what that recovery actually costs, in replayed work, extra network or storage I/O (input/output), or a brief latency hit.
• The specific nature of the statefulness: session affinity, whether writes go through a write-ahead log, how frequently the service checkpoints. "Moderately stateful" is doing a lot of work in this question; a service that checkpoints every few seconds tolerates interruption very differently from one that holds long-lived in-memory session state.

Portfolio design

$\text{Monthly}_{\text{mix}} = \text{FloorHours} \times \text{ReservedRate} + \text{ElasticHours} \times (1+\text{RetryOverhead}) \times \text{SpotRate}$

Size the floor to the steady minimum load the service never drops below, covered by reserved capacity or a Savings Plan so it's never at interruption risk. Size the elastic band to the variable load above that floor, covered by spot, with on-demand as a last-resort fallback when spot capacity isn't available in the target instance family or region. This mirrors the same floor-plus-elastic-band logic that applies to a large batch-job fleet: there, the floor is whatever backlog has to clear even in a slow week, and the elastic band is burst capacity, which is a much easier target for spot than a live stateful service because batch jobs tolerate restarts far more cheaply.

Validating the mix before committing at scale

• Canary a small percentage of production traffic onto the mixed fleet first, with autoscaling and an on-demand fallback path already wired up, rather than assuming the mix works and finding out otherwise in production.
• Run controlled interruption tests (deliberately terminating spot capacity in the canary) to validate the MTTR and recovery-cost assumptions against reality, not just the historical interruption-rate figure.
• Track cost per month, tail latency (P99, 99th percentile), error rate, and any lost or replayed work as the canary scales up, and only widen the spot percentage once those metrics hold at each step.

Worked example

Suppose the service needs a steady floor of 600 instance-hours/month plus an elastic band averaging 400 instance-hours/month on top, for 1,000 hours/month total. On-demand costs $0.10/hour, a 1-year reserved commitment (amortized) costs $0.06/hour, and spot costs $0.02/hour. Because this service is stateful (checkpoint and replay cost money, unlike a stateless batch job), assume a higher interruption-driven retry overhead than a purely stateless workload, 15%, based on the higher end of a typical measured range for this kind of workload.

Option A, all reserved:

$1{,}000 \text{ hrs} \times \$0.06 = \$60.00 \text{ per month}$

Zero interruption risk, but paying for the full 1,000-hour floor at all times even though only 600 of it is steady load.

Option B, floor plus elastic mix:

$\text{Floor}: 600 \times \$0.06 = \$36.00$

$\text{Elastic band with 15\% retry overhead}: 400 \times 1.15 = 460 \text{ effective hours}, \quad 460 \times \$0.02 = \$9.20$

$\text{Total}: \$36.00 + \$9.20 = \$45.20 \text{ per month}$

Savings:

$\$60.00 - \$45.20 = \$14.80 \text{ per month}, \quad \frac{14.80}{60.00} \approx 24.7\% \text{ cheaper than covering the whole workload with reserved capacity}$

That saving comes in exchange for accepting eviction risk on the 400-hour elastic band, plus whatever operational cost comes from handling those interruptions gracefully, which isn't captured in this dollar figure and has to be validated separately through the canary process above.

Trade-offs and pitfalls

• Sizing the floor too small exposes steady-state load to interruption risk it shouldn't have to carry. This is the specific mistake the "moderately stateful" framing in the question is pointing at: a stateful service often can't just retry cheaply, so the floor needs to protect whatever load genuinely can't tolerate an interruption.
• Sizing the floor too large gives up savings a fault-tolerant elastic band could have captured, effectively turning the whole workload back into option A without admitting it.
• Skipping the incremental validation step means finding out the MTTR assumption was wrong in production, at full scale, instead of in a controlled canary test where the blast radius is small.
• Reusing a generic industry interruption-rate figure instead of measuring your own misprices the whole decision, since interruption rates vary meaningfully by instance family, region, and time.

Direct answer. For sub-millisecond end-to-end latency financial trade processing, a self-managed streaming infrastructure, purpose-built and co-located with the trading systems, is almost always the right choice; managed streaming services introduce network hops and multi-tenant scheduling variability that make a consistent sub-millisecond target very difficult to guarantee.

Structured elaboration.

• Determinism. Trading systems need not just low average latency but low tail latency (consistent worst-case behavior); managed multi-tenant services share underlying infrastructure with other customers' workloads, which can introduce latency jitter that a self-managed, dedicated, co-located system avoids by design.
• Latency. Sub-millisecond end-to-end typically requires kernel-bypass networking (letting the network card hand data straight to the application, skipping the operating system's normal, slower network-processing path), co-located hardware, and a streaming layer tuned specifically for this profile; general-purpose managed streaming services are architected for good latency at scale and operational simplicity, not for shaving off the last hundred microseconds a trading system needs.
• Failover behavior. A managed service's failover is opaque and tuned for its general customer base's needs, not necessarily for a sub-millisecond SLA; a self-managed system lets you engineer failover behavior (hot standby, deterministic failover timing) specifically to preserve the latency budget even during a failure.
• Compliance. Financial trading infrastructure is often subject to specific regulatory requirements around auditability, timestamping precision, and data locality that may be easier to satisfy with infrastructure you control directly than with a managed service's general-purpose compliance posture, though many managed services do offer relevant certifications and this should be verified case-by-case rather than assumed.
• Operational cost. Self-managing at this level requires specialized low-latency-systems expertise that is expensive to hire and retain; this is the real cost of the recommendation, not the infrastructure itself.

Worked example. A firm building a new trading system with a genuine sub-millisecond requirement should not default to a general-purpose managed streaming service and hope it is fast enough; instead, benchmark the actual achievable latency of the managed option under realistic, sustained load (not a synthetic single-message test) early in the design process. If the managed option cannot consistently meet the target even under favorable conditions, and for genuinely sub-millisecond targets it typically cannot, self-managed, co-located infrastructure becomes the only viable path, and the team should budget for the specialized engineering expertise that comes with it as a first-class cost of the project, not an afterthought.

Trade-offs and pitfalls. The most common mistake is under-scoping how firm the sub-millisecond requirement actually is: if the true requirement is closer to single-digit milliseconds for most of the system with only a narrow, isolated piece genuinely needing sub-millisecond determinism, a hybrid design, managed streaming for the bulk of the system with a purpose-built, self-managed path for the latency-critical piece, is often both cheaper and more maintainable than self-managing the entire pipeline. Validate the actual requirement's scope before committing to the more expensive, fully self-managed answer.

Direct answer

Every row in a shared-responsibility matrix is answering the same question at a different layer: who has to act if something at that layer breaks, needs patching, or gets misconfigured? As you move from IaaS to PaaS to SaaS, the line of provider ownership climbs higher, and the one row that never fully crosses over is data: you always decide what your data means and who can see it, even when the provider secures every technical layer underneath it.

The matrix

Networking and virtualization are provider-owned at every tier: you never touch the physical switches, cables, or the hypervisor that slices one machine into many virtual ones. OS, runtime and middleware are where the boundary actually moves, crossing over entirely between IaaS and PaaS. Application crosses over last, between PaaS and SaaS. Data is the one row that stays with the customer across all three models, because ownership of meaning and access is not a technical layer the provider can absorb.

A genuine gray area: virtual networking configuration

"Networking" reads as cleanly provider-owned in the table above, and for the physical network fabric it is. But the moment you get to the virtual networking artifacts sitting on top of that fabric, such as subnets, route tables, security groups and network access control lists (NACLs), responsibility becomes shared in practice even under IaaS: the provider builds the tooling and guarantees isolation between tenants, but you configure which ports are open and which resources can reach the internet. This is exactly the boundary cloud providers describe as "security of the cloud" (the provider's job) versus "security in the cloud" (yours). It is a genuine gray area because a customer who assumes "networking is the provider's row" and never reviews their own security group rules is the single most common real-world path to an accidental public data exposure, even though the table technically marks networking as provider-owned.

A second, smaller gray area worth naming: under PaaS, the provider's automatic OS and runtime upgrades can silently change application behavior (a language runtime minor version bump, for instance). The table says "provider owns runtime," but if that upgrade breaks your application, you are the one who has to notice and fix it, which is a shared failure mode the matrix alone doesn't capture.

Trade-offs and pitfalls

The matrix is a useful starting point, but it is easy to over-trust as a complete answer. Identity and access management (IAM), the system that controls who can do what, cuts across every layer and every model: a customer's own IAM misconfiguration (an over-privileged access key, for example) can undermine security regardless of which service model is in play, and no single row in the table captures that cross-cutting risk. Treat the matrix as the starting framework for a conversation about responsibility, not as a checklist that, once satisfied, guarantees safety.

Direct answer

I would ask the author for four things before I approve: (1) per-arrow reliability annotations (timeout, retry policy, idempotency, and what the caller does on failure); (2) an explicit state machine for a payment (a diagram of every state it can be in and which events move it between them), including an "unknown outcome" state; (3) the reconciliation path that proves money and records agree; (4) trust boundaries and ownership. The gap that worries me most is a synchronous authorisation call that times out with no defined way to find out what actually happened, because that is how you get double charges or lost payments.

What to ask the author to add, grouped by the two halves

Synchronous authorisation (customer waiting)

• Latency budget: if checkout must answer within 5 s, each hop gets a share and timeouts must nest, each outer timeout longer than the inner one plus its retries. For example client 5 s, gateway 4 s, acquirer call 3 s with no retry, so the acquirer times out first and the gateway can still report the outcome. If the acquirer call is retried once, the inner total is 3 + 3 = 6 s, which already exceeds the gateway's 4 s, so the budget must change: for example 1.5 s per attempt with one retry (3 s worst case, ignoring the backoff wait), or no retry at this hop and rely on the status inquiry instead.
• Idempotency (repeat-safe requests): an idempotency key from the client, stored with the result, so a retried request returns the original outcome instead of charging again.
• Behaviour when the acquirer (the bank-side card processor) is slow or down: decline, retry a second processor, or queue? Pick one and draw it.

Asynchronous settlement (money movement later)

• Delivery guarantee on the queue (at-least-once) and consumers that dedupe on payment ID.
• Dead-letter queue (DLQ, where repeatedly failing messages are parked), with an owner and an alert on its depth.
• Reconciliation: a scheduled comparison of our ledger (our record of every payment and amount) against the processor's settlement file (the daily report of what money it actually moved), with an alert on any mismatch. Without it, drift is silent.
• The outbox pattern (write the event and the state change in one database transaction, publish afterwards) so events are never lost between the database and the queue.

Across both

• SLOs (service-level objectives) per path: authorisation availability and latency; settlement completeness within a stated window.
• Trust boundaries and PCI DSS scope (the card-industry security standard): where card data lives, and that only tokens travel elsewhere.
• Ownership per component, runbook links, and RTO/RPO (recovery time objective, how fast we recover, and recovery point objective, how much data we may lose).

The payment state machine I would insist on

stateDiagram-v2
  [*] --> Received
  Received --> Authorized: acquirer approves
  Received --> Declined: acquirer declines
  Received --> Unknown: timeout
  Unknown --> Authorized: status inquiry says approved
  Unknown --> Declined: status inquiry says none
  Authorized --> Settled: settlement file matches
  Authorized --> Mismatch: settlement file differs
  Mismatch --> Settled: manual correction

Worked example: the timeout

The acquirer call has a 3 second timeout and it fires. Without the Unknown state, the service either retries (risking a second authorisation) or fails the payment (risking a hold on the customer's card with no order). With it, the service records Unknown, asks the acquirer "what happened to reference k1?" (a status inquiry), and moves the payment to Authorized or Declined. If the inquiry also fails, a background job keeps asking, and reconciliation is the safety net.

Which gaps worry me most (in order)

1. Ambiguous timeout outcome (above).
2. No idempotency on settlement events, so duplicates double-post the ledger.
3. No reconciliation, so errors are found by customers.
4. Unmarked card-data boundary.

My rule for what blocks approval: a gap blocks if, when it happens, money is wrongly moved or nobody can tell. The first three each meet that test (double charge, double posting, silent drift); the card-data boundary is a compliance blocker. Everything else can be fixed after launch.

Extra check, service mesh (less commonly asked). If the diagram includes a service mesh (sidecar proxies handling service-to-service traffic), it must also show: whether retries are configured in the mesh AND in application code (retries multiply), timeouts per route, circuit-breaking settings, whether traffic is encrypted between services (mutual TLS: both sides prove their identity with certificates), and which calls bypass the mesh.

Extra check, shared cache (less commonly asked). If a shared Redis cache sits in a model-serving path, ask: what happens on a cache miss or Redis outage (fallback and load on the backing store)? Time-to-live (TTL, how long an entry lives) versus staleness of features; a stampede (many simultaneous misses all hitting the backing store together); eviction policy (which entries are dropped when memory is full); who owns the keyspace; and whether a single shared instance is a single point of failure (one component whose loss takes everything down). Example: a 60 s TTL on a feature means predictions may use values up to 60 s old, which the model owner must accept in writing.

Trade-offs and pitfalls

• Asking for everything makes the review unusable; mark which asks block approval (timeout outcome, idempotency, reconciliation) and which are follow-ups.
• Adding a second acquirer improves availability but adds routing complexity and reconciliation for two files.

Strong consistency guarantees that every read reflects the latest completed write, as if there were only ever a single copy of the data. Causal consistency guarantees only that operations one client's actions causally depend on are seen in that order, while operations with no causal relationship to each other can appear in a different order to different readers. Eventual consistency guarantees only that, once writes stop arriving, every replica eventually converges on the same value, with no bound on how stale any individual read can be until then.

For a leaderboard that has to serve reads in under a millisecond, eventual consistency is the only one of the three that can meet the requirement. Strong and causal consistency both need the reader's request to check or coordinate with something beyond its own local copy of the data, and that coordination alone costs more time than a sub-millisecond budget allows, before any application logic runs at all. The design that follows is to keep the write path (the score submission itself) strongly consistent against one source of truth, so a score is never lost or double-counted, while serving every leaderboard read from a separate, eventually consistent, purely local cache.

Strong, causal, and eventual consistency compared

Why a sub-millisecond budget decides this before anything else does

It is worth putting an actual number on what "coordination costs time" means. Light in optical fiber travels at roughly two-thirds its vacuum speed, because of the fiber's refractive index (about 1.5):

A round trip has to cover the distance out and back, so for a full latency budget of one millisecond (0.001 s), the farthest a single round trip can physically reach is:

That is the physical floor, before adding any serialization, queueing, or protocol overhead, and before accounting for the fact that both strong and causal consistency typically need more than one such round trip (a consensus protocol commits a write only once a quorum acknowledges it, and a linearizable read from a follower has to separately confirm it is not stale). Availability zones, the isolated data-center groups a cloud region is built from, are frequently already tens of kilometers apart, and a cross-region hop is hundreds to thousands of kilometers. Any read that must coordinate beyond its own local node is competing for a budget that a single physical hop can already exhaust on its own. A read that never leaves a local, in-memory cache pays none of this cost: its latency is bounded only by local compute and local network, which is routinely and comfortably under a millisecond.

The leaderboard architecture: split the write path from the read path

flowchart LR
    subgraph WRITE["Write path (authoritative, strongly consistent)"]
        P[Player submits score] --> W[Game service]
        W --> DB[(Primary score store: atomic max-update)]
    end
    subgraph READ["Read path (eventual, sub-millisecond)"]
        C[(In-memory sorted leaderboard cache)]
        R[Leaderboard read request] --> C
        C --> UI[Rendered rank list]
    end
    DB -. async fan-out on bounded refresh interval .-> C

The write path is where correctness lives, so it stays strongly consistent: a score submission is an atomic "set my score to X only if X beats what is stored" update against a single primary store, which stops two near-simultaneous submissions (including a cheating client racing its own requests) from corrupting the record. The read path is where the latency requirement lives, so it is eventually consistent: an asynchronous process fans the primary store's changes out to a precomputed, sorted, in-memory structure on a bounded refresh interval, and every leaderboard read is served from that local structure with no coordination at read time. This is the same idea as CQRS (Command Query Responsibility Segregation: separating the path that accepts writes from the path that serves reads so each can be optimized on its own terms), applied specifically along the consistency dimension.

User-facing trade-offs of choosing eventual consistency for reads

• A player who just posted a new high score may not see their own updated rank until the next refresh cycle runs, because their read is served from the same eventually consistent cache as everyone else's. This is the specific cost of applying eventual consistency uniformly: it breaks the one guarantee a user notices immediately, seeing the result of their own action. It is usually patched at the interface layer rather than the consistency layer: show the player's own new score optimistically the instant the write is acknowledged, while the shared leaderboard view around them stays eventually consistent.
• Two players can briefly see different top-ten lists, or different ranks for the same third player, depending on which cache replica or edge location served each of them. For a leaderboard this is usually an acceptable cost, since a rank is inherently a snapshot in time, but it has to be a deliberate call. If the actual requirement were "every player sees an identical ranking at every instant," that pushes back toward strong consistency and a completely different latency conversation.
• Anti-cheat or audit tooling must never read from the eventually consistent cache to decide something consequential, such as banning a player or voiding a score, because that data is stale or momentarily wrong by design. It should read from the strongly consistent primary store, which is exactly why the two paths were split apart in the first place instead of trying to make one path serve both needs.
• The refresh interval feeding the cache needs an explicit staleness bound, a service-level objective (SLO) such as "the cache is never more than a few hundred milliseconds behind the primary store," plus monitoring against that bound, rather than just running "as fast as the background job happens to go." An eventually consistent system with no stated limit on how eventual is not a design decision, it is an outage waiting to be noticed.
• The most common wrong turn here is reaching for strong consistency "to be safe" on the read path, which is precisely where the stated requirement (latency) lives, while the operation that genuinely needs a strong guarantee, the write, gains nothing extra from that choice. Matching the consistency model to the specific operation, instead of applying one model uniformly across the whole feature, is the actual skill this question is testing.

There is no universal answer to build versus buy or cloud versus on-premises; there is a weighted set of criteria that changes the answer depending on which ones dominate for a given decision. At least eight matter in practice: time-to-value, total cost of ownership (TCO, the full multi-year cost including staffing, not just the sticker price), security and compliance, control and customization, operational maturity, vendor lock-in, data gravity, and team skill.

The eight criteria, and why each one can flip the answer

• Time-to-value: buying or using a cloud service is almost always faster to get running, weeks instead of months. This dominates when there is a real market window or a runway constraint.
• Total cost of ownership: building looks cheaper on the invoice and more expensive in engineering headcount over time; buying looks more expensive on the invoice and often cheaper once the ongoing staffing cost of running the equivalent yourself is counted. The comparison only means something over a matched multi-year horizon.
• Security and compliance: intuitively this pushes toward "keep it in-house and controlled," but a mature vendor that already holds the relevant certifications (SOC 2, an independent audit of a vendor's security controls; ISO 27001, an international standard for managing information security; and a signed data processing agreement) can often clear a compliance bar faster than building the equivalent controls from zero.
• Control and customization: a useful gut check here is separating "core" (what your customers pay you for, what actually differentiates you) from "context" (necessary, but not differentiating). Building wins when the thing in question is genuinely core; buying wins when it's context, even if it feels important.
• Operational maturity: does the team already run this class of system reliably around the clock? If not, buying a managed offering is effectively buying someone else's on-call maturity along with the product.
• Vendor lock-in: buying increases switching cost. Mitigate it with an abstraction layer and data portability at the edges, rather than either ignoring the risk or over-engineering for a hypothetical exit at the cost of speed.
• Data gravity: where your data already lives constrains where compute should live. Moving large volumes of data to follow a new platform can dominate both the cost and the timeline of a decision more than any other single factor, and it is often left out of a first-pass scoring model entirely.
• Team skill and learning curve: building or self-managing something the team has never operated adds a hidden ramp-up cost and a reliability risk in the first six to twelve months that a clean scoring model tends to omit.

Worked example

For a customer-notification service (not a differentiator for most businesses, so it leans toward "context"), weight the eight criteria to sum to 1.0: time-to-value 0.20, TCO 0.25, security/compliance 0.15, control/customization 0.10, operational maturity 0.15, lock-in 0.05, data gravity 0.05, team skill fit 0.05. Score a managed SaaS option and an in-house build on a 1-5 scale (5 favors that option):

Weighted sum for Buy: (5x0.20)+(3x0.25)+(4x0.15)+(2x0.10)+(5x0.15)+(2x0.05)+(4x0.05)+(5x0.05) = 1.00+0.75+0.60+0.20+0.75+0.10+0.20+0.25 = 3.85.
Weighted sum for Build: (2x0.20)+(3x0.25)+(3x0.15)+(5x0.10)+(2x0.15)+(5x0.05)+(4x0.05)+(2x0.05) = 0.40+0.75+0.45+0.50+0.30+0.25+0.20+0.10 = 2.95.

Buy wins here, 3.85 to 2.95, mainly on time-to-value, operational maturity, and team skill fit, exactly the profile of a small or SMB-sized team without existing notification infrastructure experience. The same criteria, reweighted for a system that is genuinely core to the product (say, control/customization weighted at 0.30 instead of 0.10 because deep customization is the differentiator, and an equivalent open-source project exists that could be adopted and extended instead of bought as a black box), would tip the same math toward building or adopting and modifying open source rather than buying a closed managed product.

Trade-offs and pitfalls

• A scoring model can be reverse-engineered: weights chosen after seeing which answer people already wanted look objective but aren't. The discipline is to defend the weights before anyone scores the options, the same rule that keeps a proof-of-concept honest.
• This framework produces a decision for a point in time. Team skill, vendor pricing, and data volume all change, so the decision should carry an explicit revisit trigger (a renewal date, a headcount change) rather than being treated as permanent.
• The most common mistake at the "buy" end is comparing sticker prices only and skipping TCO's staffing component; the most common mistake at the "build" end is skipping the team-skill and data-gravity criteria and discovering the hidden ramp-up cost only after committing.