Cloud and Managed Database Services Questions
Running databases as managed cloud services rather than self-hosting them: choosing and provisioning managed relational and NoSQL offerings (for example Amazon RDS/Aurora/DynamoDB, Azure SQL Database/Cosmos DB, Google Cloud SQL/Spanner), sizing instances and storage, and designing an architecture (read replicas, connection proxies and pooling for managed or serverless compute) to meet a latency or availability target. It covers comparing provisioned versus serverless/autoscaling pricing and operational models, forecasting capacity and cost, and migrating a database onto or between managed deployments: moving a self-hosted database to a managed service, switching a workload from provisioned to serverless (or back), changing a single-AZ deployment to multi-AZ, and the questions to ask before adopting a new managed vendor. The core question is the split of responsibility and cost between the cloud provider and the engineering team: what the provider takes on (patching, infrastructure-level HA, automated backups) versus what remains the team's job (configuration, capacity and cost tuning, security posture like encryption and key rotation). It does not cover backup and restore mechanics, RPO/RTO planning, replication or sharding internals, or SQL-level query diagnostics.
Design an AWS RDS architecture for a read-heavy, customer-facing application that needs high availability across availability zones and horizontal read scaling. Describe your primary and read-replica placement, automated failover behavior, the connection-routing strategy for read/write separation, how you would handle replica lag, and how you would test failover.
Sample Answer
This architecture needs two structurally different mechanisms working together, not one: a Multi-AZ (availability zone) standby for automated failover, and a separate fleet of read replicas for horizontal read scaling. Conflating them is the most common design mistake here, since a Multi-AZ standby never serves read traffic at all; that is what read replicas are for.
flowchart TB
App[Application tier] --> Router[Read/write router, application- or proxy-side]
Router -->|writes| PrimaryEP[Primary instance endpoint, DNS CNAME]
Router -->|reads, chosen by app/proxy - RDS gives no built-in shared reader endpoint| Replica1[(Read replica, AZ-b)]
Router -->|reads, chosen by app/proxy - RDS gives no built-in shared reader endpoint| Replica2[(Read replica, AZ-c)]
PrimaryEP --> Primary[(Primary, AZ-a)]
Primary -.synchronous, HA only.-> Standby[(Multi-AZ standby, AZ-b)]
Primary -.asynchronous.-> Replica1
Primary -.asynchronous.-> Replica2
Primary and standby placement
The primary runs in one AZ; the Multi-AZ standby runs in a different AZ within the same region, kept current by synchronous replication (the primary does not acknowledge a write until the standby has it too). This standby exists purely for failover, it serves no read traffic under normal operation, which is the key fact that separates it from a read replica. The read replicas are placed across additional AZs (they can share an AZ with the standby; there is no requirement to keep them disjoint), sized in count to the actual read throughput need, addressed below.
Automated failover behavior
On primary failure or an AZ-level impairment, RDS promotes the standby and repoints the DB instance's single existing endpoint (its DNS CNAME, canonical name record) at the newly promoted instance rather than requiring any connection-string change. A classic RDS DB instance has exactly one endpoint total, not a separately named "writer endpoint" the way an Aurora cluster does, so failover here is a DNS re-point of that one address, not a switch between two differently-named endpoints. AWS states this happens "as quickly as 60 seconds" for a single-standby Multi-AZ deployment, with most failovers completing within one to two minutes. The part this leaves in the application's hands: a client with an existing connection to the now-dead primary, and a driver or pool that caches the resolved IP rather than re-resolving DNS promptly, can keep retrying a dead address well past the point RDS considers the failover complete. Short connection timeouts and prompt DNS re-resolution in the connection pool (the application's cache of reusable open database connections) are what actually determine how much of that 60-to-120-second window the application experiences as a real outage.
Connection-routing strategy for read/write separation
Two implementation options, and a fact that has to be stated up front: unlike Aurora, plain RDS does not give you a single, AWS-managed, load-balanced endpoint spanning multiple read replicas. Each RDS read replica is provisioned with, and reachable only through, its own individual endpoint, exactly like any other standalone RDS DB instance. Getting a single logical "read" address, if the application wants one, means building it, either in application code or via a proxy in front of the replica fleet; AWS does not hand it to you for free the way it does for Aurora's cluster reader endpoint.
- App-layer routing: hold the primary's own endpoint for writes, and the individual read replica endpoints for reads, and route each query explicitly in application code (round-robin, random choice, or a fixed per-service assignment across the known replica endpoints). Simplest to reason about, and its failure mode (a write accidentally sent to a read replica) is a loud, immediate error, since replicas reject writes, not a silent correctness bug.
- Proxy-layer routing: a pooler or router (for example a network load balancer or HAProxy in front of the replica fleet, or ProxySQL) that inspects statement type and routes automatically, which also gives the application a single logical address for reads instead of managing the replica endpoint list itself, at the cost of another moving part to operate.
For most teams, start with explicit app-layer routing; it is simpler, its failure mode is safe (a hard error, not silent data drift), and a routing proxy is worth adding later specifically once the number of call sites makes manual discipline unreliable, not as a default from day one.
Handling replica lag
Route only traffic that tolerates some staleness to the reader endpoint, browsing and search-style reads. Any read that must reflect the user's own just-completed write (an order confirmation immediately after checkout is the classic case) should either read from the writer directly for that specific call, or carry a short-lived "read your own write" hint that pins that one request to the writer rather than trusting the load-balanced reader endpoint. Monitor the replica lag metric per replica and treat a replica whose lag exceeds a defined threshold as degraded for routing purposes, since a healthy-but-lagged replica can otherwise keep receiving its normal share of read traffic through the load-balanced endpoint even while serving noticeably stale data.
Testing failover, not just designing for it
Design intent is not validation. Rehearse the actual failover, on a non-production copy at minimum, ideally occasionally against production during a planned low-traffic window once confidence is established:
aws rds reboot-db-instance --db-instance-identifier prod-webapp-db --force-failover
Measure the application's own observed error rate and recovery time during that test, not just the RDS-reported failover duration; those two numbers are not the same thing. A failover that completes on RDS's side in under a minute can still produce several minutes of application-visible errors if the connection pool does not detect the broken connection and re-resolve DNS promptly, which is exactly the gap the "automated failover behavior" section above flags as the application's responsibility, and the only way to know whether that gap is actually closed is to force a real failover and watch what the application does.
Worked example: sizing the read-replica count
peak_read_qps = 40_000
per_replica_sustained_qps = 8_000 # measured comfortable sustained capacity per replica
replicas_for_load = -(-peak_read_qps // per_replica_sustained_qps) # ceiling division
replicas_with_n_plus_1 = replicas_for_load + 1 # tolerate one replica down for maintenance or failure
print(f"Replicas needed to cover peak load: {replicas_for_load}")
print(f"Replicas with N+1 redundancy: {replicas_with_n_plus_1}")
print(f"Per-replica load with one of {replicas_with_n_plus_1} down: "
f"{peak_read_qps/(replicas_with_n_plus_1-1):,.0f} QPS")
Replicas needed to cover peak load: 5
Replicas with N+1 redundancy: 6
Per-replica load with one of 6 down: 8,000 QPS
Five replicas exactly cover the stated peak with zero slack, so losing even one to maintenance or a fault pushes the remaining four over their sustained capacity; provisioning a sixth (N+1) means the fleet still exactly meets sustained capacity, not exceeds it, with one replica out, which is the actual bar "can survive a single replica failure without degrading" sets.
Trade-offs and pitfalls
- Treating the Multi-AZ standby as if it also offloads read traffic is a common misunderstanding that leads teams to under-provision real read replicas; the standby's only job is failover.
- Aurora specifically allows a read replica to also serve as a Multi-AZ failover target if placed in a high-priority promotion tier (tiers run 0 to 15, and tier 0, the lowest number, is the highest priority for promotion, not tier 15), which blurs this line further in Aurora's favor (one fewer idle resource), and Aurora also provides a single load-balanced cluster reader endpoint spanning every Aurora Replica automatically. Neither behavior is something plain RDS read replicas provide: plain RDS replicas are each their own standalone endpoint, with no automatic promotion-to-failover-target and no built-in load balancing across them, which is exactly why a team that wants that behavior for free, rather than building it, often picks Aurora specifically for this reason.
- Skipping the forced-failover rehearsal is the single biggest gap between "designed for high availability" and "actually highly available": the DNS-flip mechanism and the AWS-stated failover time are necessary but not sufficient, since the application's own reconnect behavior is what determines real user-facing downtime.
- Load-balancing reads purely on the reader endpoint without a lag-aware exclusion means a badly lagged but technically healthy replica keeps taking its normal share of traffic, quietly serving stale reads to whichever users happen to be routed to it.
A vendor offers a managed distributed SQL database that promises automatic sharding and rebalancing. List 6 operational questions you would ask before adopting it for a latency-sensitive transactional service.
Sample Answer
Direct answer
Six questions cover the ground a "handles sharding and rebalancing automatically" pitch conveniently skips: what latency the vendor will actually guarantee under your transaction shape, whether rebalancing degrades live traffic or pauses it, what the replication and consistency model implies for data loss and recovery time, how visible internal topology changes are to your own monitoring, how the system behaves when it cannot reach a quorum (a majority of replicas agreeing before a write counts as durable), and how hard it would be to leave. Ask all six framed against "latency-sensitive transactional," not as a generic vendor-diligence checklist: an answer that would satisfy an analytics workload can still be a bad fit here.
The six questions
| # | Question | Why it matters for a latency-sensitive transactional workload | Red flag in the vendor's answer |
|---|---|---|---|
| 1. Latency guarantee and measurement basis | "What 99th-percentile (P99) commit latency do you guarantee for a single-row transactional write, under a documented workload similar to mine, and is that backed by a service-level agreement (SLA) with a financial remedy?" | Distributed SQL databases add a consensus round trip (a network exchange where a quorum of replicas must acknowledge a write) on every commit. The number that matters is the tail latency of a small, single-row transaction, not an average and not a number measured on a large analytical batch. | Vendor quotes only mean or median latency, or a benchmark run on multi-row batch writes instead of the small single-row commits typical of a transactional service. |
| 2. Rebalancing impact on live traffic | "When the cluster rebalances automatically (a node joins, a hot shard splits, a failed replica is repaired), does the affected data stay fully read and write available, or does the system pause or throttle traffic on the piece being moved? Can rebalancing be rate-limited or scheduled away from peak hours?" | Rebalancing moves a shard (a contiguous chunk of the table's rows, grouped by key range) from one node to another by streaming a full copy of it across the network. Left unthrottled, that transfer competes with production traffic for network and CPU capacity right when the cluster's internal state is least predictable. This is not hypothetical: even mature distributed SQL engines have shipped rebalancing paths where an ordinary configuration change measurably degraded foreground query latency before the vendor patched it. | Vendor cannot describe a concrete rate-limiting or scheduling mechanism, or only offers "it is usually fine in practice." |
| 3. Replication, consistency, and disaster-recovery implications | "What is the default replication factor and quorum size per shard, what consistency guarantee does a client get by default (strict ordering across all replicas versus eventual convergence), and what recovery point objective (RPO, how much recent data could be lost) and recovery time objective (RTO, how long recovery takes) does that imply if a region fails mid-transaction?" | "Automatic sharding" says nothing about whether each shard's replicas live in one region (fast, but a regional outage can lose recent writes) or are kept in sync across regions by a global consensus protocol (safer, but adds real latency to every commit). | Vendor cannot state RPO or RTO as numbers, or uses "highly available" and "zero data loss" interchangeably. |
| 4. Observability into internal topology changes | "What real-time visibility exists into shard splits, leader elections (which replica currently owns the right to accept writes for a shard), and in-progress rebalancing, and can a latency spike in my own application performance monitoring (APM) tooling be correlated back to a specific internal event?" | If the database's internal reshuffling is invisible, a 2 a.m. latency spike looks identical to an application bug, and an on-call engineer burns an hour ruling out their own code before learning the database moved a shard underneath them. | Vendor offers only an aggregate cluster-health dashboard with no exportable, timestamped events for splits, elections, or rebalances. |
| 5. Failure-mode behavior | "During a network partition or a quorum loss on a shard, does the system fail closed (refuse writes, to stay consistent) or fail open (keep accepting writes on a stale replica)? Does failover for a lost leader happen automatically, or does it require a manual runbook step?" | For a transactional service, a database that fails open during a partition can accept writes that conflict or silently overwrite each other, surfacing as corrupted business state days later. That is worse than a short, honest availability gap. | Vendor cannot describe the failure mode precisely, or the answer implies writes are accepted during a partition "to keep availability high." |
| 6. Data portability and exit strategy | "If the contract ends, is export a native logical dump, a wire-protocol-compatible export another engine can read, or a proprietary snapshot only the vendor's own restore tooling understands? Who controls the encryption keys, and how long would extracting the full current data volume take at the vendor's stated export throughput?" | Automatic sharding and rebalancing are exactly the features that make a vendor's on-disk storage format the least portable. Verifying the exit path before signing is far cheaper than discovering it during a forced migration. | Vendor's export path is undocumented, or only supports restoring into their own product. |
Worked example
Take a payments authorization service: 3,000 transactions per second, a 40 ms P99 commit latency SLA of its own, and a requirement to survive a full region outage, so the team is evaluating a three-region deployment spanning, say, New York and San Francisco.
Question 1 asks the vendor for a P99 commit latency guarantee. Before even hearing their number, it is worth computing the physics floor a synchronous cross-region quorum write cannot beat, because no amount of engineering can get a commit acknowledgment back faster than light travels through fiber:
# Physics floor on cross-region consensus commit latency: a sanity check on any
# vendor SLA number, not a benchmark. All inputs are stated explicitly below.
great_circle_km = 4130 # approx. New York <-> San Francisco great-circle distance
routing_factor = 1.5 # real fiber paths run longer than the great circle
fiber_path_km = great_circle_km * routing_factor
speed_of_light_vacuum_km_s = 299_792
refractive_index_fiber = 1.5 # typical single-mode fiber
speed_in_fiber_km_s = speed_of_light_vacuum_km_s / refractive_index_fiber
one_way_ms = (fiber_path_km / speed_in_fiber_km_s) * 1000
round_trip_ms = one_way_ms * 2
print(f"fiber path length: {great_circle_km} km x {routing_factor} = {fiber_path_km:.0f} km")
print(f"speed in fiber: {speed_of_light_vacuum_km_s} / {refractive_index_fiber} = {speed_in_fiber_km_s:,.0f} km/s")
print(f"one-way propagation time: {fiber_path_km:.0f} km / {speed_in_fiber_km_s:,.0f} km/s = {one_way_ms:.3f} ms")
print(f"round trip (quorum ack floor): {one_way_ms:.3f} ms x 2 = {round_trip_ms:.2f} ms")
Output:
fiber path length: 4130 km x 1.5 = 6195 km
speed in fiber: 299792 / 1.5 = 199,861 km/s
one-way propagation time: 6195 km / 199,861 km/s = 30.996 ms
round trip (quorum ack floor): 30.996 ms x 2 = 61.99 ms
A round trip between New York and San Francisco alone costs about 62 ms, before any disk write, queueing, or server-side processing. If the vendor's answer to question 1 is "P99 commit latency under 40 ms" and question 3 reveals that every commit needs a quorum acknowledgment from a replica in the other region, those two answers contradict each other: the deployment cannot meet its own stated SLA. The honest fix is either to keep the write quorum inside one metro region (propagation delay measured in fractions of a millisecond) and replicate to the far region asynchronously for disaster recovery only, accepting a small, named RPO, or to relax the latency SLA. Either is a legitimate design choice; a vendor who does not surface this tension when questions 1 and 3 are asked together has not actually answered the question.
Trade-offs and pitfalls
The most common wrong turn is accepting a vendor's self-reported questionnaire as verification. A "yes" to all six questions from a sales engineer is not evidence: insist on a proof-of-concept load test that replays the actual read/write mix and transaction size against the vendor's cluster, sized to the real data volume, while watching P99 latency through a real rebalance (add a node, kill a replica, and observe). "Distributed" and "managed" are not synonyms for "safe for latency-critical transactions": consensus overhead, rebalancing behavior, and failure-mode defaults vary enormously between vendors, and that is exactly the corner of the evaluation a generic due-diligence checklist glosses over.
Describe how you would deploy a managed relational database for a production web application. Cover service selection (RDS/Aurora/Cloud SQL/Managed MySQL), sizing (instance class, storage type), read scaling, encryption, maintenance windows, and routine maintenance practices.
Sample Answer
Deploying a managed relational database for production is five deliberate decisions, not one "create instance" click: which service, what size, whether you need read scaling yet, encryption configured before first write, and a maintenance window you chose rather than accepted by default.
Service selection
- Amazon RDS: the broadest engine choice (MySQL, PostgreSQL, MariaDB, SQL Server, Oracle, Db2), engine-native replication, and the most flexible instance sizing including small burstable classes. The default choice for most web applications unless you have a specific reason to want more.
- Amazon Aurora: MySQL- and PostgreSQL-compatible, with storage-layer replication instead of engine-level replication, which gives materially faster failover (automatically switching to a healthy standby when the primary fails) and a higher write-throughput (the volume of write operations the database can sustain per second) ceiling on the largest instance classes. Worth the higher baseline cost when you expect to outgrow a single well-tuned RDS instance, or when failover speed matters more than for a typical web app.
- Cloud SQL: Google Cloud's equivalent to RDS for MySQL, PostgreSQL, and SQL Server, same category of decision as RDS on AWS.
For a typical production web application starting out, RDS or Cloud SQL on the engine your team already knows is the right default; reach for Aurora specifically when you can point to a concrete scaling reason, not by default.
Sizing: instance class and storage type
Pick the instance class from measured or realistically estimated CPU and memory need with headroom, not a round number picked for comfort. Storage type matters as much as instance class:
- gp3 (general purpose SSD): the right default for most online transaction processing (OLTP) workloads. It decouples provisioned IOPS (input/output operations per second: the count of individual read or write operations the volume can handle each second, distinct from throughput's raw bytes-per-second measure) and throughput from allocated storage size, so you are not forced to over-allocate storage just to get more IOPS.
- io2 / io2 Block Express: for latency-sensitive, very high-IOPS workloads where gp3's ceiling is not enough; costs more per provisioned IOPS.
- Enable storage autoscaling so unplanned data growth raises allocated storage automatically instead of the instance running out of space and rejecting writes.
Read scaling
Add a read replica only when there is a specific, measured reason: reporting or analytics queries competing with transactional traffic for CPU and I/O, or read throughput that a single writer genuinely cannot serve even with caching in front of it. A read replica is a full additional instance (real cost, real operational surface via replication lag, the delay before a write on the primary becomes visible on the replica), not a free performance knob, so route only the traffic that needs it, typically through RDS Proxy or explicit read/write splitting in the application, rather than defaulting every read query to a replica.
Encryption
Enable storage encryption at creation time: RDS cannot add encryption to an existing unencrypted instance in place, only by snapshotting it, copying the snapshot with encryption enabled, and restoring from that copy, so getting this right on day one avoids a disruptive migration later. The AWS-managed key (which AWS automatically rotates every year) is sufficient for most applications; use a customer-managed KMS (Key Management Service) key only when you need explicit control over key policy or rotation cadence, for example a compliance requirement to control who can disable access to the key. Separately, enforce TLS (transport layer security) on connections at the engine level, rds.force_ssl for PostgreSQL or require_secure_transport for MySQL, since encryption at rest alone does not protect credentials and query data in flight between the application and the database.
Maintenance windows and routine maintenance
If you do not set a maintenance window explicitly, RDS assigns a random 30-minute window in your region, which may land during business hours. Choose one deliberately, during genuinely low-traffic hours for your actual user base, and prefer applying non-urgent changes during that window rather than immediately, so patches land predictably instead of mid-peak-traffic. Keep automatic minor-version upgrades on for security patches, but review major version upgrades manually since those can carry breaking changes to the engine's behavior. As routine practice: review the instance's "pending maintenance actions" on a regular cadence rather than being surprised by them, audit the parameter group for drift from intended settings after any manual tuning, and revisit instance rightsizing on a schedule (for example quarterly) instead of only reacting once performance has already degraded.
Worked example: a concrete creation command
aws rds create-db-instance \
--db-instance-identifier prod-webapp-db \
--engine postgres --engine-version 16.4 \
--db-instance-class db.r6g.xlarge \
--allocated-storage 200 --storage-type gp3 --iops 6000 --storage-throughput 250 \
--max-allocated-storage 500 \
--storage-encrypted --kms-key-id alias/prod-db-key \
--preferred-maintenance-window "sun:05:00-sun:06:00" \
--auto-minor-version-upgrade
This encodes the sizing decision (an r6g.xlarge, memory-optimized Graviton instance class, a common default for a moderate production web app), gp3 storage with explicit IOPS and throughput rather than whatever the default happens to be, storage autoscaling up to 500 GiB so growth does not hit a hard wall, encryption specified at creation with a named key, a chosen (not default) maintenance window, and automatic minor-version patching. Multi-AZ and backup retention are deliberately left off this example: they are availability and disaster-recovery decisions with their own trade-offs, not part of the base provisioning choice this question is about.
Trade-offs and pitfalls
- Defaulting to the largest instance class "to be safe" wastes money every hour it runs oversized; defaulting to the smallest "to save money" invites a production incident the first time traffic grows. Size from data, and revisit it, rather than picking once and forgetting.
- Adding a read replica before you have a measured reason adds real operational surface (monitoring replica lag, routing logic) for no benefit, and it is easy to add one reflexively because "that's what scaling a database looks like."
- Forgetting to set storage encryption before the first write is the single most expensive mistake in this list to fix later, since fixing it means a snapshot-copy-restore cycle on a live database rather than a config change.
- Accepting the default maintenance window is a common, easily avoided source of an unplanned mid-day blip; it costs nothing to set deliberately at creation time.
Explain read replicas in managed database offerings: what they are, how they work in services like Amazon RDS/Aurora or Cloud SQL, what operational problems they solve, and what limitations or risks you'd want to watch for when relying on them.
Sample Answer
A read replica is an asynchronous, read-only copy of a primary database. The primary streams its stream of committed changes (the binary log in MySQL, the write-ahead log, or WAL, in PostgreSQL) to one or more replicas, which replay those changes to stay current. The replica accepts read queries but never writes, and it is always at least slightly behind the primary because replication is asynchronous: the primary does not wait for the replica before acknowledging a write.
How it works in managed offerings
- Amazon RDS (MySQL/PostgreSQL/MariaDB): uses the engine's own native replication mechanism, binlog-based for MySQL/MariaDB, WAL streaming for PostgreSQL, to ship changes to up to several read replicas per primary. Each replica is a fully separate instance with its own storage and its own endpoint.
- Amazon Aurora: replicas share the same underlying cluster storage volume as the writer instead of copying data over the network, so replica lag is typically much lower than RDS's engine-native replication under normal load, low enough that AWS markets Aurora's replicas as "low-latency." Aurora also lets a replica be promoted automatically as part of Multi-AZ failover (Multi-AZ means the provider keeps a synchronized standby copy of the database in a second availability zone; failover is automatically switching traffic to that standby, or here, to a promoted replica, when the primary fails), which RDS's plain read replicas cannot do.
- Google Cloud SQL: replicates asynchronously per-transaction from the primary to one or more read replicas, including optional cross-region replicas, using the same engine-native mechanisms under the hood.
In all three, the replica is provisioned and monitored by the provider (you don't manage the replication process by hand), but the replication itself is still asynchronous and lag is not bounded to zero.
What problems read replicas solve
- Read scaling: route SELECT-heavy traffic to one or more replicas so it does not compete with writes for CPU and I/O on the primary.
- Workload isolation: point a specific class of query, for example an analytics dashboard or a nightly reporting job, at a dedicated replica so a slow or expensive query can't degrade the primary's write latency.
- Reduced primary load and latency: fewer connections and less query volume on the primary generally means more headroom and lower p95/p99 latency (the response time for the slowest 5% and slowest 1% of requests, a way of measuring worst-case performance rather than the average) for the writes and reads that must hit the primary.
- A promotable target: an RDS read replica (not a Multi-AZ standby) can be manually promoted to a standalone writable instance, which is useful for cross-region disaster recovery or for splitting off a subset of traffic to its own database.
Limitations and risks to watch
- Replication lag is not bounded. Under normal load it can be small enough to ignore; under a write burst, a long-running DDL (data definition language, for example an ALTER TABLE) statement, or a large batch job on the primary, it can grow to seconds or minutes, and the replica has no way to signal "I'm behind" to a client unless you build that check yourself.
- Eventual consistency, not read-your-writes. A client that writes to the primary and immediately reads from a replica can see stale data, or even see the write "disappear" if it lands on a replica that hasn't caught up yet. This is the single most common production bug with read replicas: a user submits a form, the confirmation page reads from a replica, and the user's own just-written data appears missing.
- No cross-replica consistency guarantee. Two replicas of the same primary can be at different points in the replication stream at the same instant, so two requests hitting different replicas can see different, mutually inconsistent snapshots.
- Promotion is a manual, disruptive operation, not an automatic failover mechanism. Promoting a read replica breaks its replication link permanently, and every other reader and writer in the application needs to be repointed. This is a materially different (and slower) recovery path than a Multi-AZ standby failover.
- Real added cost. A read replica is a full running instance, not a lightweight cache, so it roughly multiplies your compute bill by the number of replicas you run.
Worked example: how a write burst turns into visible lag
Assume a primary normally emits 12 MB/s of write-ahead log under steady state, a replica can apply changes at 20 MB/s, and a 10-minute batch job pushes the primary's write rate to 30 MB/s.
backlog growth rate during burst=30−20=10 MB/sOver the 600-second burst that backlog reaches 10×600=6,000 MB of unreplayed write-ahead log sitting on the replica the instant the burst ends.
It is tempting to convert that 6,000 MB straight into a "seconds behind" figure by dividing by the replica's apply rate (6,000 / 20 = 300 seconds), but that is not the number a real replication-lag metric would show at that instant, and it is a common mistake when reasoning about replication under load. What tools like MySQL's seconds_behind_master or Aurora/RDS's ReplicaLag CloudWatch metric actually report is how old the transaction the replica is currently replaying is, which you get by comparing cumulative bytes applied against cumulative bytes produced, not backlog divided by capacity. At the moment the burst ends (600 seconds in), the replica has applied 20×600=12,000 MB while the primary has produced 30×600=18,000 MB since the burst started. The 12,000 MB the replica has applied so far is data the primary had already produced after just 12,000/30=400 seconds of the burst, so the replica is 600−400=200 seconds, about 3 minutes 20 seconds, behind at the instant the burst ends, not 300 seconds.
Lag keeps growing for a while even after the burst is over, because the replica is still capacity-constrained at 20 MB/s working through the backlog while the primary keeps producing new data (now at the lower 12 MB/s rate) that pushes "how old is what I'm applying right now" further out. Lag peaks about 5 minutes after the burst ends, at 300 seconds (5 minutes) behind, then shrinks back toward zero as the replica keeps applying at 20 MB/s against a primary now producing only 12 MB/s, a net drain rate of 20−12=8 MB/s. The 6,000 MB backlog itself (still sitting there right when the burst ends) fully drains 6,000/8=750 seconds, 12.5 minutes, after the burst ends, which is also when lag finally returns to zero.
Total elapsed time with a nonzero backlog and elevated lag: the 600-second burst plus the 750-second drain, 1,350 seconds, 22.5 minutes, not the 17.5 minutes you get from naively adding a 300-second instant-lag figure to the drain time. Lag grows for the first 900 seconds (15 minutes: the full 10-minute burst plus 5 more minutes while the primary is still outpacing the still-catching-up replica), peaks at 5 minutes behind, then shrinks over the remaining 450 seconds (7.5 minutes). For that whole 22.5-minute window, anything reading from that replica was looking at data that was seconds to minutes old. This is exactly the kind of window where a "read after write" bug shows up, and it is why replica lag needs to be an explicit, monitored metric (RDS and Aurora both expose it as a CloudWatch metric) rather than an assumption, or worse, a hand calculation done wrong under incident pressure.
Think of a managed database service you have personally deployed and operated in production (for example RDS, Aurora, Cloud SQL, DynamoDB, or Cosmos DB). Walk through how you approached capacity planning and your scaling strategy for it, and share one concrete outcome or lesson learned from running it in production.
Sample Answer
This question is scored on specificity, not on which service gets named. A strong answer names the actual signal that triggered a capacity decision (a real metric and threshold, not "we noticed it was slow"), the actual scaling mechanism chosen and why that one over the alternatives, and one concrete lesson that changed how the system was operated afterward, a real before-and-after in process, not a restated best practice.
What a strong answer actually contains
- A specific capacity signal, not a vague symptom. "We watched CPU" is not a signal; "the burst credit balance on a burstable instance class was heading toward zero every afternoon" is, because it names the actual metric, the actual pattern, and why that pattern meant trouble was coming rather than already fine. (A burstable instance class, for example AWS's T-family, earns credits while it runs below its baseline CPU level and spends those credits to run faster than that baseline when needed; once the credit balance reaches zero, performance drops back down to the lower baseline, which is the actual mechanism that turns a shrinking balance into a real, user-facing slowdown.)
- The scaling mechanism chosen, and why it beat the obvious alternative. Interviewers are listening for the reasoning that ruled out the simpler option, not just the option that was picked.
- One concrete lesson framed as a process change, something that is genuinely different about how the system, or the team, operates now versus before the incident, not a generic takeaway that would apply to any database anywhere.
What undermines this answer
A memorized capacity-planning checklist with no specific metric, no specific trigger, and no real outcome attached reads as rehearsed rather than experienced, which is precisely what this question is designed to surface. Naming an impressive-sounding managed service is not the point; the reasoning and the actual outcome are.
Illustrative example of the shape a strong answer takes
A production RDS (Amazon Relational Database Service) MySQL instance on a burstable instance class (db.t3) ran fine most of the day, but a daily afternoon analytics batch job steadily depleted its CPU credit balance, and by the third month, users started reporting slow page loads that correlated with when that job ran. The real signal was the CPUCreditBalance CloudWatch metric (the metric AWS documents specifically for T-family instance classes, distinct from BurstBalance, which tracks gp2 storage I/O credits and applies regardless of instance class) heading toward zero every afternoon, not flatlined at zero, which meant the instance was surviving that day by borrowing against a shrinking buffer, a materially different situation than simply being fine. The first instinct was to move the batch job to overnight, but that job fed a report a stakeholder needed by mid-afternoon, so the timing was not actually negotiable. The real fix was migrating that instance off the burstable instance class onto a non-burstable class sized for the batch job's sustained CPU demand, not just its average load, since a T-family instance's baseline CPU only refills credits while running below that baseline, and a daily job that regularly pushes CPU above baseline for long enough will eventually always run the balance to zero no matter how the alarm is tuned. The lesson that changed how the team operated afterward: CPU-credit alarms got set at 50% remaining, not a default-feeling 10%, because by the time the balance actually hit 10% there were maybe two hours of runway left before it became a customer-facing incident, nowhere near enough lead time to safely plan and execute an instance-class change on a production database. That alarm-threshold change, not the specific database engine involved, is the actual takeaway a strong answer should be able to name concretely.
Trade-offs and pitfalls
- A "lesson learned" that is really a restated best practice ("always monitor your database") without a specific behavior change attached does not clear the bar this question sets; the question explicitly asks what changed, not what is generally good practice.
- Candidates sometimes reach for the most technically impressive incident they can recall rather than the one they can describe with the most real detail; a smaller, well-specified incident beats a large, vaguely-remembered one here.
- The strongest answers make clear which part was a genuine surprise at the time (the credit balance pattern, in the example above) versus which part was a deliberate, reasoned trade-off (ruling out simply rescheduling the job); collapsing those two into one flat narrative loses the signal an interviewer is actually listening for.
Unlock Full Question Bank
Get access to all 14 Cloud and Managed Database Services interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.