Cloud and Managed Database Services Questions
Running databases as managed cloud services rather than self-hosting them: choosing and provisioning managed relational and NoSQL offerings (for example Amazon RDS/Aurora/DynamoDB, Azure SQL Database/Cosmos DB, Google Cloud SQL/Spanner), sizing instances and storage, and designing an architecture (read replicas, connection proxies and pooling for managed or serverless compute) to meet a latency or availability target. It covers comparing provisioned versus serverless/autoscaling pricing and operational models, forecasting capacity and cost, and migrating a database onto or between managed deployments: moving a self-hosted database to a managed service, switching a workload from provisioned to serverless (or back), changing a single-AZ deployment to multi-AZ, and the questions to ask before adopting a new managed vendor. The core question is the split of responsibility and cost between the cloud provider and the engineering team: what the provider takes on (patching, infrastructure-level HA, automated backups) versus what remains the team's job (configuration, capacity and cost tuning, security posture like encryption and key rotation). It does not cover backup and restore mechanics, RPO/RTO planning, replication or sharding internals, or SQL-level query diagnostics.
Design an AWS RDS architecture for a read-heavy, customer-facing application that needs high availability across availability zones and horizontal read scaling. Describe your primary and read-replica placement, automated failover behavior, the connection-routing strategy for read/write separation, how you would handle replica lag, and how you would test failover.
Sample Answer
This architecture needs two structurally different mechanisms working together, not one: a Multi-AZ (availability zone) standby for automated failover, and a separate fleet of read replicas for horizontal read scaling. Conflating them is the most common design mistake here, since a Multi-AZ standby never serves read traffic at all; that is what read replicas are for.
flowchart TB
App[Application tier] --> Router[Read/write router, application- or proxy-side]
Router -->|writes| PrimaryEP[Primary instance endpoint, DNS CNAME]
Router -->|reads, chosen by app/proxy - RDS gives no built-in shared reader endpoint| Replica1[(Read replica, AZ-b)]
Router -->|reads, chosen by app/proxy - RDS gives no built-in shared reader endpoint| Replica2[(Read replica, AZ-c)]
PrimaryEP --> Primary[(Primary, AZ-a)]
Primary -.synchronous, HA only.-> Standby[(Multi-AZ standby, AZ-b)]
Primary -.asynchronous.-> Replica1
Primary -.asynchronous.-> Replica2
Primary and standby placement
The primary runs in one AZ; the Multi-AZ standby runs in a different AZ within the same region, kept current by synchronous replication (the primary does not acknowledge a write until the standby has it too). This standby exists purely for failover, it serves no read traffic under normal operation, which is the key fact that separates it from a read replica. The read replicas are placed across additional AZs (they can share an AZ with the standby; there is no requirement to keep them disjoint), sized in count to the actual read throughput need, addressed below.
Automated failover behavior
On primary failure or an AZ-level impairment, RDS promotes the standby and repoints the DB instance's single existing endpoint (its DNS CNAME, canonical name record) at the newly promoted instance rather than requiring any connection-string change. A classic RDS DB instance has exactly one endpoint total, not a separately named "writer endpoint" the way an Aurora cluster does, so failover here is a DNS re-point of that one address, not a switch between two differently-named endpoints. AWS states this happens "as quickly as 60 seconds" for a single-standby Multi-AZ deployment, with most failovers completing within one to two minutes. The part this leaves in the application's hands: a client with an existing connection to the now-dead primary, and a driver or pool that caches the resolved IP rather than re-resolving DNS promptly, can keep retrying a dead address well past the point RDS considers the failover complete. Short connection timeouts and prompt DNS re-resolution in the connection pool (the application's cache of reusable open database connections) are what actually determine how much of that 60-to-120-second window the application experiences as a real outage.
Connection-routing strategy for read/write separation
Two implementation options, and a fact that has to be stated up front: unlike Aurora, plain RDS does not give you a single, AWS-managed, load-balanced endpoint spanning multiple read replicas. Each RDS read replica is provisioned with, and reachable only through, its own individual endpoint, exactly like any other standalone RDS DB instance. Getting a single logical "read" address, if the application wants one, means building it, either in application code or via a proxy in front of the replica fleet; AWS does not hand it to you for free the way it does for Aurora's cluster reader endpoint.
- App-layer routing: hold the primary's own endpoint for writes, and the individual read replica endpoints for reads, and route each query explicitly in application code (round-robin, random choice, or a fixed per-service assignment across the known replica endpoints). Simplest to reason about, and its failure mode (a write accidentally sent to a read replica) is a loud, immediate error, since replicas reject writes, not a silent correctness bug.
- Proxy-layer routing: a pooler or router (for example a network load balancer or HAProxy in front of the replica fleet, or ProxySQL) that inspects statement type and routes automatically, which also gives the application a single logical address for reads instead of managing the replica endpoint list itself, at the cost of another moving part to operate.
For most teams, start with explicit app-layer routing; it is simpler, its failure mode is safe (a hard error, not silent data drift), and a routing proxy is worth adding later specifically once the number of call sites makes manual discipline unreliable, not as a default from day one.
Handling replica lag
Route only traffic that tolerates some staleness to the reader endpoint, browsing and search-style reads. Any read that must reflect the user's own just-completed write (an order confirmation immediately after checkout is the classic case) should either read from the writer directly for that specific call, or carry a short-lived "read your own write" hint that pins that one request to the writer rather than trusting the load-balanced reader endpoint. Monitor the replica lag metric per replica and treat a replica whose lag exceeds a defined threshold as degraded for routing purposes, since a healthy-but-lagged replica can otherwise keep receiving its normal share of read traffic through the load-balanced endpoint even while serving noticeably stale data.
Testing failover, not just designing for it
Design intent is not validation. Rehearse the actual failover, on a non-production copy at minimum, ideally occasionally against production during a planned low-traffic window once confidence is established:
aws rds reboot-db-instance --db-instance-identifier prod-webapp-db --force-failover
Measure the application's own observed error rate and recovery time during that test, not just the RDS-reported failover duration; those two numbers are not the same thing. A failover that completes on RDS's side in under a minute can still produce several minutes of application-visible errors if the connection pool does not detect the broken connection and re-resolve DNS promptly, which is exactly the gap the "automated failover behavior" section above flags as the application's responsibility, and the only way to know whether that gap is actually closed is to force a real failover and watch what the application does.
Worked example: sizing the read-replica count
peak_read_qps = 40_000
per_replica_sustained_qps = 8_000 # measured comfortable sustained capacity per replica
replicas_for_load = -(-peak_read_qps // per_replica_sustained_qps) # ceiling division
replicas_with_n_plus_1 = replicas_for_load + 1 # tolerate one replica down for maintenance or failure
print(f"Replicas needed to cover peak load: {replicas_for_load}")
print(f"Replicas with N+1 redundancy: {replicas_with_n_plus_1}")
print(f"Per-replica load with one of {replicas_with_n_plus_1} down: "
f"{peak_read_qps/(replicas_with_n_plus_1-1):,.0f} QPS")
Replicas needed to cover peak load: 5
Replicas with N+1 redundancy: 6
Per-replica load with one of 6 down: 8,000 QPS
Five replicas exactly cover the stated peak with zero slack, so losing even one to maintenance or a fault pushes the remaining four over their sustained capacity; provisioning a sixth (N+1) means the fleet still exactly meets sustained capacity, not exceeds it, with one replica out, which is the actual bar "can survive a single replica failure without degrading" sets.
Trade-offs and pitfalls
- Treating the Multi-AZ standby as if it also offloads read traffic is a common misunderstanding that leads teams to under-provision real read replicas; the standby's only job is failover.
- Aurora specifically allows a read replica to also serve as a Multi-AZ failover target if placed in a high-priority promotion tier (tiers run 0 to 15, and tier 0, the lowest number, is the highest priority for promotion, not tier 15), which blurs this line further in Aurora's favor (one fewer idle resource), and Aurora also provides a single load-balanced cluster reader endpoint spanning every Aurora Replica automatically. Neither behavior is something plain RDS read replicas provide: plain RDS replicas are each their own standalone endpoint, with no automatic promotion-to-failover-target and no built-in load balancing across them, which is exactly why a team that wants that behavior for free, rather than building it, often picks Aurora specifically for this reason.
- Skipping the forced-failover rehearsal is the single biggest gap between "designed for high availability" and "actually highly available": the DNS-flip mechanism and the AWS-stated failover time are necessary but not sufficient, since the application's own reconnect behavior is what determines real user-facing downtime.
- Load-balancing reads purely on the reader endpoint without a lag-aware exclusion means a badly lagged but technically healthy replica keeps taking its normal share of traffic, quietly serving stale reads to whichever users happen to be routed to it.
Describe how you would deploy a managed relational database for a production web application. Cover service selection (RDS/Aurora/Cloud SQL/Managed MySQL), sizing (instance class, storage type), read scaling, encryption, maintenance windows, and routine maintenance practices.
Sample Answer
Deploying a managed relational database for production is five deliberate decisions, not one "create instance" click: which service, what size, whether you need read scaling yet, encryption configured before first write, and a maintenance window you chose rather than accepted by default.
Service selection
- Amazon RDS: the broadest engine choice (MySQL, PostgreSQL, MariaDB, SQL Server, Oracle, Db2), engine-native replication, and the most flexible instance sizing including small burstable classes. The default choice for most web applications unless you have a specific reason to want more.
- Amazon Aurora: MySQL- and PostgreSQL-compatible, with storage-layer replication instead of engine-level replication, which gives materially faster failover (automatically switching to a healthy standby when the primary fails) and a higher write-throughput (the volume of write operations the database can sustain per second) ceiling on the largest instance classes. Worth the higher baseline cost when you expect to outgrow a single well-tuned RDS instance, or when failover speed matters more than for a typical web app.
- Cloud SQL: Google Cloud's equivalent to RDS for MySQL, PostgreSQL, and SQL Server, same category of decision as RDS on AWS.
For a typical production web application starting out, RDS or Cloud SQL on the engine your team already knows is the right default; reach for Aurora specifically when you can point to a concrete scaling reason, not by default.
Sizing: instance class and storage type
Pick the instance class from measured or realistically estimated CPU and memory need with headroom, not a round number picked for comfort. Storage type matters as much as instance class:
- gp3 (general purpose SSD): the right default for most online transaction processing (OLTP) workloads. It decouples provisioned IOPS (input/output operations per second: the count of individual read or write operations the volume can handle each second, distinct from throughput's raw bytes-per-second measure) and throughput from allocated storage size, so you are not forced to over-allocate storage just to get more IOPS.
- io2 / io2 Block Express: for latency-sensitive, very high-IOPS workloads where gp3's ceiling is not enough; costs more per provisioned IOPS.
- Enable storage autoscaling so unplanned data growth raises allocated storage automatically instead of the instance running out of space and rejecting writes.
Read scaling
Add a read replica only when there is a specific, measured reason: reporting or analytics queries competing with transactional traffic for CPU and I/O, or read throughput that a single writer genuinely cannot serve even with caching in front of it. A read replica is a full additional instance (real cost, real operational surface via replication lag, the delay before a write on the primary becomes visible on the replica), not a free performance knob, so route only the traffic that needs it, typically through RDS Proxy or explicit read/write splitting in the application, rather than defaulting every read query to a replica.
Encryption
Enable storage encryption at creation time: RDS cannot add encryption to an existing unencrypted instance in place, only by snapshotting it, copying the snapshot with encryption enabled, and restoring from that copy, so getting this right on day one avoids a disruptive migration later. The AWS-managed key (which AWS automatically rotates every year) is sufficient for most applications; use a customer-managed KMS (Key Management Service) key only when you need explicit control over key policy or rotation cadence, for example a compliance requirement to control who can disable access to the key. Separately, enforce TLS (transport layer security) on connections at the engine level, rds.force_ssl for PostgreSQL or require_secure_transport for MySQL, since encryption at rest alone does not protect credentials and query data in flight between the application and the database.
Maintenance windows and routine maintenance
If you do not set a maintenance window explicitly, RDS assigns a random 30-minute window in your region, which may land during business hours. Choose one deliberately, during genuinely low-traffic hours for your actual user base, and prefer applying non-urgent changes during that window rather than immediately, so patches land predictably instead of mid-peak-traffic. Keep automatic minor-version upgrades on for security patches, but review major version upgrades manually since those can carry breaking changes to the engine's behavior. As routine practice: review the instance's "pending maintenance actions" on a regular cadence rather than being surprised by them, audit the parameter group for drift from intended settings after any manual tuning, and revisit instance rightsizing on a schedule (for example quarterly) instead of only reacting once performance has already degraded.
Worked example: a concrete creation command
aws rds create-db-instance \
--db-instance-identifier prod-webapp-db \
--engine postgres --engine-version 16.4 \
--db-instance-class db.r6g.xlarge \
--allocated-storage 200 --storage-type gp3 --iops 6000 --storage-throughput 250 \
--max-allocated-storage 500 \
--storage-encrypted --kms-key-id alias/prod-db-key \
--preferred-maintenance-window "sun:05:00-sun:06:00" \
--auto-minor-version-upgrade
This encodes the sizing decision (an r6g.xlarge, memory-optimized Graviton instance class, a common default for a moderate production web app), gp3 storage with explicit IOPS and throughput rather than whatever the default happens to be, storage autoscaling up to 500 GiB so growth does not hit a hard wall, encryption specified at creation with a named key, a chosen (not default) maintenance window, and automatic minor-version patching. Multi-AZ and backup retention are deliberately left off this example: they are availability and disaster-recovery decisions with their own trade-offs, not part of the base provisioning choice this question is about.
Trade-offs and pitfalls
- Defaulting to the largest instance class "to be safe" wastes money every hour it runs oversized; defaulting to the smallest "to save money" invites a production incident the first time traffic grows. Size from data, and revisit it, rather than picking once and forgetting.
- Adding a read replica before you have a measured reason adds real operational surface (monitoring replica lag, routing logic) for no benefit, and it is easy to add one reflexively because "that's what scaling a database looks like."
- Forgetting to set storage encryption before the first write is the single most expensive mistake in this list to fix later, since fixing it means a snapshot-copy-restore cycle on a live database rather than a config change.
- Accepting the default maintenance window is a common, easily avoided source of an unplanned mid-day blip; it costs nothing to set deliberately at creation time.
Describe the step-by-step plan to migrate a production MySQL database from self-hosted VMs to Amazon RDS with minimal downtime. Include schema transfer, initial data load, how you'd keep the two in sync until cutover, migrating users and permissions, cutover strategy, smoke tests, and rollback plan.
Sample Answer
The shape of a minimal-downtime migration to a managed service is always the same regardless of engine: get the schema and a data snapshot onto the target first, then keep the target continuously in sync with the source via replication while the application keeps running against the source, then do a short, controlled cutover that only has to cover the last few seconds of lag, not the whole dataset.
Step-by-step plan (MySQL self-hosted to Amazon RDS for MySQL)
1. Schema transfer. Dump the schema only (mysqldump --no-data) and review it for anything RDS restricts, for example storage engines other than InnoDB, filesystem-level FILE privilege usage, or custom plugins, since RDS does not give you host access to install them. Apply the reviewed schema to the new RDS instance.
2. Capture a consistent starting point and load the data. On the source, briefly make it read-only and record the exact binary log (binlog) position writes will resume from:
FLUSH TABLES WITH READ LOCK;
SET GLOBAL read_only = ON;
SHOW MASTER STATUS;
-- File: mysql-bin-changelog.000031, Position: 107
Check the source's MySQL version before you run this: MySQL 8.4 removed SHOW MASTER STATUS entirely (tested against a real MySQL 8.4.11 instance: it fails with ERROR 1064: You have an error in your SQL syntax), replacing it with SHOW BINARY LOG STATUS, which returns the identical File / Position columns. Use SHOW MASTER STATUS for a MariaDB or MySQL 8.0-and-earlier source, SHOW BINARY LOG STATUS for a MySQL 8.4+ source. Since a self-hosted fleet migrating today is plausibly already on 8.4, confirm with SELECT VERSION(); first rather than assuming the older syntax works.
Then load the data into RDS with a consistent, compressed dump and immediately unlock the source (this lock window is seconds, not the length of the whole migration):
mysqldump --databases mydb --single-transaction --compress --order-by-primary \
-u local_user -plocal_password | mysql --host=mydb.xxxxx.us-east-1.rds.amazonaws.com \
--port=3306 -u rds_user -prds_password
SET GLOBAL read_only = OFF;
UNLOCK TABLES;
--single-transaction avoids a long table lock on InnoDB tables by taking a consistent snapshot inside one transaction, which is what keeps this step from blocking the live application.
3. Keep the two in sync until cutover. Create a replication user on the source with the minimum privileges replication needs, then tell RDS to become a live replica of the source starting exactly at the binlog position captured in step 2:
-- on the source
CREATE USER 'repl_user'@'%' IDENTIFIED BY 'password';
GRANT REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'repl_user'@'%';
-- on the RDS instance, as the master user (MySQL 8.4+ syntax; use rds_set_external_master
-- on MariaDB and MySQL 8.0 and earlier)
CALL mysql.rds_set_external_source('source.mydomain.com', 3306, 'repl_user', 'password',
'mysql-bin-changelog.000031', 107, 0);
CALL mysql.rds_start_replication;
RDS now streams every write the source makes from that point forward. The source's security group or firewall has to allow inbound connections from RDS's IP, and the source keeps serving the live application unchanged during this whole window, which can run for hours or days if needed.
4. Migrate users and permissions. RDS does not expose the mysql.user table for direct copying (there's no host-level MySQL account on a managed instance), so recreate accounts explicitly: script out every CREATE USER and GRANT statement from the source (SHOW GRANTS FOR user@host per account) and replay them against RDS, adjusting anything that depended on filesystem or SUPER privileges the RDS master user does not have.
5. Cutover. Watch replication lag (SHOW REPLICA STATUS / the Seconds_Behind_Source field, or the RDS ReplicaLag CloudWatch metric) until it is at or near zero. Put the application into a brief maintenance/read-only mode, wait for lag to hit zero (this should now be a few seconds, not the original full migration time), stop replication (CALL mysql.rds_stop_replication), repoint the application's connection string to the RDS endpoint, and take the application out of maintenance mode.
6. Smoke tests. Before declaring success: run row counts and CHECKSUM TABLE on the highest-value tables on both source and target and confirm they match exactly; run the application's own smoke test suite (login, a representative write, a representative read) against the RDS endpoint; confirm the replication user and any migration-only grants are revoked.
7. Rollback plan. Do not decommission the source. Keep it running, unmodified, for a defined window after cutover (for example 24 to 72 hours) so that if something is wrong with the RDS target, traffic can be pointed back to the source immediately with no data loss, since the source was only ever paused for the seconds-long lock in step 2, never stopped. This rollback window is only clean while the source has not diverged: once the application is writing to RDS post-cutover, "rolling back" means accepting that writes made after cutover are on RDS and would need to be manually reconciled back to the source, so treat the rollback option as valid within the cutover maintenance window and increasingly costly the longer you wait after it.
Postgres variant: logical replication instead of binlog shipping
For a single-node PostgreSQL source, the equivalent sync mechanism is native logical replication rather than binlog shipping, and RDS for PostgreSQL supports it directly. On RDS, an account with the rds_superuser and rds_replication roles (the two elevated permission roles RDS grants in place of true Postgres superuser access; this one needs both to turn logical replication on) sets the static parameter rds.logical_replication = 1 (this also raises wal_level, how much detail Postgres records in its write-ahead log, from just enough for crash recovery up to enough to support logical replication, and max_wal_senders, the number of concurrent replication connections the source can serve, and related parameters) and reboots the instance to apply it. On the source:
CREATE PUBLICATION mig_pub FOR ALL TABLES;
On the RDS target:
CREATE SUBSCRIPTION mig_sub
CONNECTION 'host=source.mydomain.com port=5432 dbname=mydb user=repl_user password=secret'
PUBLICATION mig_pub;
Unlike the MySQL flow, this single CREATE SUBSCRIPTION statement does both the initial data copy and then keeps streaming ongoing changes, so there's no separate "load the dump, then start replication" step. The two things to check before committing to this path that MySQL doesn't force on you: extension compatibility (RDS PostgreSQL only supports a curated allow-list of extensions, so SELECT * FROM pg_available_extensions on the RDS target against what the source actually uses, before migrating, not after) and that every table has a primary key or unique index (logical replication can't apply UPDATE/DELETE without one). For a smaller database that can tolerate a real maintenance window instead of a live sync, pg_dump/pg_restore with --jobs for parallelism is simpler and avoids the replication setup entirely; it trades a longer, bounded downtime window for one less moving part.
Post-cutover verification (both engines)
Row counts alone can hide corruption (same count, wrong rows), so verify per table with a cheap content check, not just COUNT(*):
-- MySQL
CHECKSUM TABLE orders, order_items, users;
-- PostgreSQL
SELECT count(*), sum(hashtext(t::text)) FROM orders t;
Run the same query on source and target and diff the results before you call cutover done. A mismatch here, caught before the maintenance window closes, is recoverable; the same mismatch discovered a week later, after both databases have diverged, is not.
Describe encryption in transit and at rest for managed database services. For AWS RDS/Aurora: list the steps you'd take to verify TLS is enforced on connections, confirm encryption at rest is enabled using KMS, and validate that snapshots and read replicas remain encrypted. Mention potential performance impacts and key rotation considerations.
Sample Answer
Encryption at rest and encryption in transit are two separate controls on RDS and Aurora, and verifying "the database is encrypted" means checking both independently, plus confirming the encryption actually propagates to every derived artifact (snapshots, same-region replicas), since none of that is a single toggle.
Verifying TLS is enforced on connections
- Check the engine parameter that forces secure connections:
rds.force_sslfor PostgreSQL,require_secure_transportfor MySQL, in the instance's parameter group, and confirm it is set to require (not merely allow) TLS. - Connect and confirm the connection actually negotiated TLS: on PostgreSQL,
SHOW ssl;should returnon; on MySQL,SHOW STATUS LIKE 'Ssl_cipher';should return a non-empty cipher name (an empty result means the connection is not encrypted). - Confirm the application's own connection string requests TLS (for example
sslmode=requireon PostgreSQL). Enforcing the server-side parameter without updating every client is a coordinated two-sided change: once enforcement is on, a client that does not request TLS is rejected outright at connection time, not degraded gracefully, so this needs to be tested in staging before it is flipped on in production. - Confirm the client's trust store has the current
rds-cacertificate bundle. AWS periodically rotates these certificates with advance notice and a transition period; a client trusting only the old certificate will start failing to connect at the rotation deadline, which shows up as a hard cutover, not a gradual warning.
Verifying encryption at rest and confirming it propagates
- Confirm the instance itself is encrypted:
aws rds describe-db-instances --db-instance-identifier mydb --query "*[].{StorageEncrypted:StorageEncrypted}" --output text, or the Encryption field under Configuration in the console. Also note which AWS Key Management Service (KMS) key ID is attached, an AWS-managed key or a customer-managed key. - Confirm snapshots are encrypted with the same key as the source. This is enforced by RDS, not something you can misconfigure: an unencrypted instance cannot produce an encrypted snapshot directly, only via copying an existing snapshot with encryption enabled, and any snapshot of an already-encrypted instance must use the same KMS key as that instance.
- Confirm same-region read replicas use the identical KMS key as the primary, again structurally enforced (you cannot pair an encrypted primary with an unencrypted replica, or the reverse). A cross-region replica uses a key in its own destination region instead, since KMS keys are region-scoped, so verifying a cross-region replica means checking the destination region's key, not re-checking the source's.
Illustrative shape of a verification run (not a captured live run)
$ psql "host=mydb.xxxx.rds.amazonaws.com sslmode=require" -c "SHOW ssl;"
ssl
-----
on
$ aws rds describe-db-instances --db-instance-identifier mydb \
--query "*[].{StorageEncrypted:StorageEncrypted,KmsKeyId:KmsKeyId}"
[{"StorageEncrypted": true, "KmsKeyId": "arn:aws:kms:us-east-1:111122223333:key/abcd-..."}]
This is the shape of what a passing check looks like, not a run against a real instance; the actual output depends on the specific instance being checked.
Performance impact
AWS states that encryption at rest is handled transparently, using AES-256, with minimal performance impact, since the underlying hardware accelerates the cryptographic operations. In practice, at-rest encryption overhead is not a meaningful factor in sizing decisions for the vast majority of workloads. TLS in transit adds a small, mostly one-time cost at connection setup (the handshake), not a meaningful per-query cost once a connection is established; this is one more reason connection pooling matters for high-request-rate applications, since pooling means paying that handshake cost once per pooled connection rather than once per request.
Key rotation considerations
AWS-managed KMS keys rotate automatically every year with no ability to disable it. Customer-managed keys support optional automatic rotation, off by default, with a 365-day default rotation period that can be customized. The important nuance: rotation changes only the current key material used for new encryption operations going forward. It does not re-encrypt data that is already on disk, and decrypting existing data continues to transparently use whichever key material version originally encrypted it. This means rotation is not a way to "refresh" the encryption on data already written, and it does not reduce exposure from an already-compromised key, since the compromised key material remains valid for decrypting everything it originally encrypted. Genuinely rotating away from a suspected-compromised key means creating a new KMS key and re-encrypting the actual data end to end, which for RDS means the same snapshot-copy-with-a-new-key-then-restore cycle used to add encryption to a previously unencrypted instance, not simply toggling key rotation on.
Trade-offs and pitfalls
- Enforcing TLS on the server before every client is updated causes a hard outage at the moment of enforcement, not a gradual one; roll it out in a lower environment first and confirm every consumer's connection string before flipping the production parameter.
- At-rest encryption protects data on disk; it does not protect credentials or query data while they are moving between the application and the database. Treat the two as independent requirements, both need to be explicitly verified, neither implies the other.
- Assuming key rotation re-encrypts existing data is a common and consequential misunderstanding: if a key rotation cadence is being used to satisfy a compliance requirement, confirm with whoever owns that requirement whether rotating the key material is actually sufficient, or whether the requirement really means re-encrypting the data itself.
- Forgetting to check that a cross-region replica or a copied snapshot is encrypted with the correct destination-region key is an easy gap, since the source region's key ID is not valid in another region and the failure shows up as a create-time error, not a silent gap, but only if someone is actually watching for it.
Explain read replicas in managed database offerings: what they are, how they work in services like Amazon RDS/Aurora or Cloud SQL, what operational problems they solve, and what limitations or risks you'd want to watch for when relying on them.
Sample Answer
A read replica is an asynchronous, read-only copy of a primary database. The primary streams its stream of committed changes (the binary log in MySQL, the write-ahead log, or WAL, in PostgreSQL) to one or more replicas, which replay those changes to stay current. The replica accepts read queries but never writes, and it is always at least slightly behind the primary because replication is asynchronous: the primary does not wait for the replica before acknowledging a write.
How it works in managed offerings
- Amazon RDS (MySQL/PostgreSQL/MariaDB): uses the engine's own native replication mechanism, binlog-based for MySQL/MariaDB, WAL streaming for PostgreSQL, to ship changes to up to several read replicas per primary. Each replica is a fully separate instance with its own storage and its own endpoint.
- Amazon Aurora: replicas share the same underlying cluster storage volume as the writer instead of copying data over the network, so replica lag is typically much lower than RDS's engine-native replication under normal load, low enough that AWS markets Aurora's replicas as "low-latency." Aurora also lets a replica be promoted automatically as part of Multi-AZ failover (Multi-AZ means the provider keeps a synchronized standby copy of the database in a second availability zone; failover is automatically switching traffic to that standby, or here, to a promoted replica, when the primary fails), which RDS's plain read replicas cannot do.
- Google Cloud SQL: replicates asynchronously per-transaction from the primary to one or more read replicas, including optional cross-region replicas, using the same engine-native mechanisms under the hood.
In all three, the replica is provisioned and monitored by the provider (you don't manage the replication process by hand), but the replication itself is still asynchronous and lag is not bounded to zero.
What problems read replicas solve
- Read scaling: route SELECT-heavy traffic to one or more replicas so it does not compete with writes for CPU and I/O on the primary.
- Workload isolation: point a specific class of query, for example an analytics dashboard or a nightly reporting job, at a dedicated replica so a slow or expensive query can't degrade the primary's write latency.
- Reduced primary load and latency: fewer connections and less query volume on the primary generally means more headroom and lower p95/p99 latency (the response time for the slowest 5% and slowest 1% of requests, a way of measuring worst-case performance rather than the average) for the writes and reads that must hit the primary.
- A promotable target: an RDS read replica (not a Multi-AZ standby) can be manually promoted to a standalone writable instance, which is useful for cross-region disaster recovery or for splitting off a subset of traffic to its own database.
Limitations and risks to watch
- Replication lag is not bounded. Under normal load it can be small enough to ignore; under a write burst, a long-running DDL (data definition language, for example an ALTER TABLE) statement, or a large batch job on the primary, it can grow to seconds or minutes, and the replica has no way to signal "I'm behind" to a client unless you build that check yourself.
- Eventual consistency, not read-your-writes. A client that writes to the primary and immediately reads from a replica can see stale data, or even see the write "disappear" if it lands on a replica that hasn't caught up yet. This is the single most common production bug with read replicas: a user submits a form, the confirmation page reads from a replica, and the user's own just-written data appears missing.
- No cross-replica consistency guarantee. Two replicas of the same primary can be at different points in the replication stream at the same instant, so two requests hitting different replicas can see different, mutually inconsistent snapshots.
- Promotion is a manual, disruptive operation, not an automatic failover mechanism. Promoting a read replica breaks its replication link permanently, and every other reader and writer in the application needs to be repointed. This is a materially different (and slower) recovery path than a Multi-AZ standby failover.
- Real added cost. A read replica is a full running instance, not a lightweight cache, so it roughly multiplies your compute bill by the number of replicas you run.
Worked example: how a write burst turns into visible lag
Assume a primary normally emits 12 MB/s of write-ahead log under steady state, a replica can apply changes at 20 MB/s, and a 10-minute batch job pushes the primary's write rate to 30 MB/s.
backlog growth rate during burst=30−20=10 MB/sOver the 600-second burst that backlog reaches 10×600=6,000 MB of unreplayed write-ahead log sitting on the replica the instant the burst ends.
It is tempting to convert that 6,000 MB straight into a "seconds behind" figure by dividing by the replica's apply rate (6,000 / 20 = 300 seconds), but that is not the number a real replication-lag metric would show at that instant, and it is a common mistake when reasoning about replication under load. What tools like MySQL's seconds_behind_master or Aurora/RDS's ReplicaLag CloudWatch metric actually report is how old the transaction the replica is currently replaying is, which you get by comparing cumulative bytes applied against cumulative bytes produced, not backlog divided by capacity. At the moment the burst ends (600 seconds in), the replica has applied 20×600=12,000 MB while the primary has produced 30×600=18,000 MB since the burst started. The 12,000 MB the replica has applied so far is data the primary had already produced after just 12,000/30=400 seconds of the burst, so the replica is 600−400=200 seconds, about 3 minutes 20 seconds, behind at the instant the burst ends, not 300 seconds.
Lag keeps growing for a while even after the burst is over, because the replica is still capacity-constrained at 20 MB/s working through the backlog while the primary keeps producing new data (now at the lower 12 MB/s rate) that pushes "how old is what I'm applying right now" further out. Lag peaks about 5 minutes after the burst ends, at 300 seconds (5 minutes) behind, then shrinks back toward zero as the replica keeps applying at 20 MB/s against a primary now producing only 12 MB/s, a net drain rate of 20−12=8 MB/s. The 6,000 MB backlog itself (still sitting there right when the burst ends) fully drains 6,000/8=750 seconds, 12.5 minutes, after the burst ends, which is also when lag finally returns to zero.
Total elapsed time with a nonzero backlog and elevated lag: the 600-second burst plus the 750-second drain, 1,350 seconds, 22.5 minutes, not the 17.5 minutes you get from naively adding a 300-second instant-lag figure to the drain time. Lag grows for the first 900 seconds (15 minutes: the full 10-minute burst plus 5 more minutes while the primary is still outpacing the still-catching-up replica), peaks at 5 minutes behind, then shrinks over the remaining 450 seconds (7.5 minutes). For that whole 22.5-minute window, anything reading from that replica was looking at data that was seconds to minutes old. This is exactly the kind of window where a "read after write" bug shows up, and it is why replica lag needs to be an explicit, monitored metric (RDS and Aurora both expose it as a CloudWatch metric) rather than an assumption, or worse, a hand calculation done wrong under incident pressure.
Unlock Full Question Bank
Get access to all 7 Cloud and Managed Database Services interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.