Replication, Partitioning, and Sharding Questions

Direct answer

Replication lag breaks read-after-write consistency (the guarantee that a user who just wrote something sees that write on their very next read) whenever the read happens to land on a follower that has not yet applied it. There are four standard patterns to guarantee a user sees their own recent writes despite this: sticky sessions (route a user's reads to the same node their write went to, for a window of time), session-scoped read-your-writes tracking (remember the position a user's last write reached, such as an LSN, log sequence number: a monotonically increasing marker of a database's write history, and only serve their reads from a replica that has caught up to at least that position), forcing reads-after-writes to the primary for a short window, and, less commonly, bounded-staleness reads that block briefly until a replica catches up rather than either redirecting or serving stale data outright.

The four patterns, and their trade-offs

Worked example: a social-network timeline

A user posts a comment, then immediately reloads their own timeline. If the reload hits a replica still 800 milliseconds behind (a fully normal amount of replication lag under moderate load), the comment appears to have vanished, which reads as a bug even though nothing is actually wrong. A concrete implementation: on write, the API server captures the WAL position (or an equivalent monotonic write marker) the write reached and returns it to the client as an opaque read-your-writes token embedded in the response (client-side technique); the client attaches that token to its next read request; the read-routing layer (server-side technique) checks each candidate replica's current applied position and only sends the request to one that has caught up, falling back to the primary if no replica has caught up within a short timeout (for example 50 milliseconds), rather than blocking indefinitely. This combination avoids sticky-session's node-affinity cost and avoids sending every read to the primary, while still guaranteeing the user sees their own comment.

Implementation details worth naming explicitly

A connection pooler alone does not solve this: PgBouncer, the common PostgreSQL pooler, is a pure TCP-level connection pooler with no query inspection and no awareness of a replica's replication position, so it cannot itself implement token-aware read/write splitting, and pointing it at a random or round-robin pool of replicas will pick one that has not caught up. The token-aware routing logic has to live in a layer that actually understands queries and replica state: application-level routing that picks a connection target per request based on the token, or a proxy built for query routing rather than plain pooling (for example Pgpool-II's load-balancing mode, which does inspect queries to split reads from writes) sitting in front of, or instead of, a plain pooler like PgBouncer, which continues to do only what it is built for: pooling connections to whichever destination it is told to use. A retry-on-lag pattern (catch the case where no replica is caught up, retry against the primary once, rather than retrying the same lagging replica) keeps tail latency bounded. Session-affinity state (which token belongs to which in-flight session) needs to live somewhere the load balancer or API layer can cheaply access it on every read, typically in the request context itself (the token travels with the request) rather than in a separate lookup service, to avoid adding a new dependency to every read's critical path.

Trade-offs and pitfalls

Sticky sessions are the easiest pattern to ship first, but teams often leave them in place permanently rather than migrating to a token-based approach, which silently caps how much read-scaling benefit replicas actually provide as traffic grows, since sticky users never spread across the replica fleet. Forcing all writes' subsequent reads to the primary "to be safe" is the opposite mistake: it looks correct but reintroduces exactly the primary-load problem replicas exist to solve, for reads that, in most product flows, do not actually need same-millisecond freshness. The right default is to reserve the primary-forcing pattern for the specific, small set of user actions where staleness is visibly wrong (their own just-written content), and use the token-based approach for everything else.

Direct answer

Read-your-writes consistency (a user always sees their own recent write on their next read) and monotonic reads (once a user has seen a value, they never later see an older value, even across multiple reads) are two separate guarantees, and both break by default under asynchronous replication, in ways that are especially damaging for user-facing machine-learning (ML) features: a recommendation model that appears to "forget" a user's just-expressed preference, or a fraud score that flickers between two values on consecutive requests, reads as a broken or untrustworthy model even when the model itself is working correctly and the bug is purely in which replica answered which request.

Why ML features are especially sensitive to this

A typical user-facing ML feature reads a feature store (a database or cache of precomputed inputs to a model) that is itself fed by replicated writes: a user rates an item, that rating is written, and the recommendation feature is supposed to reflect it on the very next page load. Two specific failure patterns show up constantly in this setting:

• Read-your-writes violation: the user rates an item, reloads, and the recommendation feature reads from a replica that has not yet applied that write, so the "you just told us you liked this" signal appears absent, and the model output looks stale or wrong to the user even though the underlying data pipeline is healthy.
• Monotonic-reads violation: two consecutive requests from the same user session hit two different replicas with different lag, so a feature value the user "already saw" (say, an account balance used in a risk score) appears to move backward on the second request, which is confusing at best and, for anything risk- or trust-related, alarming.

Implementation options, and their performance implications

Worked example

A user swipes to dislike a recommended item; the feature store write records that signal, and the API server captures the write's LSN as a token, returning it embedded in the response. The very next recommendation request carries that token; the feature-serving layer checks candidate replicas' applied positions and picks one that is caught up, falling back to the primary with a short timeout (for example 30 milliseconds) if none are. Separately, the client also tracks the highest position it has observed across the whole session (not just the last write), so if a later, unrelated request would otherwise be routed to a replica that is behind a position the session already saw, that request is also redirected, preventing a value flickering backward even without an intervening write from this user (for instance, another process' write that a previous request already reflected).

Trade-offs and pitfalls

The most common mistake is only solving read-your-writes (routing based on the user's own last write) while leaving monotonic reads unaddressed, which still allows a value to regress across two reads whenever the replica pool's individual lag levels vary, even with no new write from the user in between; the two guarantees need to be designed together, using a per-session high-water mark rather than only a per-write token, if both matter for the feature in question. The second pitfall is applying strict, token-based routing to every feature read in the system rather than only the ones where staleness is genuinely visible and confusing to the user; over-applying it defeats much of the purpose of having read replicas at all, so it should be scoped deliberately to the specific features where a stale or regressing value would be user-visible and damaging to trust in the product.

Direct answer

Hold total capacity fixed and compare the two topologies on the dimensions that actually differ, since raw compute cost is close to a wash: most cloud instance families price roughly linearly with capacity within the family, so 5x the capacity costs close to 5x the hourly rate either way. The real trade-off is fixed per-instance costs (operational overhead, and for per-core-licensed engines, licensing) working in favor of fewer, larger instances, against failover blast radius and migration granularity working in favor of many, smaller ones. Build the model, then apply the workload's actual constraints (how much does an operator-hour cost, and how much write-capacity loss during a single-instance failure is tolerable) to pick a side, rather than defaulting to whichever is cheaper on the compute line alone.

The model

Inputs: a target sustained capacity (queries per second, QPS), a chosen instance size for each topology with its capacity and hourly price, a fixed monthly operational-overhead cost per instance (patch windows, monitoring and alerting setup, on-call cognitive load, backup scheduling, amortized as labor cost), and a redundancy factor (replicas per primary for high availability). From these, compute: total instance count for each topology, total monthly compute cost, total monthly operational cost, and blast radius (the fraction of total write capacity lost if exactly one primary instance fails).

"""Cost model: many-small-instances vs fewer-large-instances for a sharded datastore
holding a FIXED total capacity target. All unit costs below are stated, illustrative
assumptions (labeled as such), not live vendor pricing -- the model's structure and
the resulting trade-off is the point, not the specific dollar figures."""

TARGET_QPS = 400_000          # sustained peak the cluster must serve
REPLICAS_PER_SHARD = 1        # +1 synchronous replica per primary for HA (synchronous: must confirm a write before it counts as committed), both configs

# -- illustrative unit economics (compute pricing assumed LINEAR in capacity within
# a family: 5x the capacity costs 5x the hourly price -- realistic for most
# compute-optimized cloud instance families, stated as a simplifying assumption) --
SMALL_CAPACITY_QPS = 20_000
SMALL_PRICE_PER_HR = 0.68
LARGE_CAPACITY_QPS = 100_000  # 5x the small instance's capacity
LARGE_PRICE_PER_HR = SMALL_PRICE_PER_HR * (LARGE_CAPACITY_QPS / SMALL_CAPACITY_QPS)

FIXED_OPS_OVERHEAD_PER_INSTANCE_MONTHLY = 150  # patch windows, monitoring/alerting
                                                 # setup, on-call load, backup scheduling
                                                 # -- amortized labor, illustrative
HOURS_PER_MONTH = 730

def cluster_plan(capacity_per_primary, price_per_hr_per_instance):
    num_primaries = -(-TARGET_QPS // capacity_per_primary)  # ceil
    total_instances = num_primaries * (1 + REPLICAS_PER_SHARD)
    compute_cost = total_instances * price_per_hr_per_instance * HOURS_PER_MONTH
    ops_cost = total_instances * FIXED_OPS_OVERHEAD_PER_INSTANCE_MONTHLY
    blast_radius_pct = 1 / num_primaries * 100  # % of write capacity lost if 1 primary fails
    return {
        "num_primaries": num_primaries,
        "total_instances": total_instances,
        "compute_cost_monthly": compute_cost,
        "ops_cost_monthly": ops_cost,
        "total_monthly": compute_cost + ops_cost,
        "blast_radius_pct": blast_radius_pct,
    }

small = cluster_plan(SMALL_CAPACITY_QPS, SMALL_PRICE_PER_HR)
large = cluster_plan(LARGE_CAPACITY_QPS, LARGE_PRICE_PER_HR)

print(f"target: {TARGET_QPS:,} QPS sustained, {REPLICAS_PER_SHARD} replica/primary\n")
for name, plan in (("many small instances", small), ("fewer large instances", large)):
    print(f"-- {name} ({plan['num_primaries']} primaries, {plan['total_instances']} instances total) --")
    print(f"   compute:  ${plan['compute_cost_monthly']:,.0f}/mo")
    print(f"   ops overhead: ${plan['ops_cost_monthly']:,.0f}/mo")
    print(f"   TOTAL: ${plan['total_monthly']:,.0f}/mo")
    print(f"   blast radius if one primary fails: {plan['blast_radius_pct']:.1f}% of write capacity")
    print()

savings = small["total_monthly"] - large["total_monthly"]
savings_pct = savings / small["total_monthly"] * 100
print(f"fewer-large saves ${savings:,.0f}/mo ({savings_pct:.1f}%) at this target, driven entirely by "
      f"the {small['total_instances'] - large['total_instances']} fewer instances' worth of fixed ops "
      f"overhead -- compute cost itself is ~equal by construction (linear pricing assumption).")
print()

# -- resharding friction: how many individual shard-migration operations to move from
# the current topology to 2x capacity, one direction each way --
def split_steps(num_primaries):
    # each split takes 1 shard -> 2 shards; going from N to 2N needs N split operations
    return num_primaries

print("-- resharding friction: operations needed to DOUBLE total capacity --")
print(f"   many-small ({small['num_primaries']} primaries): {split_steps(small['num_primaries'])} "
      f"individual shard-split operations, each moving ~1/{small['num_primaries']} of the data")
print(f"   fewer-large ({large['num_primaries']} primaries): {split_steps(large['num_primaries'])} "
      f"individual shard-split operations, each moving ~1/{large['num_primaries']} of the data "
      f"-- {small['num_primaries'] / large['num_primaries']:.0f}x FEWER migrations needed, but each one relocates "
      f"{small['num_primaries'] / large['num_primaries']:.0f}x more data and briefly holds "
      f"{100 / large['num_primaries']:.1f}% of total capacity at reduced redundancy mid-split")

Output:

target: 400,000 QPS sustained, 1 replica/primary

-- many small instances (20 primaries, 40 instances total) --
   compute:  $19,856/mo
   ops overhead: $6,000/mo
   TOTAL: $25,856/mo
   blast radius if one primary fails: 5.0% of write capacity

-- fewer large instances (4 primaries, 8 instances total) --
   compute:  $19,856/mo
   ops overhead: $1,200/mo
   TOTAL: $21,056/mo
   blast radius if one primary fails: 25.0% of write capacity

fewer-large saves $4,800/mo (18.6%) at this target, driven entirely by the 32 fewer instances' worth of fixed ops overhead -- compute cost itself is ~equal by construction (linear pricing assumption).

-- resharding friction: operations needed to DOUBLE total capacity --
   many-small (20 primaries): 20 individual shard-split operations, each moving ~1/20 of the data
   fewer-large (4 primaries): 4 individual shard-split operations, each moving ~1/4 of the data -- 5x FEWER migrations needed, but each one relocates 5x more data and briefly holds 25.0% of total capacity at reduced redundancy mid-split

At this specific target (400,000 QPS, "small" instances rated at 20,000 QPS each versus "large" instances rated at 100,000 QPS each, 5x the capacity at 5x the hourly price, one HA replica per primary), fewer-large saves about 18.6 percent per month, entirely from having 32 fewer instances' worth of fixed operational overhead to carry, since compute cost is identical by construction (both plans total $19,856/mo in compute). But fewer-large also concentrates 25 percent of total write capacity behind a single instance failure, versus 5 percent for many-small, a 5x difference in blast radius for an 18.6 percent cost saving.

Failover impact

Blast radius scales as 1 divided by the number of primaries: many-small spreads a single-instance failure across a larger denominator, so losing one instance costs less of the total in relative terms, at the (usually acceptable) cost of more moving parts to monitor. This is the sharpest trade-off in the whole model and the one most likely to actually decide the answer: a 25 percent capacity loss during a failover (the process of detecting a dead instance and promoting a replacement to take over its work) is a very different incident than a 5 percent one, and that difference does not show up on the cost line at all.

Migration complexity

Doubling total capacity by splitting each primary in two costs one shard-split operation per existing primary: many-small needs more individual operations (more shards to split), but each one moves a smaller fraction of the data and holds a smaller fraction of total capacity at reduced redundancy during the split. Fewer-large needs fewer operations, but each one is a bigger, riskier undertaking: more data in flight per split, and a larger fraction of total capacity briefly exposed while that split is in progress.

Applying the model to a decision

• Choose fewer-large when: operator time is the actual scarce resource (a small team managing the cluster), the workload can tolerate a real capacity dip during a single-instance failure (redundancy and fast failover elsewhere in the stack absorb it), and the per-instance overhead assumption in the model reflects real toil, not a number picked to make the comparison come out a certain way.
• Choose many-small when: the workload has a tight availability commitment where a 20+ percent capacity loss during a routine failure is unacceptable, resharding needs to happen incrementally and safely (smaller, lower-risk operations), or the engine is licensed per core with a meaningful PER-INSTANCE base fee on top, in which case more instances directly costs more in licensing, not just more operational overhead, and the model above would need that term added explicitly before trusting its conclusion.
• The model's answer is only as good as its inputs. Before applying it, verify the operational-overhead-per-instance figure against real toil (ticket volume, on-call load) rather than a guess, and verify whether the specific engine's licensing model is truly per-core (favors neither topology directly) or has a per-instance component (favors fewer-large).

Trade-offs and pitfalls

• Do not let "compute cost is roughly equal" get mistaken for "the two options are roughly equivalent." The blast-radius and migration-granularity differences are real and often matter more than the modest overhead saving.
• The linear-pricing assumption in the model is a simplification stated explicitly, not a verified fact for every vendor and instance family; re-derive the compute-cost lines with the actual current pricing for the instance types under consideration before using this for a real budget decision, since cloud list pricing changes and sometimes is not perfectly linear at the largest instance sizes.
• A cost model that never gets revisited goes stale. Redundancy requirements, instance pricing, and the team's actual operational capacity all drift over time; treat this as a model to re-run periodically, not a one-time decision.

Direct answer

Do not trust the directory table as ground truth; audit it against reality by asking each shard what it actually owns and diffing that against what the directory claims, on a schedule, then correct any mismatch with a compare-and-swap update to the directory (never a blind overwrite) followed by propagating the correction to routers before declaring it fixed. The directory is a cache of an assignment decision, not the assignment itself, and treating it as authoritative is exactly what lets partial-update drift go unnoticed until a client gets routed wrong.

Structured elaboration

Why directory entries go stale

A directory-based scheme stores an explicit mapping (key range or key, to owning shard) in a small database, updated whenever ownership changes (a migration, a manual reassignment, a failover). "Occasionally becomes inconsistent due to partial updates" describes exactly the failure mode you would expect from a two-step process (update the directory, then have the shard start actually serving that range, or vice versa) that is not atomic: a crash, a timeout, or a retry landing in between those two steps leaves the directory saying one thing while reality is another.

Audit workflow

1. Get an independent report of reality from every shard: each shard reports the key ranges it is CURRENTLY actually serving (not what it was told to serve), for example via a lightweight heartbeat or admin endpoint.
2. Diff that report against the directory's claimed ownership for the same key ranges.
3. Any mismatch is a candidate for correction, but classify it first: a range mid-migration (both an old and new shard legitimately have partial state) looks similar to genuine drift and needs a timestamp or migration-status check to tell apart, so you do not "fix" an in-progress, intentional transition.
4. Run this as a continuous or scheduled job, not a one-time cleanup, since the root cause (non-atomic updates) will keep producing new drift.

Correcting routing without downtime

• Never blindly overwrite the directory entry with what the shard reports; that risks overwriting a legitimate, still-in-progress migration state with a snapshot taken mid-transition. Correct only entries confirmed stable (unchanged across two consecutive audit runs, or explicitly outside any active migration window).
• Use a compare-and-swap update (update the directory entry only if its version/timestamp still matches what you observed when you decided to correct it) so a correction does not clobber a legitimate concurrent update that happened in between.
• Propagate the correction before declaring it fixed: a directory fix that has not reached the routers or their caches yet has not actually corrected any routing; push an invalidation (or wait out the cache TTL) and re-verify the audit is clean before closing the incident.

Tools and verification

• A periodic reconciliation job (the audit query below) as the primary detection tool, alerting on any confirmed-stale entry.
• The same reconciliation query, re-run after a correction, as the verification that the fix actually took, not just that the update statement succeeded.
• A migration-status or "in-flight" flag on the directory table that the reconciliation job checks before flagging a range as drifted, to avoid false positives during legitimate transitions.

Worked example

Executed against a real PostgreSQL 16 instance. A directory table and an independent "what does each shard actually report owning" table, with one deliberately injected drift (a completed migration the directory was never updated for):

CREATE TABLE directory (key_range_start INT, key_range_end INT, owner_shard TEXT, version INT);
INSERT INTO directory VALUES
 (0, 999, 'shard-A', 4),
 (1000, 1999, 'shard-B', 3),
 (2000, 2999, 'shard-B', 2);

CREATE TABLE shard_ownership_report (key_range_start INT, key_range_end INT, reporting_shard TEXT, reported_at TIMESTAMPTZ);
INSERT INTO shard_ownership_report VALUES
 (0, 999, 'shard-A', now()),
 (1000, 1999, 'shard-C', now()),   -- migration completed to shard-C, directory never got the update
 (2000, 2999, 'shard-B', now());

SELECT d.key_range_start, d.key_range_end, d.owner_shard AS directory_says, r.reporting_shard AS shard_actually_reports,
       (d.owner_shard IS DISTINCT FROM r.reporting_shard) AS stale
FROM directory d
JOIN shard_ownership_report r
  ON d.key_range_start = r.key_range_start AND d.key_range_end = r.key_range_end
ORDER BY 1;

Output:

 key_range_start | key_range_end | directory_says | shard_actually_reports | stale
------------------+---------------+-----------------+-------------------------+-------
               0 |           999 | shard-A        | shard-A                | f
            1000 |          1999 | shard-B        | shard-C                | t
            2000 |          2999 | shard-B        | shard-B                | f
(3 rows)

The audit query correctly isolates exactly the one drifted range (1000-1999: directory still says shard-B, but shard-C is the one actually reporting ownership) without flagging either of the two consistent ranges. The correction is then a version-guarded update, UPDATE directory SET owner_shard='shard-C', version=version+1 WHERE key_range_start=1000 AND key_range_end=1999 AND version=3, which fails harmlessly (updates zero rows) if the directory's version has moved on since the audit ran, forcing a re-check rather than silently clobbering a newer state.

Trade-offs and pitfalls

• Running the audit against a shard's OWN self-report, rather than an independent probe of the data itself (for example, sampling a known key and confirming it is actually answerable there), only catches drift the shard itself is aware of; a shard that has crashed or is otherwise confused about its own state can under-report or over-report ownership and this audit alone will not catch that class of failure.
• A reconciliation job with no in-flight-migration awareness will generate false-positive corrections during every legitimate migration, training the on-call team to ignore its alerts, exactly the failure mode that lets a REAL drift go unnoticed.
• Fixing the directory without propagating the fix (or without waiting out client cache TTLs) leaves the audit green while live traffic is still misrouted; the audit's own re-run is what actually confirms the end-to-end fix, not the UPDATE statement's success.
• The deepest fix is addressing why partial updates happen at all: making the directory update and the shard's ownership change part of one atomic, ordered operation (or explicitly two-phase, with a recorded in-flight state) removes the root cause instead of only detecting and repairing its symptom.

Direct answer

Treat a resharding tool the same way you would treat any other piece of production infrastructure that moves data: it needs a workload generator that can produce both uniform and skewed (hot-key) traffic, a chaos layer that injects node failures and network partitions mid-migration, an automated verification pass (checksums and aggregate metrics, not eyeballing), and a canary rollout that proves the tool on a small, low-risk shard before it ever touches a critical one, all wired into the normal CI/CD pipeline so a change to the resharding tool itself cannot ship without passing the same gauntlet.

Structured elaboration

Workload generators

• Uniform baseline: synthetic keys with an even access distribution, to establish the tool's behavior under the easy case first.
• Skewed / hot-key traffic: a small fraction of keys (for example, the top 1%) receiving a disproportionate share of requests, generated with a Zipfian or similarly skewed distribution, since this is the traffic shape most likely to expose a split threshold that is miscalibrated or a hot range the migration itself makes worse mid-copy.
• Realistic write/read mix and rate, ideally replayed from a sanitized sample of real production traffic shape rather than invented from scratch, so the test exercises the actual read-after-write and range-query patterns the tool will see in production.

Chaos tests

• Node failures: kill a shard mid-backfill and confirm the migration resumes from its last durable watermark rather than restarting or, worse, silently continuing with a gap.
• Network partitions: partition the coordinator from a subset of shards during the verification or cutover phase and confirm the tool either blocks (safe) or clearly surfaces the partition rather than proceeding on a partial view of the cluster.
• Disk saturation and elevated replication lag: this and the two failure modes above are exactly the class of behavior a distributed system must survive before going to production, and are worth validating generally (not only during a resharding run) as part of the same chaos harness, confirming the system degrades predictably rather than in a way that surprises the on-call engineer.

Canary rollout

Roll the tool out in stages of increasing blast radius: first against a synthetic or shadow shard with no real traffic, then against one real but low-criticality shard (an internal or low-traffic dataset), monitored closely for a bake period, before it is trusted against a critical, customer-facing shard. Each stage should have an explicit go/no-go gate (the automated verification below passing cleanly, plus no anomalous production metrics during the bake period) rather than a fixed timer.

Automated verification

• Row-level checksums: bucket rows by key range and compare an aggregate checksum per bucket between source and destination, isolating any mismatch to a specific range without a full-table diff. Run against a real database with a deliberately corrupted row, this catches the corruption immediately by isolating it to the one mismatched bucket, rather than requiring a full-table scan to find it.
• Aggregated metrics: row counts, sum/min/max of key numeric columns, and distinct-value counts per bucket as a cheap, fast cross-check that catches gross errors (a dropped range, a doubled range) even before the more expensive checksum pass runs.

Integration with deployment pipelines

The resharding tool is itself a piece of software with its own release cycle; a change to it (a new split-threshold default, a new verification check) should go through the same CI pipeline as any other service: unit tests for the pure logic (threshold math, checksum computation), then an integration stage that runs the workload generator and chaos tests above against a throwaway test cluster, and only a build that passes both stages is eligible for the canary rollout. Treat "passed on a synthetic cluster" and "cleared the canary bake period" as two separate, both-required gates, not one.

Worked example

A concrete pipeline stage sequence for validating a change to the split-threshold logic before it reaches production, as it would appear in CI:

1. Unit stage: run the threshold and checksum functions against fixed, hand-constructed inputs, cases that exercise the split/merge threshold math (a range crossing the hot-shard size limit, a pair of cold ranges below the merge floor) and the checksum mismatch detector (a deliberately corrupted row that must isolate to one bucket) are good candidates to codify directly as regression tests.
2. Integration stage: spin up a disposable multi-node test cluster, load it with both uniform and Zipfian-skewed synthetic traffic, run the resharding tool end-to-end (split a hot range, merge a cold one), and assert the automated verification pass reports zero mismatches.
3. Chaos stage: repeat the integration stage, but kill one node partway through the copy and confirm the tool resumes from its watermark and still reports zero mismatches at the end.
4. Canary gate: only a build that clears stages 1 to 3 is eligible to be pointed at a real, low-criticality shard for a bake period before wider rollout.
   A build failing at any stage blocks promotion to the next; this is what "integration with deployment pipelines" concretely means here, not a separate manual sign-off bolted on afterward.

Trade-offs and pitfalls

• A workload generator that only produces uniform traffic will pass every test while hiding exactly the hot-key behavior most likely to cause a real incident; skewed traffic generation is not optional depth, it is the test that matters most for this class of tool.
• Chaos testing only the resharding tool's own failure paths, and never the surrounding cluster's ordinary failure modes (a node dying for reasons unrelated to the migration), misses the more common real-world case where a migration is already in flight when an unrelated failure happens.
• A canary stage with no explicit go/no-go criteria (just "watch it for a while") tends to either ship too early under schedule pressure or never confidently ship at all; define the pass condition (verification clean, no anomalous metrics for N hours) before the first canary run, not during it.
• Treating verification as a manual, one-time check rather than an automated pipeline gate is the single most common way a resharding tool's regression reaches production, because a person tired at the end of a long migration is the least reliable verification step available.