Senior Site Reliability Engineer Interview Preparation Guide - Amazon

I would facilitate the discussion by making the decision criteria explicit before people debate solutions. A trade-off is a choice where improving one goal, like performance, may hurt another, like maintainability or scope.

My role in the room

• Restate the decision we need and the deadline for it.
• Ask the technical owner to summarize the options and their risks.
• Separate facts from preferences.
• Push the group toward criteria: user impact, long-term cost, and delivery timing.

If the room is stuck
I would ask, "Which risk is hardest to recover from later?" That usually surfaces the real priority. If there is no clear answer, I would propose a small follow-up experiment or time-boxed spike.

Example
If option A is faster to ship but harder to maintain, and option B is cleaner but slips the launch by 2 weeks, I would not decide for the engineers. I would ask them to recommend the least risky path, then confirm the business trade-off with the group.

The respect part is important: I make the conversation structured, but I do not override technical judgment. My job is to help the team leave with either a decision, an owner, or a clear next step.

Requirements (clarify): bind SCM commit → CI build artifacts → SBOM → test results → final image; cryptographic, auditable, verifiable at runtime; scalable; resilient to key compromise.

High-level design:

• Data model (canonical provenance document per build):

  • build_id, scm: {repo, commit_sha, signed_commit_proof}, builder: {id, image}, artifacts: [{name, digest}], sbom: {format, uri, digest}, tests: [{name, result, uri, digest}], timestamp, env: {ci_job, runners}, parent_builds[], attestations: [{type, signer, signature, uri}], policy_version.
  • Use JSON-serialized in in-toto/COSIGN provenance schema (with SPDX/CycloneDX links for SBOM).

• Signing strategy:

  • Use short-lived workload identity keys (OIDC tokens -> ephemeral keys) for CI runners; sign provenance with cosign using keyless (OIDC) or KMS-backed keys.
  • Git commit signing using developers' SSH/GPG; record git commit signature verification as part of provenance.
  • Test results and SBOM signed by the builder process; CI orchestration signs final attestation asserting that SBOM+tests+artifacts match image digest.
  • Key hierarchy: root key stored in HSM/KMS for signing long-lived policy changes; workload keys ephemeral.

• Attestation storage:

  • Primary: OCI registry as image attestation artifacts (OCI ref: image@sha256 and companion signatures/attestations via referrers API) — allows atomic tie to image digest.
  • Secondary: Transparency log (e.g., Rekor) for append-only public/verifiable entries (store attestation hashes and signer certs).
  • External datastore (immutable object store like GCS with retention) for large artifacts (full SBOMs, test logs) referenced by signed URIs; store content-addressed by digest.

• Runtime verification:

  • Admission controller (K8s) or deployment pipeline component:
    • Given image digest, fetch referrers from registry, locate provenance attestation, verify signer certificate via KMS/keys or OIDC token binding, check signature and transparency log entry.
    • Validate policy: matches allowed SCM repo/branch, required test results (pass, coverage), SBOM allowed components (deny list), image built by approved builder identity and timestamp window.
    • If attestation references external SBOM/logs, fetch by digest, verify digest matches attestation before evaluating content.
    • Fail fast on missing/invalid attestations; provide clear audit trail for incidents.

Trade-offs / resilience:

• Registry referrers -> low latency and atomicity; transparency log -> non-repudiation. External store keeps large payloads off-chain.
• Mitigate key compromise via short-lived keys, KMS/HSM, transparency logs, and re-attestation/rotation policies.

Monitoring & SRE ops:

• Alert on attestation verification failures, missing attestations, transparency-log mismatches.
• Expose metrics: percent of deploys with valid attestations, verification latency.
• Ensure rollout strategies allow emergency bypass with strict audit and rollback controls.

Direct answer

In thirty minutes I would give the C-suite (the most senior executives, such as the CEO and COO) and legal a short, structured brief that separates confirmed facts from unverified ones and from unknowns, then spend most of the time on decisions and on statements: who speaks, what the holding statement (a short, pre-approved public message that acknowledges the issue without claiming facts we have not confirmed) says, and what gets held back until verified. "Verified" throughout means confirmed by the team that owns the data or system, from evidence such as logs, not from an assumption. The goal is that legal can judge exposure from facts, not from my guesses, and that nothing public contradicts what we later confirm.

Suggested thirty-minute agenda

What I say (three buckets)

• Confirmed: we are seeing failures for the checkout service since 13:52 UTC. Customers in all regions are affected. Rollback in progress.
• Suspected, not verified: a recent deploy is the leading suspect.
• Unknown: whether any customer data was exposed or lost, whether payments were charged twice, the full customer count.

I state each bucket in those words. I also say what evidence would move something from suspected to confirmed and roughly when I expect it.

What to hold back until verified

• Root cause and fault (including anything about a vendor, an employee or an attacker).
• Any claim about data: "no data was exposed", "no personal data affected".
• Customer or revenue counts.
• Any fix time or "resolved".
• Anything that assigns liability.

Legal and regulatory questions

The clocks that may start, such as contractual notice to customers (for example, a contract saying affected customers must be told within 72 hours of discovery) or data-protection notification rules (laws requiring regulators or individuals to be told of a personal data breach within a set time), depend on facts and jurisdiction (which country or state's law applies) and are legal's decision. The 72 hours is only an example. My role: give legal the exact detection time, the current scope, and a running log of confirmed facts with timestamps, and ask them what obligations might apply and by when. I should not assert a deadline myself. Legal may also ask that sensitive written analysis go through them; I should follow that.

Lining up public statements

1. One spokesperson (often communications or a named executive), no freelancing.
2. A holding statement prepared now and approved by legal: it acknowledges the issue, says we are investigating and will update by a time, and makes no claim on cause or data. For example:

We are aware that some customers are having trouble completing purchases.
Our teams are investigating and working to restore service. We will share
an update by 15:30 UTC. We are sorry for the disruption.

3. One fact log as the only source for statements, so status page, customer emails, social posts and sales talking points match.
4. A trigger list agreeing what verified fact allows which statement:

5. A review path that is fast: legal review of the holding statement now, and a short turnaround for updates.

Worked example (illustrative)

Executive: "Were customer cards exposed?" Answer: "We have no evidence of that today, but we have not finished verifying; the payments team will confirm by 15:00. Until then we will not say either way publicly." That phrasing is accurate, and it does not create a false assurance.

Trade-offs and pitfalls

• Reassuring early ("no data was lost") is the classic mistake; it is easy to say and expensive to retract.
• Saying nothing to legal until everything is certain can cost time on obligations that already started.
• Over-long technical briefings waste the thirty minutes; leaders need decisions, not debugging detail.

Match the purchasing option to two properties of the workload: how predictable its usage is, and how much interruption it can tolerate. On-Demand is the flexible, no-commitment baseline; Reserved Instances and Savings Plans trade commitment for discount on predictable usage; Spot trades interruption risk for the deepest discount on fault-tolerant work. Most real fleets blend two or three of these rather than picking one.

The options

A detail worth knowing on Savings Plans: Compute Savings Plans apply flexibly across instance family, region, and OS, and even to Fargate and Lambda usage, at a slightly lower discount; EC2 Instance Savings Plans lock to a specific instance family in a specific region for a higher discount. The choice mirrors RI Standard vs. Convertible: more flexibility costs you some discount.

Spot interruption handling

Spot capacity can be reclaimed with a two-minute interruption notice, delivered via the instance metadata service and (optionally) an EventBridge (AWS's event-bus, used for routing events between services) event, giving the workload a short window to checkpoint or drain. Standard mitigations:

• Diversify across multiple instance types, sizes, and AZs using an Auto Scaling Group's mixed-instances policy or Spot Fleet with the capacity-optimized allocation strategy, so a single pool's interruption doesn't take out the whole fleet.
• Design the workload to checkpoint progress (batch jobs) or to be stateless behind a load balancer (services), so losing an individual instance costs at most a few seconds of in-flight work.
• Keep a base layer of On-Demand or committed capacity for the portion of the fleet that can't tolerate interruption, and let Spot absorb the elastic/overflow portion.

Worked example: a mixed steady-plus-batch fleet

Say a service runs a steady web tier of 10 instances 24/7, plus a nightly batch job that needs up to 40 additional instances for 4 hours a night, over a 30-day month.

The steady 7,200 instance-hours is exactly the shape a commitment wants: constant, predictable, easy to size a 1-year Compute Savings Plan against. The 4,800 batch instance-hours is short, spiky, and only runs 4 hours a night, so committing to it wastes money on the other 20 hours; it's a better fit for Spot (or On-Demand fallback via a mixed-instances policy if Spot capacity is thin in that pool), since the batch job is exactly the fault-tolerant, checkpointable kind of work Spot interruption risk is acceptable for.

Trade-offs and pitfalls

• Buying RIs or Savings Plans before rightsizing locks in the wrong instance shape for the term.
• Committed spend is a finance risk, not just a technical one: if the org's usage shrinks, the commitment doesn't shrink with it.
• Putting a single-instance stateful database on Spot is a classic anti-pattern; interruption there means an outage, not graceful degradation.
• Concentrating an entire Spot fleet in one instance-type/AZ pool defeats the point of diversification: correlated interruptions can reclaim most of the fleet at once.
• Savings Plans and RIs don't apply to Spot usage, so a team that shifts workload onto Spot to save money can end up under-utilizing a commitment they already bought for On-Demand capacity.

As IC (Incident Commander, the person directing the response) for a high-severity, cascading outage, the first 30 minutes are for stopping the bleeding and getting evidence-based clarity on the decision that matters most, whether to scale the database, roll back a recent change, or divert traffic away from it, not for finding the deep root cause. I declare severity and my role immediately, run diagnosis and mitigation-prep in parallel rather than sequentially, and commit to a decision by minute 25 to 30 based on the evidence gathered, because an IC who's still gathering evidence at minute 45 has effectively made the decision to do nothing.

The first 30 minutes

flowchart TD
  A[0-5m: Declare SEV, I am IC, page DB/SRE/service leads] --> B[5-15m: Parallel diagnosis]
  B --> C[Check DB metrics: CPU, IOPS, locks, recent migrations]
  B --> D[Check deploy timeline: what shipped in the last hour]
  B --> E[Check blast radius: which downstream services, what percent of customers]
  C --> F[15-25m: Weigh evidence against decision criteria]
  D --> F
  E --> F
  F --> G[25-30m: Commit to one action: scale, rollback, or divert]
  G --> H[Execute, assign owner, set 10-15m re-check]

Deciding between scaling, rollback, and traffic diversion

Coordinating across many teams at once

A cascading outage touching several downstream services usually means five or six engineering teams, plus customer support and executive stakeholders, all needing something different from the IC at the same time. The way to not drown in that is strict channel separation: one technical incident channel for engineering coordination and raw findings, one status thread for a fixed-cadence, plain-language update (every 10 to 15 minutes) that support and execs read without needing to parse technical detail. Each engineering team gets one named point of contact reporting into the IC, rather than the IC personally fielding questions from every team, so the coordination overhead scales with the number of contacts, not the number of engineers.

Trade-offs and pitfalls

The most dangerous failure mode in the first 30 minutes is analysis paralysis, continuing to gather evidence past the point where the marginal information changes the decision, because every extra minute at this severity is customer-facing impact. The opposite failure, committing to rollback or failover in the first five minutes before any evidence is in, is equally common in a panic and just as costly if the guess is wrong. The discipline that prevents both is the fixed decision checkpoint (commit by minute 25 to 30 based on whatever evidence exists then, not whatever evidence you wish you had).

Approach (iterative): Use a stack to process items in order, pushing list elements in reverse so original order is preserved. This avoids recursion depth limits.

Implementation:

def flatten_list(nested):
    """Flatten nested lists (iterative)."""
    result = []
    stack = list(reversed(nested))

    while stack:
        item = stack.pop()
        if isinstance(item, list):
            stack.extend(reversed(item))
        else:
            result.append(item)
    return result

Worked example, verified on CPython 3.12:

print(flatten_list([1, [2, [3, 4], 5], 6]))
# [1, 2, 3, 4, 5, 6]

Tracing the stack (pop() takes from the end, so pushing a sub-list's items in reverse order means they come back out of the stack in their original left-to-right order, since a stack is last-in-first-out: the last thing pushed is the first thing popped, so reversing before pushing cancels out the stack's own order-reversal): stack starts as [6, [2, [3, 4], 5], 1] (the whole input list, reversed). Popping 1 (a plain value) appends it to result, leaving stack = [6, [2, [3, 4], 5]]. Popping [2, [3, 4], 5] (a list) pushes its contents back on in reverse, [5, [3, 4], 2], giving stack = [6, 5, [3, 4], 2]. Popping 2 appends it (result = [1, 2]). Popping [3, 4] pushes [4, 3] back on, giving stack = [6, 5, 4, 3]. Popping 3, then 4, then 5, then 6 in that order appends each in turn, building result = [1, 2, 3, 4, 5, 6], matching the original left-to-right order exactly, and matching the printed output above.

Key points: preserves original order; handles arbitrary nesting; non-recursive.

Time Complexity: O(n) where n is total number of elements (visit each once).
Space Complexity: O(n) for result plus O(d) stack where d is nesting breadth (worst-case O(n)).

Edge cases: empty list, lists with non-list iterables (only lists are flattened per spec), deeply nested single chains handled without recursion.

Direct answer

The clearest example is picking up enough of an authentication protocol I'd never implemented before to unblock a partner integration that was already costing the team real time. I scoped tightly to what the task actually needed rather than the whole specification, gave myself a rough one-week target, and only trusted my own understanding once I could predict how it would behave before testing it, not once I got a demo to run.

Structured elaboration

Situation: a partner integration was stalled because none of us had implemented this particular authentication flow before, and every day it stayed stalled was a day of that partner's traffic we couldn't process.

Action: I split what I needed to know from what was merely interesting. I read the official specification for just the flow we needed (not the whole standard), found one working reference implementation in a language I understood to check my own reading against, and set a rough target of usable proficiency within a week, spread around my other commitments.

Validation: before I trusted it, I deliberately tried to break my own understanding rather than confirm it. I wrote out what should happen with an expired token mid-request and with a malformed callback, predicted the outcome for each in writing, then ran the cases and checked whether my predictions matched. I only considered myself done once they did, twice, not once I'd copied a working snippet that happened to pass a demo.

Worked example

Some of the reading happened on my own time over a weekend, because the trigger was time-sensitive and I wanted to walk in on Monday with a working plan rather than an intention to start one. By the end of the week I implemented the flow, it passed both the edge cases I'd predicted and integration testing, and the partner's traffic went live without a follow-up incident tied to the auth logic.

It's worth being honest that this doesn't always work out. On a similar problem later, I spent about a week learning a caching approach that looked promising, built a working prototype, and concluded during evaluation that it added more operational complexity than it actually saved, so I shelved it instead of shipping it. The week wasn't wasted: ruling it out with evidence, instead of guessing, was the useful outcome.

Trade-offs and pitfalls

The common wrong turn here is claiming you "learned" something because a demo worked once; that's evidence you copied something correctly, not that you understand the failure modes. The other pitfall is treating every hour spent learning as equally necessary: scoping tightly to what the specific problem needs, and being willing to defer or drop the rest, is what makes the timeline realistic under real pressure.

Requirements & constraints:

• Enforce security, reliability, compliance policies across CI/CD and IaC at scale (multi-repo, multi-team).
• Low friction for developers, fast feedback loops, auditable changes, ability to evolve policies safely.
• Support policy-as-code, automated checks, code review standards, training, and governance with rollback.

High-level architecture:

• Policy-as-code repo(s) → Policy Engine (OPA/Gatekeeper, Conftest, Sentinel) integrated into:
  • Pre-commit hooks & local CLI checks
  • CI pipeline gates (lint/test/policy)
  • GitOps admission controllers for clusters and infra
• Centralized Policy Registry + Versioned Catalog (semantic versions + changelogs)
• Policy CI: test harness that runs policy tests against fixtures, mocks, and terraform plan/json
• Developer-facing tools: CLI, VSCode extensions, PR bots, policy diff previews
• Governance UI: propose/review/approve policy changes, risk classification, rollout controls, audit logs

Core components & responsibilities:

1. Policy-as-code repo(s): policies written in Rego/HCL/OPA, unit & integration tests, sample manifests.
2. Policy CI pipeline: run static analysis, policy unit tests, integration tests against ephemeral infra (sandbox) and CI artifacts (terraform plan).
3. Gate layers:
   • Local: pre-commit, dev CLI for fast feedback
   • CI: enforce on PRs; failing policies block merge
   • Admission: runtime enforcement for GitOps/applications (fail-fast)
4. Code review standards: mandatory policy owner review for policy changes, checklist (test coverage, backward compatibility, risk), automation enforces reviewers.
5. Developer enablement: training workshops, runbooks, “policy playground” sandboxes, clear docs, sample fixes, office hours.
6. Observability & alerting: metrics for policy failures, false-positive rate, mean-time-to-fix; dashboards and weekly reports.

Policy evolution & governance model:

• Change proposal flow: propose PR to policy repo → automated tests → risk classification (low/medium/high) via metadata → staged rollout:
  • Canary: enable as advisory (logging only) for N days to collect violations
  • Gradual enforcement: quota-based or team-by-team enforcement
  • Full enforcement: global admission gates enabled
• Approvals: steering committee (SRE + security + product) for medium/high risk; automated merges for low-risk changes with tests passing.
• Versioning & release notes: semantic versions and explicit migration guides.
• KPIs & guardrails: track errors prevented, policy false positives, developer friction metrics (PR sleep time).

Safety & rollback strategy:

• Default to advisory mode: new/changed policies start with monitoring (non-blocking) to surface issues.
• Feature flags / toggles per-policy with scope (org/team/repo); use targeting to enable/disable quickly.
• Blue/green policy deployment: maintain previous policy version in registry; when enabling enforcement, deploy alongside and keep fallback.
• Automated rollback triggers:
  • If policy enforcement causes spike in failed merges, deploy-blocking incidents, or increased error budget consumption, alert on runbook thresholds.
  • Use CI gate to auto-disable enforcement toggle and open emergency PR to revert policy change.
• Emergency runbook: one-click disable via policy-control service, notify stakeholders, create follow-up RFC for fix.
• Post-incident: blameless postmortem, add more integration tests (including real-world fixtures), update rollout cadence.

Trade-offs:

• Strong enforcement early reduces incidents but increases developer friction — mitigated with advisory phase and gradual rollout.
• Investment in policy CI and sandboxes upfront reduces long-term toil and rollback frequency.

This plan embeds excellence by making policy changes safe, testable, visible, and reversible while providing developers with low-friction tools and clear governance.

Direct answer

A highly extensible, plugin-based CI architecture maximizes what any individual team can do with the platform, at the cost of a larger, harder-to-secure surface area and more operational burden maintaining that flexibility; a simple, opinionated set of pipeline templates trades some flexibility for lower operational burden, a smaller security surface, and much easier support at scale across many teams.

Structured elaboration

Developer velocity. A plugin-based architecture lets a team with an unusual need (an exotic build tool, a niche deployment target) solve it themselves without waiting on the platform team, which is a real velocity win for teams whose needs genuinely fall outside the common case. An opinionated template set is faster for the common case (most teams never need anything beyond the template) but can genuinely block or badly slow down the uncommon case, where a team's real need isn't expressible within the template's assumptions.

Operational burden. Every plugin (or every degree of freedom in a highly extensible system) is something the platform team has to account for when reasoning about the system's behavior, security posture, and upgrade path; a plugin ecosystem's collective maintenance burden (as covered concretely in the Jenkins-at-scale question) grows with the number of distinct plugins in active use across the organization, not just with the platform's own complexity. An opinionated, templated system has a much smaller surface: the platform team supports a small, known set of patterns, not an open-ended combinatorial space of whatever any team has chosen to install or configure.

Security surface area. Every plugin or extension point is a potential vulnerability, and a highly extensible system's actual security posture depends on the weakest plugin any team has installed, not just on the core platform's own security; an opinionated system's security review can focus on a small, fixed set of supported patterns instead of an open-ended, ever-changing set of third-party extensions.

Upgrade and testing complexity. A highly extensible system has to validate that platform upgrades don't break the potentially huge combinatorial space of plugin combinations teams have actually deployed, which is close to intractable to fully test; an opinionated system's upgrades only need to be validated against its own small set of supported templates, which is genuinely tractable.

Scaling support across hundreds of teams. An opinionated system scales support effort roughly with the number of distinct template patterns (small and roughly constant as team count grows), while a highly extensible one scales support effort closer to the number of distinct plugin/extension combinations in actual use (which tends to grow at least linearly, and often faster, with team count), making it a much harder support model to sustain as the organization grows.

Worked example

A 20-person startup benefits from a highly extensible, plugin-based system: few enough teams that the platform team can reasonably track what's installed, and the flexibility lets early, fast-moving teams solve their own unusual needs without a platform bottleneck. A 2,000-engineer enterprise with hundreds of teams is much better served by a small, opinionated set of well-supported templates with a genuinely rare, carefully-reviewed exception path for the handful of teams whose needs don't fit, because the alternative (letting hundreds of teams each install whatever plugins they want) produces an untestable, unsupportable combinatorial mess, as the earlier Jenkins-plugin-governance discussion covers concretely.

Trade-offs and pitfalls

The most common mistake is choosing maximal extensibility by default without accounting for how the operational and security burden scales with organization size, which works fine at small scale and becomes genuinely unsustainable well before an organization reaches hundreds of teams. The second is choosing a rigidly opinionated system with no legitimate exception path at all, which either blocks teams with genuinely different needs or, more likely, drives them to quietly work around the platform entirely, which is worse for consistency than a small, deliberate, reviewed set of exceptions.

A CRDT (Conflict-free Replicated Data Type) is a data structure whose update and merge operations are mathematically guaranteed to make every replica converge to the same value, with no locking, coordination, or central authority, as long as every update eventually reaches every replica. State-based CRDTs ship the whole replica state and merge it with a commutative, associative, idempotent join function; operation-based CRDTs ship individual operations that must themselves be commutative and be delivered with causal ordering. A grow-only counter (G-Counter) and an observed-remove set (OR-Set) are the two simplest concrete examples of this guarantee in action.

Why convergence is guaranteed

Convergence works because the merge operation is commutative (order doesn't matter), associative (grouping doesn't matter), and idempotent (merging a state with itself changes nothing), so applying merges in any order, any number of times, produces the same final state. Formally, this kind of merge is called a join, and a replica's state is modeled as an element of a join-semilattice: a partially ordered set where the join always computes the least upper bound of two states. That is the actual property behind convergence without coordination: it is not that conflicts never happen, it is that the merge function is defined so a conflict has exactly one well-defined resolution no matter how or when it gets computed.

G-Counter (grow-only counter)

• State: a vector with one non-negative integer slot per replica, c[i].
• Local update: a replica only ever increments its own slot.
• Merge: element-wise maximum across the two vectors.

$c'[i] = \max(c_1[i], c_2[i])$

• Read: sum across all slots.

$\text{total} = \sum_{i} c[i]$

• Because each slot only ever grows for its own replica, taking the max per slot can never lose an increment either side already recorded.

OR-Set (observed-remove set)

• State: a set of (element, unique tag) pairs, split into an add-set and a remove-set of tags.
• Add(e): mint a fresh tag, insert (e, tag) into the add-set.
• Remove(e): copy every tag currently observed for e into the remove-set; it removes only tags this replica has actually seen, never tags added elsewhere that haven't arrived yet.
• Merge: union the add-sets, union the remove-sets.
• An element counts as present if it has at least one tag in the add-set that is not in the remove-set.

Naming the comparators explicitly

Worked example: G-Counter convergence

Three replicas A, B, C start at (0,0,0):

• Replica A processes 2 local increments: its state becomes (2,0,0).
• Replica B processes 3 local increments, concurrently, before hearing from A: (0,3,0).
• Replica C stays idle: (0,0,0).

A and B exchange state and merge (element-wise max): merge((2,0,0),(0,3,0)) = (2,3,0). Read = 2+3+0 = 5. C later merges with that result: merge((0,0,0),(2,3,0)) = (2,3,0). Read = 5. Whichever order the three replicas merge in, the final vector is (2,3,0) and the read is 5, exactly matching the 2+3=5 real increments actually performed. No increment is lost and none is double-counted.

Worked example: OR-Set add and remove race

Replicas R1 and R2 have already converged on a set containing 'milk' with tag t1. The two replicas are then partitioned from each other:

• R1's user removes 'milk': remove-set gains {t1}, the only tag R1 has ever observed for 'milk'.
• R2's user, unaware of the removal, re-adds 'milk': add-set gains a brand-new tag {t2}, so the add-set is now {t1, t2}.

On merge: add-set = {t1, t2} (union), remove-set = {t1} (union). 'milk' is present because t2 is in the add-set and not in the remove-set. This is the correct outcome: R2's re-add introduced a tag the remover never saw, so it survives, exactly the observed-remove semantics the name describes.

Trade-offs & pitfalls

• Storage and bandwidth: every element needs extra metadata (a vector slot per replica for counters, a unique tag per add for sets), and removed elements don't disappear until a garbage-collection pass establishes causal stability across replicas.
• The edge case that catches teams out: CRDTs don't compose across non-commutative operations. A G-Counter or OR-Set is safe because the operations that define it (increment, tagged add and remove) are commutative by construction. But if you build an append-only log CRDT and then bolt on an application-level 'delete the last 3 entries' operation defined by position, that composition is not well-defined under concurrency: 'last 3' means something different on each replica depending on how many entries have been concurrently appended there at the time the delete runs, so two replicas can end up deleting different entries even though each individually applied a correct-looking CRDT merge. The fix is the principle OR-Set already uses: target deletions by a stable element identifier, never by position or count.
• When to avoid: anywhere a global invariant spans multiple keys (uniqueness, a balance that must never go negative), or the business logic genuinely isn't commutative. CRDTs solve convergence, not arbitrary correctness.