Staff-Level Site Reliability Engineer Interview Preparation Guide for Amazon

Requirements & assumptions (clarify up-front):

• 1,000,000 reads/sec, 200,000 writes/sec steady-state.
• Replication factor RF=3, synchronous replicate to 2 followers.
• Average item size = 1 KB payload + 200B metadata = 1.2 KB.
• Read: mostly single-key point reads; 95% hits in cache assumed possible.
• Use SSD NVMe; nodes homogeneous.

Capacity math:

• Raw request bandwidth (payload only): (1,000,000+200,000) * 1.2KB ≈ 1.44 GB/s inbound+outbound combined.
• Replication network: each write must be sent to RF-1 = 2 replicas => write replication traffic = 200k 1.2KB2 ≈ 480 MB/s.
• Read traffic served from nodes: assume 50% reads served from replicas; worst-case network egress ≈ 1,000,000 *1.2KB ≈ 1.2 GB/s. Plan for 2–3 GB/s total egress across cluster.

Node sizing (example baseline):

• Network: 100 Gbps per node (~12.5 GB/s) gives headroom for spikes; can host multiple rack placements. With 100 Gbps, cluster-level egress demand (≈1.2–3 GB/s) divided across N nodes.
• CPU: assume ~200–400 CPU cores total needed for request processing, compaction and GC. Per node: 32–64 vCPUs (continuous low-latency processing + background threads).
• Memory: allocate enough cache + JVM heap (if Java store). Example: 256–512 GB RAM per node (256 GB heap reserved carefully; leave host memory for page cache).
• Disk/IOPS: writes = 200k/sec (durable writes). With 1.2KB per write => 240 MB/s sequential write plus compaction rewrite amplification (x2–3) -> plan 600 MB/s sustained per cluster. Per node NVMe: 3–6 GB/s and 300k IOPS (4KB) comfortable. Use many drives; target <70% utilization.

Accounting background ops:

• Compaction/repairs: model write amplification factor (WAF). If WAF=2.5, effective write bandwidth = 200k1.2KB2.5 = 600 MB/s. Provision IOPS/throughput for that plus headroom. Schedule major repairs/anti-entropy during low-traffic windows or throttle via token bucket; prioritize read/write latency by throttling compaction when percentiles exceed SLO.
• GC: minimize stop-the-world pauses: run with G1/ZGC or native runtimes. Keep heap per JVM modest (≤32–64GB) and use many service threads across containers to limit GC impact. Monitor and set alerts for GC pause tail.
• Background I/O isolation: use cgroups/blkio or QoS on disk to reserve IOPS for foreground operations.

Planning for read tail latency spikes:

• Multi-tier cache: use local node memory cache + global edge caches (CDN/redis) to absorb spikes; aim to serve 80–95% reads from cache.
• Client-side retries with jitter and hedged reads: issue secondary read after 5–10ms to another replica for high-percentile mitigation.
• Prioritized scheduling: prioritize small reads over compaction I/O; use io_uring or native async IO.
• Autoscaling: add nodes when request rate sustained; use rolling restarts to avoid correlated GC/compaction.

Replication bandwidth & durability:

• With RF=3 plan for ~480 MB/s replication + overhead (protocol, TLS) -> provision ~600 MB/s margin.
• For cluster elasticity, ensure leaderless or configurable quorum to avoid single-node bottlenecks. Use batching for replication (group writes into 10–20KB batches) to reduce per-write overhead.

Operational practices:

• SLOs: set read/write p99/p999 targets; monitor CPU, GC, disk latency, IOPS, network.
• Load testing: synthetic load that exercises compaction and repair simultaneously to validate tail behavior.
• Capacity formula summary: provision per-node network/IO such that (cluster throughput)/(nodes) ≤ 50% of a node’s network/disk capacity to allow headroom for background ops and failure scenarios.

Example final sizing (rough):

• Nodes: 50 nodes, each: 100 Gbps NIC, 48 vCPUs, 384 GB RAM, NVMe 8TB (3–5 GB/s, 300k IOPS). This yields cluster headroom for replication, compaction, failover — adjust after benchmarking with real data and WAF.

Approach: For execs and account managers, keep language business-focused, quantify customer impact, state owned actions and mitigations, and commit to a clear next update/ETA. Avoid technical detail unless it affects customer-facing timelines.

Three-bullet executive update template:

• Impact: [What customers/business are experiencing, scope, and numbers] — e.g., "Customers in EU region experiencing intermittent payment failures; ~12% of transactions affected, impacting ~8k users and 3 top-tier accounts."
• Action: [What we’re doing and who’s owning it; immediate mitigations] — e.g., "Engineering (on-call SRE + payments team) is running a targeted rollback and routing payments to backup processor; rate-limiting applied to reduce error volume. Customer support has escalation playbooks for affected accounts."
• Next update / ETA: [When and what will be in the next update; if unknown, give cadence] — e.g., "Next update in 30 minutes or sooner if the rollback completes; ETA for recovery ~45–60 minutes. If unresolved, we'll provide a mitigation timeline and customer communication plan."

Concise example message you can send:

• Impact: "Intermittent payment failures for EU customers — ~12% of transactions failing; 3 premium customers impacted."
• Action: "On-call SRE and payments team performing rollback to last stable release; routing to backup processor and applying rate limits; CS notified with escalation steps."
• Next update / ETA: "Next update in 30 minutes. Estimated recovery 45–60 minutes; will include recovery status and customer communication wording."

Profiling in SRE is the systematic measurement of where a system spends resources (CPU, memory, I/O, network, lock contention) so you can target performance and reliability improvements with evidence. Main goals:

• Identify hotspots and bottlenecks (hot code paths, memory leaks, heavy syscalls)
• Quantify impact (cost, latency, error amplification)
• Reproduce and validate optimizations safely
• Guide capacity planning and SLO decisions

Common profiler types:

• Sampling profilers: periodically sample program state (stack traces) at intervals. Low overhead, good for CPU and latency hotspots; produces statistical view.
• Instrumentation profilers: insert timers/counters around functions or events. Higher fidelity and overhead; useful for precise timing, allocations, and custom metrics.

Typical metrics collected:

• CPU: CPU usage by process/thread, per-function CPU time
• Memory: heap size, allocation rate, GC pauses, object counts
• I/O: syscall counts, read/write bytes, latency distributions
• Network: bytes/sec, packet rates, per-call latency, retransmits
• Concurrency: thread counts, lock contention, wait times

Measure-first, change-later rationale:

• Prevents “optimizing the wrong thing” by grounding changes in data
• Ensures reproducibility: record baseline metrics and profiles, apply change, compare
• Keeps safety: small, verifiable changes reduce regressions; use canary rollouts and benchmarks
• Provides audit trail for cost/benefit and informed postmortems

Concrete practice: capture a sampling CPU profile and allocation profile in prod (low overhead), reproduce locally or in staging, design a minimal change, run A/B/canary with the same profiling to confirm improvement before full rollout.

A safe restart uses kubectl rollout restart, which recreates pods through the normal RollingUpdate strategy rather than deleting them directly, so the same availability guarantees that protect a routine deployment protect the restart too.

Commands

Trigger and watch a rolling restart:

kubectl rollout restart deployment my-app -n prod
kubectl rollout status deployment my-app -n prod --watch

View rollout history and inspect a specific revision:

kubectl rollout history deployment my-app -n prod
kubectl rollout history deployment my-app -n prod --revision=3

Roll back:

kubectl rollout undo deployment my-app -n prod --to-revision=3
kubectl rollout status deployment my-app -n prod

Ship an image change with a recorded reason (the --record flag some older references use for this is deprecated; annotate explicitly instead):

kubectl set image deployment/my-app my-app=registry/app:1.2.3 -n prod
kubectl annotate deployment my-app kubernetes.io/change-cause="bump to 1.2.3, ticket OPS-441" --overwrite -n prod

What keeps a restart from becoming a cascading failure

• RollingUpdate parameters: maxUnavailable and maxSurge (both default to 25% of desired replicas) bound how many old pods can be down and how many extra new pods can exist at once. For a consumer-facing service, a conservative setting (for example maxUnavailable: 0, maxSurge: 1) never drops capacity below the current replica count during the restart, at the cost of briefly running more pods than the steady-state count.
• Readiness probes: a Service only sends traffic to pods that pass their readiness probe, so a newly restarted pod that's still initializing doesn't receive requests it can't yet handle. This is the single biggest lever against a restart-induced error spike; without a readiness probe, the rollout has no signal that a "new" pod is actually ready and can start routing traffic to it immediately.
• PodDisruptionBudget (PDB): guarantees a minimum number (or percentage) of replicas stay available throughout the restart, independent of the Deployment's own maxUnavailable setting, which matters when other voluntary disruptions (a node drain, a cluster upgrade) happen to overlap with the restart window.
• Graceful shutdown: a preStop hook plus a terminationGracePeriodSeconds long enough for in-flight requests to finish, combined with the Service removing the pod's endpoint before the container actually stops, avoids dropping requests that were already in progress when the restart began.
• Staged rollout for risk-sensitive services: restarting (or deploying) to a small subset first, watching error rate and latency, then proceeding, catches a bad new revision before it reaches full traffic; this is a general staged-rollout practice, not a specific traffic-splitting mechanism (traffic-splitting techniques like weighted canary routing are a load-balancing/ingress-layer concern, not something the Deployment object itself provides).

Trade-offs and pitfalls

• kubectl rollout restart only recreates pods; it does not change the Deployment's spec, so rollout history records it as a new revision with the same template, which is easy to forget when later trying to undo your way back past a restart that changed nothing.
• Setting maxUnavailable: 0 guarantees no capacity loss but requires enough spare cluster capacity for maxSurge extra pods to schedule; on a tightly packed cluster this can leave the rollout stuck Pending on the surge pods instead of proceeding.
• A rollback only restores the pod template (image, env, resource requests, and so on). If the Deployment reads a ConfigMap or Secret by a fixed name and that ConfigMap was edited in place rather than replaced with a new name or hash-suffixed name, rolling the Deployment back does not restore the old configuration content, only the old pod template pointing at the same (already-mutated) ConfigMap. This is the most common way a rollback fails to actually roll back.
• The same gap applies to a PersistentVolumeClaim (PVC): a Deployment's rollback restores the pod template's volume mount references, not the data on the volume itself. If the new version wrote a schema migration or otherwise mutated data in place on that volume, rolling the Deployment back gives you the old code pointing at already-changed data, not the old data. Anything stateful needs its own restore path (a volume snapshot or application-level backup) alongside the Deployment rollback, not instead of thinking about it separately.

Direct answer

A highly extensible, plugin-based CI architecture maximizes what any individual team can do with the platform, at the cost of a larger, harder-to-secure surface area and more operational burden maintaining that flexibility; a simple, opinionated set of pipeline templates trades some flexibility for lower operational burden, a smaller security surface, and much easier support at scale across many teams.

Structured elaboration

Developer velocity. A plugin-based architecture lets a team with an unusual need (an exotic build tool, a niche deployment target) solve it themselves without waiting on the platform team, which is a real velocity win for teams whose needs genuinely fall outside the common case. An opinionated template set is faster for the common case (most teams never need anything beyond the template) but can genuinely block or badly slow down the uncommon case, where a team's real need isn't expressible within the template's assumptions.

Operational burden. Every plugin (or every degree of freedom in a highly extensible system) is something the platform team has to account for when reasoning about the system's behavior, security posture, and upgrade path; a plugin ecosystem's collective maintenance burden (as covered concretely in the Jenkins-at-scale question) grows with the number of distinct plugins in active use across the organization, not just with the platform's own complexity. An opinionated, templated system has a much smaller surface: the platform team supports a small, known set of patterns, not an open-ended combinatorial space of whatever any team has chosen to install or configure.

Security surface area. Every plugin or extension point is a potential vulnerability, and a highly extensible system's actual security posture depends on the weakest plugin any team has installed, not just on the core platform's own security; an opinionated system's security review can focus on a small, fixed set of supported patterns instead of an open-ended, ever-changing set of third-party extensions.

Upgrade and testing complexity. A highly extensible system has to validate that platform upgrades don't break the potentially huge combinatorial space of plugin combinations teams have actually deployed, which is close to intractable to fully test; an opinionated system's upgrades only need to be validated against its own small set of supported templates, which is genuinely tractable.

Scaling support across hundreds of teams. An opinionated system scales support effort roughly with the number of distinct template patterns (small and roughly constant as team count grows), while a highly extensible one scales support effort closer to the number of distinct plugin/extension combinations in actual use (which tends to grow at least linearly, and often faster, with team count), making it a much harder support model to sustain as the organization grows.

Worked example

A 20-person startup benefits from a highly extensible, plugin-based system: few enough teams that the platform team can reasonably track what's installed, and the flexibility lets early, fast-moving teams solve their own unusual needs without a platform bottleneck. A 2,000-engineer enterprise with hundreds of teams is much better served by a small, opinionated set of well-supported templates with a genuinely rare, carefully-reviewed exception path for the handful of teams whose needs don't fit, because the alternative (letting hundreds of teams each install whatever plugins they want) produces an untestable, unsupportable combinatorial mess, as the earlier Jenkins-plugin-governance discussion covers concretely.

Trade-offs and pitfalls

The most common mistake is choosing maximal extensibility by default without accounting for how the operational and security burden scales with organization size, which works fine at small scale and becomes genuinely unsustainable well before an organization reaches hundreds of teams. The second is choosing a rigidly opinionated system with no legitimate exception path at all, which either blocks teams with genuinely different needs or, more likely, drives them to quietly work around the platform entirely, which is worse for consistency than a small, deliberate, reviewed set of exceptions.

Requirements & constraints:

• Greenfield product, customer-facing, must detect regressions early, low MTTR, minimal initial instrumentation cost, iterate over 90 days.
• Target users: product + engineering; integrate with existing tooling (Prometheus/Grafana, Datadog, or equivalent).

Day 0–7: Instrumentation & SLIs

• SLIs (customer-centric + system metrics):
  • Availability: successful user requests / total user requests (HTTP 2xx/3xx within timeout) per minute.
  • Latency P95/P99 for key user journeys (API call, page load).
  • Error rate: 5xx rate and client-visible functional errors (business errors).
  • Throughput: requests/sec per endpoint.
  • Queue/backlog depth and processing latency for async jobs.
  • Key resource health: CPU, memory, disk, DB connection pool saturation.
• Implement lightweight tracing hooks (span IDs) and structured logs.

Day 7–21: Minimum dashboards

• Overview dashboard (ops): Availability, Error Rate, Latency P95/P99, Throughput, Error budget burn rate.
• Endpoint health dashboard: per-critical-endpoint requests, errors, latency heatmap.
• Infrastructure dashboard: host/cluster CPU, memory, disk, network, DB connections, queue depth.
• Deployment/CICD dashboard: last deploys, deploy frequency, canary metrics.
• Alert inbox dashboard: active alerts and owner.

Day 14–30: Alerting policy (noise-minimizing)

• Alert tiers:
  • P0 (immediate page): Availability < 99.5% over 5m OR error-rate spike (>5% absolute) for core API affecting customers.
  • P1 (pager for on-call): Latency P99 > 2x baseline for 10m or error budget burn rate > 5%/hour.
  • P2 (email/Slack): Resource saturation nearing threshold (DB conn > 80% for 5m), queue backlog > threshold, deploy failure rates.
  • P3 (ticket): Non-urgent regressions, test failures.
• Alert rules include runbooks, suggested mitigations, owner rotation, and automatic deduplication & suppression during known deploy windows.
• Use burn-rate alerts to abort rollouts automatically if canaries fail.

SLO targets (initial, revise after 90 days)

• Availability SLO: 99.95% monthly for core user flows (error budget 22 mins/month).
• Latency SLO: P95 < 300ms, P99 < 1.5s for API.
• Error-rate SLO: <0.1% customer-visible errors.
• Infrastructure SLOs: DB connection saturation < 70%.

First 90-day feedback loop

• Weeklies: Review dashboards + incidents with product and engineers; prioritize missing coverage.
• After every deploy: Canary review meeting for first 2 weeks of feature rollouts.
• Bi-weekly SLO review: compare burn rates, adjust thresholds, add/remove SLIs based on false positives/negatives.
• Runbooks & playbooks: iterate after each incident (blameless postmortem within 48h); update runbooks within 72h.
• Product collaboration: map user journeys -> tag SLIs to personas; product validates which journeys are critical.
• Month 1: Validate telemetry completeness (sampling rates, missing spans).
• Month 2: Add business metrics (conversion, sign-ups, checkout success) as SLIs.
• Month 3: Harden: automate remediation for common failures, add synthetic tests, formalize SLO review cadence and error budget policy (release freezes when budget exhausted).

Trade-offs & rationale

• Start with high-signal customer-facing SLIs to detect regressions early; avoid over-instrumentation.
• Conservative alerting to reduce pager fatigue; rely on burn-rate + canaries to protect customers during deploys.
• Iterate with product using data — SLOs are living contracts that balance velocity and reliability.

Direct answer

Two things can be happening. If the backend (where Terraform's state file actually lives and gets locked) enforces state locking (S3 with a DynamoDB lock table, GCS, Terraform Cloud, Consul), the second apply should be rejected outright with a lock-acquisition error, so genuine concurrent-write corruption almost always means locking isn't actually being enforced: no lock table configured, someone ran apply with -lock=false, or someone force-unlocked while the first apply was still in flight. Separately, even a single apply is not transactional: Terraform applies resources one by one in dependency order, so a mid-apply failure (an API timeout, throttling, a killed CI job) leaves some resources created and others not, independent of whether a second run was involved at all. The fix is to make concurrency structurally impossible (backend locking plus a CI-level mutex) and to always apply a saved plan rather than a freshly generated one.

How this happens and how to prevent it

Why concurrent applies corrupt state

Both runs compute a plan against the same starting state. If locking isn't enforced, the second run's plan doesn't know the first run already changed things; when both finish, the state write from whichever run finishes last silently overwrites the state written by the other, discarding its record even though the real infrastructure it created still exists.

Prevention

• Backend locking: an S3 backend with a DynamoDB lock table (or GCS/Terraform Cloud's built-in locking) makes Terraform itself refuse to run a second apply against the same state while a first one holds the lock.
• CI-level mutex: backend locking only protects the moment apply runs, not the whole review window. Two engineers can each get an approved plan and then race to click apply. A CI concurrency gate (GitHub Actions concurrency: group per workspace, GitLab resource_group, or an explicit lock service) serializes the pipeline jobs themselves.
• Plan-then-apply-exact-plan: run terraform plan -out=tfplan, review that artifact, then apply with terraform apply tfplan rather than a bare terraform apply. If the real state has moved since the plan was generated, Terraform detects the mismatch and refuses to apply the stale plan, instead of silently recomputing a new one.

Worked example

Suppose a plan touches six resources: a security group, two subnets, a NAT gateway, a route table, and a route table association. If locking is enforced, a second engineer's terraform apply at the same time fails immediately with an error acquiring the state lock, that's the safe, expected outcome, not corruption. Corruption happens when locking is missing or bypassed: engineer A's apply reaches the NAT gateway create step while engineer B's unlocked run, computed from the same pre-apply state, doesn't know a NAT gateway is already being created and also attempts to create one. The result is two NAT gateways in the account, and depending on which run's state write lands last, the state file ends up tracking only one of them, the other becomes an orphaned, unmanaged, billable resource that terraform plan won't even show anymore because nothing in state or config references it.

Trade-offs and pitfalls

• Locking prevents corruption but a genuinely stuck lock (a CI job killed mid-apply, a crashed laptop run) blocks every future apply until someone force-unlocks. Force-unlock is a manual, risky escape hatch, use it only after confirming no apply is truly still in flight (check the CI job status and the cloud provider's own activity log, not just the lock table).
• A CI-level mutex is necessary in addition to backend locking specifically to close the review-to-apply race window; relying on backend locking alone still lets two approved plans collide at the apply step.
• Applying a stale saved plan fails safely, Terraform detects the plan no longer matches current state, but only if the pipeline is disciplined about always applying the plan artifact rather than falling back to a bare terraform apply that silently regenerates a fresh plan against whatever the current state happens to be.

Direct answer. A 504 specifically means the gateway or load balancer gave up waiting on an upstream response, so the investigation should center on finding which hop in the chain is actually slow or unresponsive, not on the gateway itself, which is usually just the messenger.

Structured elaboration.

1. Confirm where the timeout is actually happening. Check the gateway or load balancer's own timeout configuration and logs: a 504 tells you the gateway gave up, but not which upstream it gave up on. If you have distributed tracing, pull a handful of the failing requests and look at which span never completes or takes far longer than normal.
2. Check load and concurrency on the microservice itself. Look at request rate, thread-pool or connection-pool saturation, and queue depth. A service that's fine at low traffic can start timing out purely from being overwhelmed, well before it would show as unhealthy on CPU or memory.
3. Check its own downstream calls. If the microservice calls a database, cache, or another service, check THOSE for elevated latency; the microservice may be blocked waiting on something further down the chain rather than being slow itself.
4. Review the code for the specific bugs that turn a slow dependency into a hung request. Three areas are worth checking directly: whether the client for each downstream call has an explicit timeout set at all (a client with no timeout can hang indefinitely, which is a different and worse bug than one with a merely generous timeout); whether the connection or thread-pool size is hardcoded or read from a config value that may not have been updated as traffic grew; and whether there's a retry-without-backoff loop anywhere in the request path that would amplify load on an already-slow dependency instead of failing fast.
5. Reproduce under load if you can. A quick load test against a staging or canary instance, mimicking the traffic pattern that triggers the 504s, can confirm a capacity-related hypothesis in minutes rather than guessing from production alone.
6. Mitigate. Immediate options include raising the gateway timeout slightly if the upstream is close to finishing anyway (a stopgap, not a fix), adding capacity if it's a load problem, or failing fast with a clear error instead of hanging if the root cause is a stuck downstream dependency.

Worked example. Traces show requests hanging in a call from the microservice to its database, with the query itself sometimes taking 8 to 10 seconds under load versus under 200ms normally. Checking the database's own connection pool shows it's fully saturated during the incident window. That converges on a specific, testable hypothesis: the service's connection pool to the database is too small for current traffic, requests queue up waiting for a connection, and eventually the gateway's shorter timeout fires before the query even starts. The fix is raising the pool size (or adding a circuit breaker so requests fail fast instead of queuing indefinitely) rather than anything at the gateway layer.

Trade-offs and pitfalls. A common mistake is treating the 504 itself as the problem and only tuning the gateway timeout, which can mask a real capacity or dependency issue and just delays the same failure. The other pitfall is not distinguishing 'slow' from 'stuck': a slow downstream call that eventually returns is a capacity problem you can often scale your way out of, but a stuck call that never returns (a connection that hangs instead of timing out) needs a timeout and circuit breaker at every hop, or one hung dependency can eventually exhaust every caller's resources.

Approach (brief):

• Use Consul KV CAS (ModifyIndex) plus a short-lived Consul session lock to minimize races. Read current value and ModifyIndex, acquire a session lock on the key (or a companion lock key), perform a CAS write that includes metadata (who, when, change-id) appended to the value or stored in a companion key, run post-update health checks, and if checks fail, CAS the previous value back (using the ModifyIndex observed when we read it). If rollback CAS fails because somebody else changed the key, escalate/alert (automated retry window possible) — never silently overwrite someone else’s change.

Sample implementation (Python, using requests):

import requests, json, time, uuid, os
from datetime import datetime

CONSUL = os.getenv("CONSUL_ADDR", "http://127.0.0.1:8500")
KV_KEY = "service/config/mykey"
LOCK_KEY = KV_KEY + ".lock"
SESSION_TTL = "15s"
HEALTH_CHECK_URL = "http://localhost:8080/health"
HEADERS = {"Content-Type": "application/json"}

def consul_put(path, params=None, data=None):
    url = f"{CONSUL}/v1/{path}"
    return requests.put(url, params=params, data=data)

def consul_get(path, params=None):
    url = f"{CONSUL}/v1/{path}"
    return requests.get(url, params=params)

def create_session(name):
    payload = {"Name": name, "TTL": SESSION_TTL, "Behavior": "delete"}
    r = requests.put(f"{CONSUL}/v1/session/create", json=payload)
    r.raise_for_status()
    return r.json()["ID"]

def destroy_session(sid):
    requests.put(f"{CONSUL}/v1/session/destroy/{sid}")

def acquire_lock(key, sid):
    r = consul_put(f"kv/{key}", params={"acquire": sid}, data="")
    return r.text == "true"

def release_lock(key, sid):
    consul_put(f"kv/{key}", params={"release": sid}, data="")

def get_kv(key):
    r = consul_get(f"kv/{key}")
    if r.status_code == 200:
        obj = r.json()[0]
        val = obj.get("Value")
        if val:
            val = base64_decode(val)
        return obj["ModifyIndex"], val, obj
    return 0, None, None

def base64_decode(s):
    import base64
    return base64.b64decode(s).decode()

def base64_encode(s):
    import base64
    return base64.b64encode(s.encode()).decode()

def health_check():
    try:
        r = requests.get(HEALTH_CHECK_URL, timeout=3)
        return r.status_code == 200
    except Exception:
        return False

def cas_write(key, value, cas_index):
    # write value with cas=ModifyIndex
    data = value
    r = consul_put(f"kv/{key}", params={"cas": cas_index}, data=data)
    return r.text == "true"

def safe_update(new_value, who):
    change_id = str(uuid.uuid4())
    ts = datetime.utcnow().isoformat() + "Z"
    meta = {"who": who, "when": ts, "change_id": change_id}
    payload = json.dumps({"value": new_value, "meta": meta})
    sid = create_session("cfg-updater-"+change_id)
    try:
        if not acquire_lock(LOCK_KEY, sid):
            raise RuntimeError("Could not acquire lock; another operator active")

        old_index, old_val, _ = get_kv(KV_KEY)
        # attempt CAS with observed ModifyIndex
        success = cas_write(KV_KEY, payload, old_index)
        if not success:
            raise RuntimeError("CAS failed: concurrent update detected")

        # short stabilization delay, then run health checks
        time.sleep(1)
        if health_check():
            print("Update successful and healthy")
            return True
        else:
            print("Health checks failed; attempting rollback")
            rb_success = cas_write(KV_KEY, json.dumps({"value": old_val}), old_index+1 if old_index else old_index)
            #Note: we try to CAS to previous index; if someone else changed it meantime, this may fail
            if rb_success:
                print("Rollback successful")
                return False
            else:
                # final attempt: if CAS failed, try to write an error marker and alert humans
                print("Rollback failed due to concurrent changes. Alerting and leaving state for manual resolution.")
                return False
    finally:
        release_lock(LOCK_KEY, sid)
        destroy_session(sid)

# Example usage:
# safe_update({"threshold": 10}, who="sre-jenkins")

Key concepts and reasoning:

• Consul KV CAS uses ModifyIndex to ensure you only write if the key’s state is unchanged. That gives atomicity for the swap attempt.
• A short-lived session lock reduces races by preventing concurrent updaters from attempting CAS at the same time; sessions give a simple mutex-like mechanism.
• Store metadata together with the value (or in a companion key) so every version has who/when/change-id. Using a structured JSON value keeps metadata queryable.
• Health checks run after the CAS—if they fail, rollback uses CAS to reapply the old value only if the key hasn’t changed since our original read. If somebody else changed it during our window, automatic rollback could overwrite someone else’s valid change—so detect that and escalate.

Trade-offs and robustness:

• Stronger guarantee: use Consul transactions (KVTxn) to write value and metadata atomically and use a lock to prevent concurrent operators.
• For bulletproof rollback, implement a change-log (append-only) key storing versions so you can always inspect history and restore safely rather than blindly CASing older index.
• Add retries with exponential backoff for transient failures, alerting when rollback cannot proceed automatically.

Edge cases:

• CAS fails because of concurrent changes — abort and retry or escalate.
• Health check false positives/negatives — run multiple probes and longer stabilization window before rollback.
• Network partitions between operator and Consul — avoid making decisions without quorum visibility; include safeties (manual approval) if uncertain.

Complexity:

• Operational complexity is bounded; Consul calls are O(1) per operation. The main concerns are correctness under concurrency and flaky health checks rather than algorithmic complexity.

Direct Answer

Split the fleet into a fixed agent tier (one per host, DaemonSet-deployed, doing only light initial processing) and a horizontally autoscaled gateway tier that does the heavier work (sampling, enrichment, batching) and fronts the actual export. Put a durable buffer between the gateway and the exporter so a backend outage fills the buffer instead of blocking or dropping at the gateway, and scale the gateway tier on real backpressure signals (queue depth, CPU) rather than a fixed replica count.

Structured Elaboration

Fleet topology

flowchart LR
  AGENT["Per-Host Agents"] --> GW["Gateway Tier: HPA-scaled"]
  GW --> PROC["Processor Chain: memory_limiter, enrich, sample, batch"]
  PROC --> BUF[("Durable Buffer")]
  BUF --> EXP["Exporter"]
  EXP --> BACKEND[("Backend")]
  BUF -.->|"queue depth metric"| HPA{"Autoscaler"}
  HPA -.->|"scale replicas"| GW

Agent versus gateway split
Agents run one per host, receiving OTLP from local processes and forwarding onward with minimal processing, so their resource footprint per host stays flat and predictable regardless of fleet-wide load. The gateway tier does the load-dependent work (see below), which is exactly what needs to scale with traffic, so isolating it from the fixed agent tier is what makes autoscaling meaningful.

Horizontal autoscaling
Scale the gateway tier on a metric that reflects real backpressure, most reliably queue depth in front of the exporter (or CPU as a proxy if queue depth isn't exposed as a scalable metric), using a standard proportional scaling rule: desired replica count scales with how far the current metric is above target.

Durable buffer in front of the exporters
A persistent queue (backed by local disk or an external durable log like Kafka) sits between the processor chain and the exporter. During a backend outage, the gateway keeps accepting and processing data, writing into the buffer instead of failing the export, and drains the buffer once the backend recovers.

Processor chain structure
Order matters: memory_limiter first (shed load before spending CPU on anything else), then resource and attribute enrichment, then sampling (tail-based sampling needs to see whole traces, so it runs before batching, not after), then batch last, right before the exporter, so batches only ever contain data that already survived the sampling decision.

Worked Example

Autoscaling thresholds. Assume each gateway pod sustains 20,000 spans/sec (a stated sizing assumption reflecting the processor chain's per-pod cost), and incoming load ranges from a floor of 50,000 spans/sec to a peak of 400,000 spans/sec through the day.

Applying the standard horizontal-scaling proportional rule, desiredReplicas = ceil(currentReplicas x currentMetricValue / desiredMetricValue), for a concrete example at 5 current replicas running at 90% CPU against a 70% target:

Durable buffer sizing. Assume the gateway continues accepting spans at 100,000/sec during a backend outage (average 800 bytes/span post-enrichment, a stated design input), and the outage is expected to last up to 15 minutes (900 s) before either recovery or a paging escalation forces a decision:

A roughly 72 GB buffer requirement is well within the local disk capacity of a typical commodity broker or persistent-volume-backed queue, so this is a provisioning number to size the buffer's storage class against, not a hard architectural constraint.

Trade-offs and Pitfalls

Scaling on CPU alone is a lagging proxy: CPU can look fine even while the queue in front of the exporter is growing, if the bottleneck is actually export throughput to a slow backend rather than processing throughput. Scaling on queue depth directly is a better signal when it's available, since it reflects the thing you actually care about (are we falling behind) rather than a correlated but imperfect stand-in.

A durable buffer removes the immediate pressure to fix an outage fast, which is good for availability but can mask a slow backend degradation if nobody is watching buffer fill rate: a buffer that's growing steadily but not yet full looks the same as a healthy system on a dashboard that only shows "buffer not full yet," unless fill rate itself is alerted on.

Running sampling before batching (rather than after) is correct for tail-based decisions that need whole traces, but it means the sampling stage has to hold and correlate spans across a trace, which is real memory pressure on the gateway tier specifically, on top of whatever the autoscaling math above accounts for from raw throughput alone. Size gateway pod memory against expected in-flight trace count, not just span throughput.