Comprehensive Interview Preparation Guide: Site Reliability Engineer (Senior Level) at Airbnb

Direct answer

An automated rollback policy needs to watch both infrastructure SLOs and model-quality SLOs as independent triggers (either one alone should be able to fire a rollback), with safety checks and a cooldown window preventing a rollback from firing on noise or flapping repeatedly.

Structured elaboration

1. Dual trigger conditions: an infra-SLO breach (latency, error rate exceeding threshold) and a model-quality-SLO breach (a drop in conversion or AUC beyond threshold) are each SUFFICIENT on their own to trigger: a rollback shouldn't require both to fire simultaneously, since a purely model-quality regression (fast, error-free, but wrong) is exactly the class of incident an infra-only trigger would miss entirely.
2. Thresholds: set each threshold using the same control-chart logic as any other alerting threshold (based on historical healthy-period variance), not an arbitrary round number, and require the threshold breach to PERSIST across a defined window (not a single reading) before triggering: this is the same anti-noise discipline as any alerting design.
3. Safety checks before executing: before an automated rollback actually fires, verify a known-good previous version genuinely exists and is ready to receive traffic (don't roll back to a state that itself has a known issue), and confirm the CURRENT metric reading isn't itself a measurement artifact (a monitoring-pipeline glitch producing a spurious reading) via a quick secondary confirmation check.
4. Cooldown windows: after an automated rollback fires, enforce a minimum cooldown before ANY further automated action (another rollback, or an automated re-promotion) is allowed: this prevents a flapping cycle where a marginal metric oscillates around the threshold and triggers repeated automated actions in quick succession.
5. Escalation and human approval: for a HIGH-severity trigger (a dramatic breach, not a marginal one), the policy can act immediately and automatically; for a borderline or ambiguous trigger, route to human approval rather than auto-executing, since a marginal signal is exactly where a false-positive automated rollback is most likely.

Worked example

Concretely: define the model-quality SLO as "conversion rate within 3 percentage points of its 30-day trailing baseline, sustained over 3 consecutive 15-minute windows" and the infra SLO as "p99 latency under 300ms, sustained over 5 minutes": either breach alone triggers a safety check (confirm a known-good previous version is ready) and then an automatic rollback, followed by a 30-minute cooldown during which no further automated rollback or promotion can fire, giving the team time to investigate without the system fighting itself.

Trade-offs & pitfalls

The tension throughout is trigger sensitivity versus trust in full automation: a policy tuned to catch every real regression fast will also occasionally fire on noise, and each false-positive automated rollback erodes confidence in the system, tempting teams to disable automation entirely after a bad experience. Starting with a HIGH-confidence-only automated trigger (only the most dramatic, unambiguous breaches auto-execute) and a human-approval path for everything else is the safer default until the policy has built up a track record of being reliably right.

Direct answer: Handling a distributed transaction that fails partway through means first correctly detecting WHICH steps actually completed (not assuming based on where the failure was reported), then running the compensating actions for exactly those completed steps in reverse order, alerting an operator if any compensation itself fails or the situation is ambiguous, and following a runbook that tells the support team exactly what to check and do rather than relying on improvisation during an incident.

Structured elaboration

Detection. The saga's persisted state (its audit trail of which steps started, completed, or are still pending) is the source of truth for what actually happened, not an assumption based on where an error was thrown. A step that "failed" from the caller's point of view (timeout, exception) might have actually succeeded server-side; before compensating, check the step's actual outcome via a status lookup where the downstream system supports one, rather than assuming failure and risking a compensation that undoes something that was never actually applied and is also currently unconfirmed.

Compensating actions, in reverse order. Once you know which steps genuinely completed, trigger their compensating actions starting from the most recently completed step and working backward, this mirrors the forward order and avoids a compensation depending on a later step's data that's already been compensated out from under it.

Operator alerts. Any partial-commit situation that reaches this point (a saga step failed mid-flight) should generate an alert, not require a human to notice it independently, with enough context (saga ID, which step failed, what compensations are being attempted) that an on-call engineer doesn't have to dig through logs from scratch to understand what's happening.

Automated reconciliation. For failure classes that are well-understood and have a known-safe automated fix (e.g. "release an inventory hold that's stuck because the downstream payment step definitively failed"), the compensating action should run automatically rather than waiting for a human, reserving human involvement for genuinely ambiguous or high-stakes cases (compensation itself failing repeatedly, or a situation the automated logic doesn't have a defined response for).

Runbook steps for the support team. A concrete, written procedure: (1) look up the saga by ID or affected customer/order, (2) check its current state and which compensations have run, (3) if a compensation itself has failed, follow the documented manual-fix procedure for that SPECIFIC step (not a generic "figure it out"), (4) confirm resolution and update the saga's state to reflect the manual intervention, (5) if this is a recurring pattern, file it for the automated-remediation backlog rather than treating every occurrence as a one-off.

Worked example. A saga reaches step 3 (charge payment) which times out. The orchestrator checks the payment gateway's status API for this transaction ID (detection, not assumption) and gets back "declined", a definitive, confirmed failure. It triggers compensation for steps 1-2 (release inventory hold, cancel the pending order) in reverse order, logs the outcome, and moves on, no human involvement needed since this is a well-understood, automatable case. In a different incident, the compensating "release inventory hold" call itself fails three times (the inventory service is having an outage), the orchestrator alerts on-call with the saga ID and the specific failing compensation, and the runbook's documented manual procedure for "inventory service unreachable during compensation" is to manually flag the hold for release once the service recovers, verified against a checklist, rather than leaving it silently stuck.

Trade-offs and pitfalls. Compensating based on an ASSUMED failure (rather than a confirmed one) when the downstream system actually succeeded is a common way this goes wrong in practice, it's worth explicitly building the "check actual status before compensating" step into the design rather than treating a timeout as automatically equivalent to a failure.

Direct answer

For a growing company with a small operations team, default to a hosted platform unless one team already has strong operational muscle for a specific piece of the stack (most often metrics, via Prometheus). The variable that actually decides this is engineer-hours available for care and feeding, not sticker price: a self-hosted stack usually undercuts hosted pricing on paper, but that gap closes or reverses once you count the SRE time spent on cluster capacity, upgrades, and retention tuning.

Decision framework

When to tip toward each:

• Hosted: team is small, time-to-value matters more than unit cost, nobody owns the observability stack as their primary job, or you need APM/anomaly detection you don't want to build yourself.
• Self-hosted: you already run Kubernetes at scale with a platform team that can treat observability as just another workload, data residency or compliance forces on-prem storage, or ingestion volume is high enough that hosted per-GB pricing becomes the single largest line item in the infra budget.
• Hybrid: a common middle path is hosted APM and log search (where turnkey correlation and UI matter most) paired with a self-hosted Prometheus for metrics you already understand and want fast, cheap, high-resolution queries on. This works because metrics are the cheapest and most mechanical piece to self-host, while tracing and log search UX is where vendors differentiate most. Re-evaluate the split as the team and ingestion volume grow.

Worked example

The crossover point between the two options can be derived, not guessed, once you write cost as a function of ingestion volume.

Let $G$ be GB/day ingested. Model hosted cost as a per-GB ingestion rate, and self-hosted cost as a fixed infra floor plus a much cheaper per-GB storage rate plus ongoing engineer-hour labor:

Using illustrative, explicitly-assumed rates (not any specific vendor's current published price, since list prices change): hosted ingestion rate $r_{hosted} = \$0.10$/GB, self-hosted object-storage rate $r_{storage} = \$0.02$/GB, fixed self-hosted compute floor $C_{infra} = \$300$/month, and $H = 4$ hours/month of engineer time at a loaded rate $R_{labor} = \$150$/hour:

Under these assumptions, below roughly 375 GB/day of ingestion, hosted comes out cheaper on pure dollar terms even before counting the small-team operational risk. Above it, self-hosting's lower per-GB rate starts to outweigh the fixed compute and labor floor. The point isn't the exact number, it's that a candidate should reason in terms of where the cost curves cross, not assert a universal winner.

Trade-offs and pitfalls

• Sunk-cost fallacy: don't keep self-hosting because it was already built, once engineer time becomes scarce elsewhere the calculus can flip.
• Hidden costs of hosted: egress and API costs, per-seat pricing for dashboard users, and price increases once you're locked in and migration feels expensive.
• Hidden costs of self-hosted: the observability stack becomes a second production system that itself needs monitoring, on-call, and capacity planning. If it goes down during an incident, you're debugging blind.
• Migration cost is asymmetric: moving off a hosted platform later means rebuilding dashboards and alerts in a new query language; moving off self-hosted OSS is comparatively easier since OpenTelemetry-based data is portable to almost any backend.
• Common wrong turn: picking self-hosted purely because a spreadsheet says it's cheaper while ignoring engineer-hours, then rediscovering the true cost six months later at the first major version upgrade.

Direct answer
I treat a company's published culture and values as a claim to be tested, not a fact to accept, and I look for evidence in three places: how people describe real, specific incidents (not slogans) when I ask about them, whether the org's actual structures and incentives would make the stated behavior easy or hard to practice, and whether the story is consistent across different people I talk to in the process.

Structured elaboration

1. Ask for a specific recent incident, not a description of the value. A question like "tell me about a time the team had to choose between shipping fast and following the documented review process" forces a real story; a question like "how would you describe the engineering culture here" invites a rehearsed, values-page-adjacent answer that tells you little.
2. Check whether the org's structure actually supports the stated value, independent of what anyone says. If a company claims to value psychological safety but every interviewer you meet is visibly guarded about naming any team problem, or if a company claims strong autonomy but every technical decision in the loop turns out to require a director's sign-off, the structural evidence contradicts the claim regardless of the wording used to describe it.
3. Triangulate across multiple people, ideally at different levels and tenures. A single enthusiastic interviewer proves little; a hiring manager, a peer-level engineer, and someone from a different function independently describing the same specific behavior (not the same slogan) is much stronger evidence.
4. Ask what the company would do differently if it stopped believing the value, and watch for a concrete, structural answer versus a vague one. People who work inside a genuinely lived value can usually name a real trade-off it costs them; people describing marketing usually cannot.
5. Treat your own discomfort as data. If a described norm (pace, feedback directness, decision-making style) makes you visibly uneasy during the process itself, that is a more reliable signal about fit than anything printed on the careers page, because it is your own live reaction rather than a claim you are being asked to evaluate secondhand.

Worked example
Suppose a company's careers page says it "empowers engineers with high autonomy." During the loop, ask the hiring manager for a specific recent example: "Tell me about the last time an engineer on this team made a production architecture decision without it going through a review committee first." A genuine, lived-autonomy answer sounds like: "Last quarter one of our engineers decided independently to switch a service from synchronous to async processing after noticing latency complaints; she looped in two people for a sanity check, shipped it, and reported the outcome in the next team sync." A marketing-only answer sounds like: "We really believe in empowering our engineers," repeated with no specific incident when pressed twice. If a peer engineer you speak to separately can also describe a comparable specific incident in their own words, that consistency is strong corroborating evidence; if the hiring manager's story turns out to be the ONLY example anyone can produce company-wide, that is itself informative about how common the behavior actually is.

Trade-offs & pitfalls
The main failure mode is accepting an interviewer's fluent, confident description of the culture as sufficient evidence on its own; confidence and specificity are not the same thing, and a well-rehearsed answer to a values-page question is exactly what a company under-delivering on its stated culture is most likely to have prepared. A second pitfall is over-weighting a single glowing anecdote from one enthusiastic interviewer without checking whether it generalizes; one great story is an anecdote, not a pattern. A third is treating any inconsistency you find as automatically disqualifying: it is normal for a large or growing organization to have real variance across teams, so the useful conclusion is usually about the SPECIFIC team and manager you'd actually join, not the company as a monolithic whole.

Direct answer

By default, DynamoDB's GetItem and Query calls perform eventually consistent reads, which can be served from a replica that hasn't yet applied the most recent write, so a read immediately after a write can come back stale or missing. The fix is either to request a strongly consistent read on that specific call, or to redesign the flow so the downstream step doesn't need to re-read a value it just wrote.

Structured elaboration

Why it happens: DynamoDB replicates every write across multiple storage nodes in the region before acknowledging the write as successful, but a default GetItem/Query can be routed to a replica that hasn't received that write yet. This is a deliberate cost and latency trade-off, not a bug: eventually consistent reads use half the read capacity and typically have lower latency than strongly consistent reads.

Your options as the operator:

1. Pass ConsistentRead: true on the specific read that needs the fresh value. It costs twice the read capacity units of an eventually consistent read and only works within the same region, but it guarantees you see every write that was acknowledged before the read started.
2. Avoid the read-after-write pattern entirely: have the writer hand the value it just wrote directly to the downstream step instead of making it re-read from the table.
3. If a separate process genuinely has to re-read, add a short retry with backoff, since same-region replication lag is typically single-digit milliseconds.

Where the ConsistentRead flag has limits: DynamoDB Global Tables (cross-region replicas) replicate asynchronously, so a strongly consistent read in one region's replica still only guarantees you see every write already acknowledged in that region, never a write still in flight from another region; there is no cross-region strong-consistency option. DynamoDB Accelerator (DAX, an in-memory cache in front of DynamoDB) is a different case: requesting ConsistentRead: true through DAX does work, DAX simply passes that request straight to DynamoDB without serving or populating it from cache, so you get a genuinely fresh read at the cost of losing DAX's cache acceleration for that one call.

Worked example

A checkout service writes an order row, then a fulfillment worker in the same request path reads it back to grab the shipping address. For a sub-4KB item, an eventually consistent GetItem consumes 0.5 RCU while a strongly consistent one consumes 1 RCU, so setting ConsistentRead: true on that one call costs an extra 0.5 RCU and removes the race entirely, versus a blind retry loop that adds latency and still isn't guaranteed to succeed on the first attempt.

Trade-offs and pitfalls

• Turning on strongly consistent reads everywhere "to be safe" roughly doubles read capacity cost and latency across the service; it should be applied surgically to the one call with the race, not the whole read path.
• Strongly consistent reads do nothing for a Global Table's cross-region replica (there is no cross-region equivalent), and while they do work through DAX, they lose all cache acceleration when they do, a gap teams often discover only after a multi-region or DAX rollout.
• The most robust fix is usually architectural (pass the value forward instead of re-reading it), since it removes the timing dependency entirely and also removes the extra read-capacity cost.

What the interviewer probes next

Expect a follow-up comparing this to S3, which has provided strong read-after-write consistency for every operation (new objects, overwrites, and deletes) since late 2020, unlike DynamoDB's opt-in ConsistentRead, and how you'd catch this class of race in production before a customer reports it.

Approach

The decorator has two competing requirements: capture rich structured data on every call, but add essentially zero overhead when instrumentation is turned off, so teams aren't disincentivized from leaving it enabled by default.

import functools, hashlib, json, logging, time

logger = logging.getLogger("instrument")
INSTRUMENT_ENABLED = True  # a module-level flag, or read from config/env

def instrument(fn):
    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        if not INSTRUMENT_ENABLED:
            return fn(*args, **kwargs)  # zero overhead: skip hashing, timing, logging entirely
        args_hash = hashlib.sha256(repr((args, kwargs)).encode()).hexdigest()[:12]
        start = time.time()
        try:
            result = fn(*args, **kwargs)
            duration_ms = int((time.time() - start) * 1000)
            logger.info(json.dumps({"function": fn.__name__, "args_hash": args_hash,
                                     "duration_ms": duration_ms, "status": "ok"}))
            return result
        except Exception as e:
            duration_ms = int((time.time() - start) * 1000)
            logger.info(json.dumps({"function": fn.__name__, "args_hash": args_hash,
                                     "duration_ms": duration_ms, "status": "error",
                                     "exception": repr(e)}))
            raise
    return wrapper

@instrument
def process_payment(order_id, amount):
    ...

Verified in a sandbox: an instrumented function that returns normally produces one JSON log record with status: "ok" and a correctly-computed duration_ms; an instrumented function that raises ValueError("bad") produces one JSON log record with status: "error" and the exception repr embedded, AND still correctly re-raises the original exception to the caller (the decorator observes and logs, it never swallows).

Args hashing, not args logging

Hashing (args, kwargs) rather than logging them directly is deliberate: raw arguments could contain sensitive data (this connects directly to the secrets-in-logs concern discussed elsewhere in this topic), and a hash still gives you a genuinely useful correlation tool -- 'were these two calls invoked with identical arguments' is answerable from the hash alone without ever writing the actual argument values to a log stream. This is a real trade-off worth stating explicitly: you lose the ability to see the actual argument VALUES in the log for debugging, in exchange for the log being safe to leave enabled everywhere by default without a case-by-case sensitivity review of every instrumented function's parameters.

Minimizing overhead when disabled

The if not INSTRUMENT_ENABLED: return fn(*args, **kwargs) early-return is the whole trick -- when disabled, the wrapper does ONE boolean check and then calls straight through, with the hashing, timing, and JSON serialization code paths never executing at all. This is meaningfully cheaper than an approach that always computes the instrumentation data and only skips the FINAL log write when disabled, since hashing and JSON serialization aren't free even if their output goes nowhere.

Trade-offs and pitfalls

A module-level boolean flag (as shown) is simple but global -- every call to every @instrument-decorated function shares one on/off switch. A more flexible (and more complex) version would support per-function or per-call enabling, useful for temporarily turning on detailed instrumentation for ONE suspect function during an investigation without paying the overhead everywhere else -- worth naming as a natural extension, not something this simple version handles.

Edge cases: a wrapped function whose arguments include something unhashable or very large (a big DataFrame, say) makes repr((args, kwargs)) expensive or even itself slow -- for functions where this matters, hash only a caller-specified subset of arguments rather than blindly hashing everything passed in.

To implement consistent hashing with virtual nodes, map node replicas to points on a hash ring (0..2^32-1) using a stable hash, keep a sorted list of replica hashes, and map keys to the nearest clockwise replica. Virtual nodes spread a physical node's replicas around the ring, reducing the fraction of keys that move when nodes join/leave (from O(1/N) per node to more even distribution).

import bisect
import hashlib

def _hash(value):
    return int(hashlib.md5(value.encode()).hexdigest(), 16)  # 128-bit space

class ConsistentHashRing:
    def __init__(self):
        self.ring = []               # sorted list of replica hashes
        self.hash_to_node = {}       # map replica hash -> node_id
        self.node_replicas = {}      # node_id -> list of replica hashes

    def add_node(self, node_id, replicas=100):
        reps = []
        for i in range(replicas):
            h = _hash(f"{node_id}-{i}")
            if h in self.hash_to_node:  # handle accidental collision
                continue
            bisect.insort(self.ring, h)
            self.hash_to_node[h] = node_id
            reps.append(h)
        self.node_replicas[node_id] = reps

    def remove_node(self, node_id):
        reps = self.node_replicas.pop(node_id, [])
        for h in reps:
            idx = bisect.bisect_left(self.ring, h)
            if idx < len(self.ring) and self.ring[idx] == h:
                self.ring.pop(idx)
                self.hash_to_node.pop(h, None)

    def get_node(self, key):
        if not self.ring:
            return None
        h = _hash(key)
        idx = bisect.bisect_right(self.ring, h)
        if idx == len(self.ring):
            idx = 0
        return self.hash_to_node[self.ring[idx]]

Key points:

• Virtual nodes (replicas) scatter each physical node across the ring -> better uniformity and smaller key movement on changes.
• Add_node: O(R * log M) for inserting R replicas into ring of size M.
• Remove_node: O(R * log M).
• Get_node: O(log M) for a binary search on M replicas.
  Edge cases: empty ring (return None), hash collisions, choose replicas based on load/weight for weighted nodes. Alternative: use sortedcontainers for faster operations or consistent hashing libraries for production.

Situation: In my previous SRE role we were repeatedly firefighting performance regressions that hit launches because reliability goals weren’t defined early. Product and Design saw constraints as blockers to creativity.

Task: Convince those teams to adopt SLOs and performance budgets during design, while preserving product velocity and collaboration.

Action:

• Framed benefits in product terms: translated SLOs into user impact (e.g., “99.9% success = <1 minute/year downtime for core flow”) and tied error budgets to release capacity so teams could trade features vs. risk.
• Ran a short pilot with one product squad: co-created a lightweight SLO (latency P95, availability) and a simple performance budget for initial wireframes—kept it non-blocking by making the budget advisory for the pilot.
• Embedded SRE in early design touchpoints: 30-minute “reliability check” in design reviews with a one-page checklist (user journeys, critical paths, expected qps, fallback UX).
• Provided concrete, low-effort tools: templates for SLOs, a dashboard showing projected error-budget burn given expected traffic, and a pre-commit CI perf test scaffold that flags major regressions but doesn’t block PRs—only annotates.
• Trained PMs/Designers with a 1-hour workshop using examples showing how small design changes (e.g., optimistic UI, graceful degradation) preserve UX within budgets.
• Created decision rules: if projected burn > threshold, require a short mitigation plan; otherwise proceed.

Result / Measurement:

• Adoption metric: percent of new features with an associated SLO at design time (target 80% in 3 months). Pilot reached 90% for that squad.
• Impact metrics: reduction in post-release performance incidents by 40% and decrease in emergency rollbacks by 30% in quarter following rollout.
• Collaboration metrics: average time in design review unchanged; NPS from PMs improved (qualitative feedback: less surprise, clearer trade-offs).

Learnings / Maintenance:

• Keep constraints lightweight and framed as enablers, not blockers.
• Automate visibility (dashboards, CI annotations) so decisions are data-driven.
• Re-evaluate SLOs quarterly with product to align to changing priorities.
• Use error budgets as a governance tool to enable safe experimentation rather than as a hard gate.

Direct answer

Evaluate against the failure you are actually trying to fix, not the tool's feature list. Pilot on a small, real, high-friction slice of work with the people who will use it, not a toy example. Then treat adoption as something you have to earn, low switching cost, hands-on training, and visible evidence, rather than something you can mandate.

Structured elaboration

Evaluation: name the specific problem before comparing options, "deploys are manual and undocumented," not "we should modernize." Score a short list of real candidates against criteria that matter for this team specifically: integration cost with what you already run, learning curve for the team you actually have, and total cost including ongoing maintenance, not just the sticker price.

Pilot: pick a real, currently painful piece of work, not a demo, put a hard time box on it, and migrate a handful of concrete cases rather than the whole system. Instrument it so you can compare before and after, qualitatively at minimum, and with numbers you can show your work for where you actually measure them.

Getting real adoption, not a tool nobody uses:

1. Reduce switching cost directly: a starter template, a migration script, or paired sessions, not just published docs.
2. Find a credible first team, ideally one that is already vocal and frustrated with the status quo, and let their success be the pitch to the next team rather than a top-down mandate.
3. Expect and budget for a short-term velocity or quality dip during migration, for example a temporary regression while old and new systems run side by side, and get that dip pre-approved with your pilot data so it is not read as failure mid-rollout.
4. Make the new tool the path of least resistance. If the old way is still just as easy, most teams will quietly keep using it regardless of how much better the new one is.
5. Watch for adoption in name only: count teams actually using it in production, not teams who attended a training session.

Knowing when to reverse course: the same pilot discipline should let you kill an unpopular or risky tool cleanly too. If a pilot shows real operational risk, or the team genuinely cannot use it, that is a valid pilot outcome, not a failure of the champion.

Worked example

A team's nightly pipeline jobs were opaque and deploys were manual; engineers avoided touching the pipeline because a bad deploy was hard to diagnose and roll back. The proposal was a transformation framework with version-controlled, testable definitions orchestrated by a scheduler, instead of hand-rolled scripts. The pilot migrated three of the most-touched, most fragile pipelines, not the whole system, over four weeks, added tests for each, and wired basic deploy automation. Adoption plan: two hands-on working sessions instead of a slide deck, a working example repo new pipelines could copy from, and pairing with two engineers who became the first internal advocates.

Illustrative cost framing, stated up front rather than claimed after the fact: if a bad deploy previously cost about a day of debugging and happened roughly monthly, that is about 12 engineer-days a year, against an estimated 15 to 20 engineer-days to build and pilot the migration, so the pilot was expected to pay for itself within the first year even before counting ongoing savings from easier onboarding. The real adoption signal to watch for is simpler and harder to fake than any dashboard: did the next three pipelines that got touched get migrated voluntarily, or did people quietly keep writing the old way.

Trade-offs and pitfalls

• A pilot on a toy or greenfield example proves the tool works in ideal conditions, not that it survives your team's real mess. Pilot on something painful and real.
• Mandating adoption before switching cost is low produces compliance theater: people check the box during the pilot window and revert once attention moves on.
• Overselling the pilot's results erodes trust the first time someone re-runs your comparison and gets a different answer. Only claim what you can show.
• Committing to a tool because one senior engineer is enthusiastic about it, without a real pilot against a real failure, is how orgs end up maintaining tools nobody chose deliberately.

Serverless (AWS Lambda / GCP Functions) vs Container-based (Kubernetes / Fargate) — SRE-focused pros & cons

Cold starts

• Serverless: Con — cold starts can add latency (language/runtime dependent). Pro — reduced for provisioned/concurrency features but increases cost. Good for bursty workloads where occasional latency is acceptable.
• Containers: Pro — long-running pods avoid cold starts. Con — more responsibility to scale nodes and manage idle resources.

Observability

• Serverless: Con — limited low-level telemetry (no host metrics), tracing and logs are vendor-managed; need vendor integrations and extra instrumentation for context propagation.
• Containers: Pro — full access to host/sidecar-based metrics, richer tracing; easier to standardize on Prometheus/Jaeger and sidecar patterns.

Vendor lock-in

• Serverless: Con — function APIs, event sources, and deployment tooling can be proprietary; mitigations: use abstractions (CloudEvents), frameworks (Knative) or keep business logic portable.
• Containers: Pro — higher portability (OCI images, k8s manifests), easier multi-cloud/hybrid strategies.

Scaling characteristics

• Serverless: Pro — instant-ish autoscale to zero and scales per-invocation; Con — concurrency limits, cold starts at scale, less control over scheduling.
• Containers: Pro — predictable, tunable scaling (HPA/VPA/cluster autoscaler); Con — more ops to reason about capacity and bin-packing.

Cost predictability

• Serverless: Pro — pay-per-invocation can be cost-effective for spiky/low-usage. Con — hard to predict at high sustained workloads; costs can exceed reserved instances.
• Containers: Pro — predictable with reserved instances/spot; Con — baseline costs for always-on nodes.

Local debugging & dev experience

• Serverless: Con — harder to replicate cloud event environment locally; emulators exist but imperfect.
• Containers: Pro — easy to run locally, replicate full stack, attach debuggers, integration testing.

Operational control & Security

• Serverless: Pro — less infra to manage (patching, OS). Con — limited control over runtime env and custom networking; cold platform security model.
• Containers: Pro — full control over OS, runtime, networking; Con — need to manage patching, runtime security, and admission controls.

When to choose

• Serverless: lightweight APIs, event-driven, unpredictable traffic, minimal ops team.
• Containers/Kubernetes: Stateful services, high throughput/low-latency needs, strong observability/control, multi-cloud portability.

Recommendations

• Use serverless for glue logic, sporadic jobs, and event-driven APIs; instrument heavily (structured logs, distributed tracing) and consider provisioned concurrency for latency-sensitive paths.
• Use containers for core services needing consistent performance, advanced networking, or portability; automate observability, autoscaling, and cost controls.