Serverless and Function-as-a-Service Architecture Questions

Direct answer

Monitor three layers. Platform signals tell you whether requests are arriving, succeeding, being throttled, and how long they take. Function-level lifecycle signals separate time spent initializing a new execution environment (a cold start) from time spent in your handler, because they have different causes and different fixes. Service-specific output tells you whether the function is doing the right thing, which platform metrics cannot see: a function can return HTTP 200 all day with wrong answers.

1. Infrastructure-level signals

Names below are AWS Lambda's CloudWatch metrics; other platforms expose equivalents.

2. Function-level breakdown: initialization versus handler

A single invocation's time is:

• Init: a new environment is created, the runtime starts and your global-scope code runs (imports, SDK clients, loading a model).
• Handler: your function body for this request.

They must be separated because Lambda's Duration metric excludes cold-start time. The init time appears as Init Duration on the REPORT log line of the invocation that paid it, and only on those invocations. So:

• Cold-start rate = invocations whose REPORT line carries Init Duration, divided by all invocations. Compute it from logs (for example with a CloudWatch Logs Insights query, a query language and console for searching and aggregating over your CloudWatch log data directly, over REPORT lines) or with CloudWatch Lambda Insights (a monitoring add-on that automatically collects a function's CPU, memory and cold-start metrics without you writing the queries yourself).
• Init duration distribution (p50 and p99), watched per deploy: a new dependency that adds 800 ms of imports shows up here and nowhere else.
• Handler duration broken down further with tracing spans (the individual timed segments, for example one downstream call, that a distributed tracing tool such as AWS X-Ray stitches together into one request's timeline): time in downstream calls versus your own computation.
• Memory: Max Memory Used against configured memory, from the same REPORT line. Near the limit means out-of-memory risk; far below means you may be over-paying.

Worked example: a function shows p99 Duration of 120 ms, yet users see 1.4 s spikes (about the 1,200 ms Init Duration plus the 120 ms handler time, plus the normal network and invocation overhead neither metric captures on its own). Logs show 0.8% of invocations carry Init Duration of about 1,200 ms. The spikes are cold starts, invisible in Duration, because Duration only ever measures handler time and excludes Init.

Here is why the percentile hides them. Sort every invocation's user-facing latency (Init plus handler, for the ones that pay Init) from fastest to slowest. The slowest 0.8% of that list are exactly the cold ones, so they occupy the top 0.8% of the distribution: from the 99.2nd percentile up to the 100th. The 99th percentile sits at the 99.0 mark, still just below where the cold cluster starts at 99.2, so p99 lands on a warm value, about 120 ms, even though 0.8% of real users are waiting over a second. Since 0.8% is under 1%, the p99 of user latency sits right at the edge: if the cold-start rate crept up past 1% (more traffic, a new dependency, fewer warm environments), the cold cluster would grow to cover the 99th percentile itself, and the reported p99 would jump from about 120 ms to roughly 1,200 to 1,300 ms in one step, even though no individual request got any slower. The fix is on the init side (trim imports, pre-warm), not in the handler.

3. Service-specific output

Emit these yourself as custom metrics (for example with the CloudWatch embedded metric format, which turns structured log lines into metrics):

• Business outcome counts: orders accepted, payments declined, messages delivered. A drop with flat Invocations means requests arrive but fail logically.
• Output quality: for a validation or scoring function, the distribution of outputs (share of rejects, score histogram). For an ML inference function, prediction distribution and confidence; a shift means input drift (the real-world data the model now sees has statistically changed shape from what it was trained or validated on) or a bad model version.
• Downstream dependency health: latency and error rate per dependency, so a slow database is distinguishable from slow code.
• Idempotency or duplicate signals: duplicates detected and skipped, since event sources deliver at least once (the same event may arrive more than once).

Keep dimensions (the labels, such as function version or region, you attach to a metric so you can slice and filter it) low-cardinality (built from few distinct values): tag by function version or region, never by user ID, because each unique combination of dimension values is stored and billed as its own separate metric time series, so a high-cardinality dimension like user ID can silently create millions of them.

Why each category matters, in one line each

• Platform signals answer "is it up and keeping up?"
• Init versus handler answers "is latency coming from environment start-up or from my code?", which decides between pre-warming and code optimization.
• Service output answers "is it right?", the only layer that catches silent correctness failures.

Pitfalls

• Alarming on error rate but not on Throttles, so a saturated function looks healthy.
• Reading average Duration and concluding there is no cold-start problem.
• No alarm on DLQ depth.
• Putting request IDs or user IDs into metric dimensions and receiving a large bill for millions of metrics.

Direct answer

Instrument every hop with OpenTelemetry (the open standard and SDKs for traces, metrics and logs), propagate one trace context through the API gateway, the functions and every message across async hops, and put a correlation ID in every log line. Keep costs bounded with low head sampling for successful traffic plus guaranteed capture of failures, cheap aggregated metrics for SLOs (service-level objectives) and anomaly detection, and logs with short hot retention and cheap archive. To debug one failed request, walk from the request ID the client saw, to the trace ID in its logs, to the trace, across the queue via its link to the consumer's span.

Requirements I am designing to

• 100 requests per second average at the gateway (259.2M requests per month), each producing a sync function call and an async message to a worker.
• p99 (the 99th-percentile latency: the value only the slowest 1% of requests exceed) latency SLO on the API; error-budget alerting (paging when the service has spent too much of the failure rate its SLO allows, rather than on every individual error; the full mechanism is below).
• Find any specific failed request within minutes, including failures that happen after the async hop.
• Observability spend should be a small fraction of compute spend, with arithmetic below.

Architecture

flowchart LR
  C[Client] --> G[API gateway]
  G --> F1[API function]
  F1 -->|message plus trace context| Q[Queue]
  Q --> F2[Worker function]
  F2 --> D[Downstream API or DB]
  F1 -.spans, EMF metrics, logs.-> T[Telemetry pipeline]
  F2 -.spans, EMF metrics, logs.-> T
  T --> TR[Trace store]
  T --> M[Metrics store]
  T --> L[Log store plus archive]

Components

• Tracing: the AWS Distro for OpenTelemetry Lambda layer (AWS's own supported build of the open-source OpenTelemetry instrumentation, attached to a function without changing its code) or native AWS X-Ray (AWS's distributed tracing service) on each function. Spans (timed records of one unit of work) cover the gateway, handler, each downstream call, and the enqueue.
• Metrics: SLO metrics (requests, errors, latency histograms) emitted as structured logs in the CloudWatch embedded metric format (EMF), which the platform turns into metrics without a synchronous API call inside the function; or a Prometheus-compatible managed store (Prometheus is a widely used open-source metrics system; a managed store that speaks its query language) if the organization standardizes on it.
• Logs: structured JSON with request_id, trace_id, function_version, outcome. Kept in hot retention (the fast, queryable store, as opposed to cheap cold archive) for 14 days, then archived to object storage.
• Anomaly detection: statistical bands (a normal range computed from a metric's own recent history, so an alert fires only once it moves outside that range) on a few high-value series (error rate, p99, throughput per route), plus multi-window burn-rate alerts: an SLO like 99.9% availability allows a fixed amount of failure over its measurement window, the error budget (here, 0.1% of requests over 30 days); a burn-rate alert pages when errors are consuming that budget faster than a sustainable rate, checked over both a short window (to catch a real incident fast) and a long window (so a brief blip does not page anyone) at once, which is why it is "multi-window" (for example, page when 2% of the monthly error budget is consumed in one hour). Anomaly detection runs on aggregated metrics, never on raw traces, which keeps it cheap.

Propagating context across the async hop

The trace context (a trace ID plus parent span ID, the ID of the span that caused this one, in the W3C (World Wide Web Consortium, the standards body) traceparent header or AWS's X-Amzn-Trace-Id) must ride inside the message, because there is no HTTP connection between producer and consumer.

• SQS (Amazon Simple Queue Service) to Lambda on AWS: SQS carries the X-Ray trace header in the reserved AWSTraceHeader message system attribute, and a Lambda consumer picks it up automatically; the console shows the producer's trace linked to the consumer's.
• Other brokers (EventBridge, Kafka (Apache Kafka, a distributed event-streaming log), SNS fan-out, one SNS message delivered to every subscriber at once): inject traceparent into message attributes or headers at publish and extract it in the consumer.
• Batches: one consumer invocation may process 10 messages from 10 different traces. Model the consumer's work as its own span with span links (references from one span to other related spans that are not its direct parent or children, so one span can point back to many origins) to each message's producing span, rather than pretending it has a single parent.
• Redundant safety net (belt and braces): also put a business correlation ID (order ID, request ID) in the message body and in every log line, so the chain can be reconstructed even for unsampled traces.

Sampling strategy and the cost arithmetic

Prices: X-Ray records traces at $5.00 per million, and CloudWatch Logs standard ingestion is $0.50 per GB (US East, current list).

Decision: head-sample 5% of successful requests (the decision is made at the entry point and propagated, so a trace is complete or absent, never half-recorded), and capture 100% of errors and slow requests. Capturing all errors needs tail sampling: deciding after the trace finishes. A function cannot do that alone because it sees only its own spans, so either route spans through an OpenTelemetry Collector gateway (a standalone process that receives every function's spans and applies rules to them centrally, before they reach the trace store) running tail-sampling rules (errors, latency above the SLO threshold, specific tenants), or, if you stay on head sampling only, guarantee that every error log line carries the trace ID and full context, so unsampled failures are still debuggable from logs.

Logs: 1 KB per request is 259.2 GB per month, $129.60 of ingestion. Log at INFO for outcomes only, DEBUG sampled, errors in full.

Metrics cardinality is where observability bills explode. Each unique combination of metric name and dimension values is billed as a separate custom metric ($0.30 per metric per month for the first 10,000, $0.10 for the next tier). Adding a customer_id dimension across 10,000 customers for 3 metrics creates 30,000 metrics:

That is more than all tracing at 5%. Per-customer analysis belongs in logs or traces (queried on demand), not metric dimensions.

Where telemetry is stored

Debugging one specific failed request

1. Client reports failure and returns the request_id from the error response (always return it).
2. Log query on request_id finds the API function's line and its trace_id.
3. Open the trace: gateway span, handler span, enqueue span, status OK. The failure is after the hop.
4. Follow the link from the enqueue span to the worker's span (automatic for SQS to Lambda; via span links for batches).
5. The worker span shows a 5 s downstream timeout; its log lines, filtered by the same trace_id, show the payload that triggered it; the message is in the dead-letter queue, ready to redrive (resend through the same processing path for another attempt) after the fix.

If the request was not in the 5% sample and tail sampling is not deployed, steps 3 and 4 fall back to logs by trace_id and correlation ID, which is why the log discipline is non-negotiable.

Pitfalls

• Head sampling only, then discovering the failure you need was in the 95% dropped.
• Losing trace context at the queue because the producer never injected it.
• High-cardinality metric dimensions.
• Synchronous telemetry exports inside the handler that add latency and billed duration to every request.

Direct answer

Record a timestamp at the very first line of module-level code, record another when module-level setup finishes, and keep a module-level flag cold = true. Module-level code runs exactly once per execution environment, so the first invocation that sees cold == true is the cold one: it reports init duration (the two timestamps' difference) and flips the flag; every later invocation in that environment is warm. Around the handler, time from entry to exit in a finally block so failures are measured too. To push metrics, don't call a metrics API synchronously from the request path; write one structured log line per invocation in a format the backend turns into metrics (on AWS, CloudWatch, the account's monitoring and log service, reads log lines written in its Embedded Metric Format, EMF, and turns them into metrics on its own), or hand the data to a local agent that ships it asynchronously.

Pseudo-code (language-agnostic)

# ---- module scope: runs once per execution environment ----
INIT_START = monotonic_now()
COLD       = true
... imports, clients, config, large dependency ...
INIT_END   = monotonic_now()

function wrap(handler):
    return function(event, context):
        start    = monotonic_now()
        was_cold = COLD
        COLD     = false                 # every later call in this environment is warm
        try:
            return handler(event, context)
        finally:                         # measured on success and on exception
            metrics = { HandlerDuration: ms(monotonic_now() - start) }
            if was_cold:
                metrics.InitDuration         = ms(INIT_END - INIT_START)
                metrics.InitToFirstInvokeGap = ms(start - INIT_END)
            emit(metrics, dimensions = { FunctionName, ColdStart: was_cold })

Use a monotonic clock (one that never jumps when the system clock is corrected) for durations, and wall-clock time only for the metric's timestamp.

Runnable implementation

The wrapper takes the clock as a parameter so the demo can use a fake, deterministic clock; in production you pass time.monotonic_ns. emit prints an EMF document: a JSON log line with an _aws block telling CloudWatch which fields are metrics and which are dimensions (the labels a metric is grouped by).

import json, os, time

class FakeClock:
    """Deterministic clock for the demo. In production pass time.monotonic_ns."""
    def __init__(self):
        self.ns = 0
    def __call__(self):
        return self.ns
    def advance_ms(self, ms):
        self.ns += int(ms * 1_000_000)

class ColdStartMetrics:
    def __init__(self, namespace, function_name, clock=time.monotonic_ns,
                 wall_ms=lambda: int(time.time() * 1000), sink=print):
        self.clock, self.wall_ms, self.sink = clock, wall_ms, sink
        self.namespace, self.function_name = namespace, function_name
        self.init_type = os.environ.get("AWS_LAMBDA_INITIALIZATION_TYPE", "on-demand")
        self.init_start = clock()      # first line of module scope
        self.init_end = None
        self.cold = True               # module scope runs once per environment

    def init_done(self):
        self.init_end = self.clock()

    def wrap(self, handler):
        def wrapped(event, context):
            start = self.clock()
            was_cold, self.cold = self.cold, False
            try:
                return handler(event, context)
            finally:                    # runs on success and on exceptions
                handler_ms = (self.clock() - start) / 1e6
                fields = {"HandlerDuration": round(handler_ms, 3)}
                if was_cold:
                    fields["InitDuration"] = round((self.init_end - self.init_start) / 1e6, 3)
                    fields["InitToFirstInvokeGap"] = round((start - self.init_end) / 1e6, 3)
                self.emit(fields, was_cold)
        return wrapped

    def emit(self, fields, was_cold):
        doc = {
            "_aws": {
                "Timestamp": self.wall_ms(),
                "CloudWatchMetrics": [{
                    "Namespace": self.namespace,
                    "Dimensions": [["FunctionName", "ColdStart"]],
                    "Metrics": [{"Name": k, "Unit": "Milliseconds"} for k in fields],
                }],
            },
            "FunctionName": self.function_name,
            "ColdStart": "true" if was_cold else "false",
            "InitType": self.init_type,          # logged property, not a dimension
            **fields,
        }
        self.sink(json.dumps(doc, separators=(",", ":")))

# ---- demo driver: simulate one environment's life with a pinned clock ----
clock = FakeClock()
metrics = ColdStartMetrics("MyApp", "resize-images", clock=clock, wall_ms=lambda: 1767225600000)
clock.advance_ms(850)        # module-scope work: imports, SDK clients, config load
metrics.init_done()

def handler(event, context):
    clock.advance_ms(event["work_ms"])
    if event.get("fail"):
        raise RuntimeError("downstream timeout")
    return {"ok": True}

h = metrics.wrap(handler)
clock.advance_ms(2)          # time between init finishing and the first request
h({"work_ms": 120}, None)    # cold
clock.advance_ms(5000)       # environment frozen, then thawed
h({"work_ms": 95}, None)     # warm
try:
    h({"work_ms": 40, "fail": True}, None)   # warm, handler raises
except RuntimeError as e:
    print("handler error propagated:", e)

Output:

{"_aws":{"Timestamp":1767225600000,"CloudWatchMetrics":[{"Namespace":"MyApp","Dimensions":[["FunctionName","ColdStart"]],"Metrics":[{"Name":"HandlerDuration","Unit":"Milliseconds"},{"Name":"InitDuration","Unit":"Milliseconds"},{"Name":"InitToFirstInvokeGap","Unit":"Milliseconds"}]}]},"FunctionName":"resize-images","ColdStart":"true","InitType":"on-demand","HandlerDuration":120.0,"InitDuration":850.0,"InitToFirstInvokeGap":2.0}
{"_aws":{"Timestamp":1767225600000,"CloudWatchMetrics":[{"Namespace":"MyApp","Dimensions":[["FunctionName","ColdStart"]],"Metrics":[{"Name":"HandlerDuration","Unit":"Milliseconds"}]}]},"FunctionName":"resize-images","ColdStart":"false","InitType":"on-demand","HandlerDuration":95.0}
{"_aws":{"Timestamp":1767225600000,"CloudWatchMetrics":[{"Namespace":"MyApp","Dimensions":[["FunctionName","ColdStart"]],"Metrics":[{"Name":"HandlerDuration","Unit":"Milliseconds"}]}]},"FunctionName":"resize-images","ColdStart":"false","InitType":"on-demand","HandlerDuration":40.0}
handler error propagated: downstream timeout

The first line is the cold invocation (850 ms init, 2 ms gap before the first request, 120 ms handler). The next two are warm and carry only HandlerDuration. The third invocation raised an exception, and its metric was still emitted before the error propagated. Notice what the _aws.CloudWatchMetrics[0].Dimensions list is doing: [["FunctionName", "ColdStart"]] tells CloudWatch to turn those two fields into metric labels you can filter and group a graph by. InitType sits outside that list, so CloudWatch stores it as a plain property on the log entry, searchable in Logs Insights but not usable to split a metric.

How cold versus warm detection works, and where it breaks

• Why the flag works. The runtime imports your module once per environment. If the environment crashes or times out, Lambda resets it and re-runs module code, so the flag correctly resets to true.
• Provisioned concurrency and pre-initialization. Provisioned concurrency is a Lambda setting that pays to keep a chosen number of environments already initialized and sitting idle, ready before any request arrives, instead of only initializing one when a request shows up needing it. So an environment can be initialized long before the first request. The first request is then "first in this environment" but the user did not wait for init. That is why the wrapper also emits InitToFirstInvokeGap and logs AWS_LAMBDA_INITIALIZATION_TYPE (on-demand, meaning an environment created on demand by an incoming request; provisioned-concurrency; snap-start, covered next; or lambda-managed-instances, an instance type where Lambda runs your function on capacity you provision and one execution environment can serve more than one concurrent invocation). A user-visible cold start is ColdStart=true with a gap near zero.
• SnapStart. SnapStart is a Lambda feature that initializes one environment fully, freezes a snapshot of its memory and disk state once that init finishes, and boots every later environment for that function by restoring from that snapshot instead of re-running your module code. Because module code never runs again in a restored environment, the COLD flag is left true inside the snapshot itself (init ran once, before the snapshot was taken), so every environment that restores from it starts already believing it is cold, and reports the same stale InitDuration the original environment measured, not its own. Worse, INIT_START and INIT_END were read from a monotonic clock (a clock that only measures elapsed time going forward on the machine that reads it, never wall time) running on the environment that made the snapshot, so a reading taken before the snapshot cannot say anything about how long this restored environment took to come up; the clock's origin point is frozen into the snapshot along with everything else. Lambda gives you an escape hatch for this: an after-restore runtime hook, a callback the runtime invokes once a restore finishes and before the handler runs again, where you can reset the timers. Then take the real start-up cost from the platform instead of your own clock: for SnapStart functions, the REPORT log line AWS writes after each invocation carries a Restore Duration field (the time to thaw the snapshot and run your after-restore hook), and AWS defines the user-visible cold-start duration for these functions as Restore Duration plus Duration (the handler's own run time).
• What the wrapper cannot see. Code download, sandbox start and runtime boot happen before your first line runs. The platform's own Init Duration field on the REPORT log line covers the whole Init phase, and the Duration metric excludes init, so compare the two: platform init minus your init is the part you cannot optimize from code.
• Timeouts. When a function times out, the runtime is stopped before finally runs, so no metric is emitted. Detect those from the platform's REPORT line (it carries Status: timeout) or the Errors metric.

Pushing to a monitoring backend

Cardinality. Every distinct combination of dimension values becomes a separate billed metric. FunctionName and ColdStart (two values) are safe; putting request IDs or user IDs in dimensions creates millions of metrics. High-cardinality fields go in the log line as properties (like InitType here), not dimensions.

Complexity and edge cases

• Cost: O(1) time per invocation and one log line; memory O(1).
• Edge cases: exceptions (handled by finally); timeouts (not visible to the wrapper, use platform logs); SnapStart restores (reset timers after restore); provisioned concurrency (use the gap and init type to separate "first in environment" from "user waited"); clock adjustments (monotonic clock only for durations).

Direct answer

The platform's strength is the problem here. A serverless function can scale from 5 to 500 copies in seconds. Each copy has its own retry loop and its own memory, so a traffic spike turns into a flood aimed at a dependency that cannot scale. The fix is to match the rate at which you call the dependency to what it can take:

• put a queue between the trigger and the work
• cap how many function instances may call the dependency at once
• retry with exponential backoff and jitter (a small random extra delay added to each retry, so many instances retrying at once do not all fire again at the same instant) at one layer only
• trip a circuit breaker when the dependency is failing
• decide in advance what a degraded response looks like

On a serverless platform most of this is configuration: the queue trigger's maximum concurrency, reserved concurrency, the visibility timeout (the window during which a message SQS has handed to a consumer is hidden from everyone else; if it is not deleted before the window ends, it reappears on the queue) used as a backoff timer, partial batch responses, and a dead-letter queue.

1. Buffer with a queue

If the caller doesn't need the result immediately, accept the request into a queue (Amazon SQS, Simple Queue Service, here) and return 202 Accepted with a status link. The queue absorbs the spike and holds work while the dependency is down. SQS keeps messages for 4 days by default and up to 14.

2. Rate-match the consumers (the core control)

Set MaximumConcurrency on the SQS event source mapping (the piece of Lambda configuration that polls the queue on your behalf and invokes your function with each batch it reads; allowed range 2 to 1,000). This caps how many function instances the poller (that same polling component) will run at once. Set the function's reserved concurrency to at least that value. Work out the number from the dependency's limit, not from your traffic. See the worked example below.

Reserved concurrency alone is the wrong tool. The poller keeps invoking, Lambda throttles the invocations, and throttled batches go back to the queue and use up their receive count (SQS's own counter of how many times a message has been handed to a consumer without being deleted) on the way to the dead-letter queue (DLQ). MaximumConcurrency stops the poller from invoking in the first place.

If you need an exact global rate across instances (for example, "100 calls per second" rather than "20 in flight"), keep a shared token bucket (a counter that refills at a fixed rate; each call must take one token before it is allowed to proceed, which caps the combined rate no matter how many instances are calling) in a fast store such as Redis or a DynamoDB counter. The cost is one extra round trip per call.

3. Retries: one layer, with backoff and jitter

Retries can happen in five places: the HTTP/SDK client, the function code, Lambda's asynchronous retries (2 by default), SQS redelivery after the visibility timeout, and the original caller. Stacked, they multiply (see the worked example). Choose one:

• Retry only retryable failures: 429, 503 and timeouts. Honour Retry-After when the dependency sends it. Never retry a 400.
• Use the queue as the retry engine. On failure, call ChangeMessageVisibility (the API that resets a message's visibility timeout) with a delay that grows with ApproximateReceiveCount (that same receive-count attribute). This gives exponential backoff without holding a function instance open while it waits.
• Enable partial batch responses (ReportBatchItemFailures), so only the failed messages come back.
• Send poison messages (messages that keep failing no matter how many times they are retried) to a DLQ (dead-letter queue) once their receive count passes maxReceiveCount, a threshold you set to at least 5, and alarm on DLQ depth.

4. Circuit breaker

A circuit breaker counts recent failures. When they cross a threshold it moves from closed (calls flow) to open (fail immediately without calling). After a cool-down it moves to half-open (let a few probe calls through) and closes again if they succeed.

On serverless the breaker's state lives in one environment's memory, so each instance learns about the outage on its own. With concurrency capped at 20, that is acceptable: at most 20 probes per half-open window. For hundreds of instances, share the state in one small record (Redis key or DynamoDB item with a time-to-live), at the price of one read per call.

The serverless-specific trap: when the breaker is open, "fail the batch fast" puts messages back on the queue and increments their receive count. A 30-minute outage then pushes healthy messages into the DLQ. Two options:

• Pause consumption. Automation disables the event source mapping (UpdateEventSourceMapping with Enabled=false) when the breaker opens, and re-enables it after the cool-down.
• Keep the breaker's fast failures below maxReceiveCount by using long visibility delays.

5. Rate limiting at the edge

Reject excess traffic before it costs a function invocation:

• API Gateway stage and method throttling (rate caps set on the whole API stage, or on one method within it), plus usage plans (API Gateway's per-API-key throttle and quota configuration) with per-API-key rate and burst limits (the same token bucket mechanism as above).
• A rate-based rule in the web application firewall (AWS WAF) that limits each source IP.
• A throttled request gets a 429 and never runs your code.

6. Degraded mode when the dependency is down

Timeouts are part of the design. Give the dependency call a short client timeout (for example 2 s) that is well under the function timeout. Otherwise a hung partner keeps all 20 capped instances busy doing nothing.

Worked example

A partner API allows 100 requests per second. Each call takes 200 ms, and a function instance makes its calls one after another. Traffic arrives at 60 messages per second.

vendor_limit_rps = 100      # partner API contract
call_s = 0.200              # one partner call, sequential inside an invocation
calls_per_invocation_per_s = 1 / call_s
max_concurrency = vendor_limit_rps / calls_per_invocation_per_s
print(f"queue consumer MaximumConcurrency = {vendor_limit_rps} / {calls_per_invocation_per_s:.0f} = {max_concurrency:.0f}")

layers, retries = 3, 3
print(f"worst-case attempts reaching the partner per user request: (1+{retries})^{layers} = {(1 + retries) ** layers}")

arrival_rps, outage_s = 60, 600
backlog = arrival_rps * outage_s
drain_s = backlog / (vendor_limit_rps - arrival_rps)
print(f"backlog after a {outage_s // 60}-minute outage: {backlog:,} messages; "
      f"drain time at {vendor_limit_rps}/s while {arrival_rps}/s keep arriving: {drain_s:.0f} s ({drain_s / 60:.0f} min)")

queue consumer MaximumConcurrency = 100 / 5 = 20
worst-case attempts reaching the partner per user request: (1+3)^3 = 64
backlog after a 10-minute outage: 36,000 messages; drain time at 100/s while 60/s keep arriving: 900 s (15 min)

• One instance makes 5 calls per second, so 20 instances exactly fill the 100 RPS contract. Set MaximumConcurrency=20 and reserved concurrency to at least 20. (If a handler makes its calls in parallel, divide by that fan-out as well.)
• Three layers each retrying 3 times means 64 attempts reach the partner for one user request in the worst case. That is how a small blip becomes a retry storm (retries compounding across layers until the call volume reaching the dependency is far higher than the original traffic, which can itself take the dependency down).
• A 10-minute outage leaves 36,000 messages queued, well inside the retention period. Once the dependency recovers, the backlog drains in about 15 minutes, because only 40 RPS of headroom remains while new traffic keeps arriving. Add the 10-minute outage itself and the incident runs for about 25 minutes end to end, from the moment the outage starts to the moment the backlog is fully cleared, which is the number to put in front of the product owner as "how long until we are fully caught up." That is not how long any single message sits stale: SQS works through the backlog in roughly arrival order, so the very first message queued is also first in line once the dependency is back, and its own wait is close to the 10-minute outage, not 25 minutes.

flowchart LR
  client[Client] --> gw[API Gateway, throttled]
  gw --> q[Queue]
  q --> fn[Function, max concurrency 20]
  fn -->|breaker closed| partner[Partner API, 100 RPS]
  fn -->|breaker open| cache[(Last known values)]
  q -.->|after 5 receives| dlq[DLQ]

Pitfalls

• Scaling the function to "handle the load" when the dependency is the bottleneck. More instances produce more throttling.
• Retries at every layer, with no jitter. Every instance retries in the same second.
• In-memory rate limiters that assume a single process.
• A visibility timeout barely above the function timeout: a batch that was throttled or slow reappears while it is still being processed, producing duplicates. Follow AWS's 6x guidance: set the queue's visibility timeout to at least six times the function's timeout (plus any batching window), so a batch that gets retried after a throttle still has room to finish before it goes invisible-then-visible again. (A visibility timeout shorter than the function timeout is rejected by Lambda outright.)
• No DLQ alarm, so failed work sits unseen for days.

Direct answer

Don't ask Lambda to absorb a 50x spike synchronously. Let components that never cold-start take the spike:

• a CDN (content delivery network) serves the event pages
• a virtual waiting room admits buyers at a controlled rate
• API Gateway writes purchase requests straight into a queue, with no function in between

Keep the only synchronous function on the hot path small, and pre-warm it with scheduled provisioned concurrency that starts before the doors open. Let queue workers drain purchases at the rate the inventory database can safely take. Ticket on-sales have a published start time, so "predictive scaling" is mostly calendar-driven, not a forecasting model. Evaluate SLA (service-level agreement) risk in four steps: write SLOs (service-level objectives, the internal targets behind the customer-facing SLA) per path, find the refusal point of every component in the chain, load-test above forecast, and pre-decide the kill switches (config toggles you can flip in seconds to shed load or change behavior, prepared and tested before the event rather than invented during the incident).

Requirements I am designing to

• Baseline 200 RPS (requests per second), peaking at 50x = 10,000 RPS for a few minutes around the on-sale.
• 50,000 tickets. Never oversell. That is a correctness invariant, not a latency goal.
• Accepting a purchase attempt: p99 (99th percentile) under 300 ms. Confirmation to the buyer within about 2 minutes.
• Pay for extra capacity only around the event.

Architecture

flowchart LR
  buyer[Buyers] --> cdn[CDN: event page and assets]
  buyer --> room[Waiting room: signed admission tokens]
  room --> gw[API Gateway]
  gw -->|direct SendMessage| q[Purchase queue]
  gw --> status[Status function, pre-warmed]
  q --> workers[Reservation workers, max concurrency 32]
  workers --> inv[(Inventory DB)]
  workers --> res[(Reservation status table)]
  status --> res
  q -.->|after 5 failed receives| dlq[Dead-letter queue]

1. Browse: served from the CDN cache. No function runs.
2. Admission: the waiting room hands out signed, expiring tokens at a chosen rate. From the buyer's side this looks like a queue page: they land on it, see a position or estimated wait, and are redirected into the real purchase flow once admitted with a token. It is the first throttle, and it is also what makes the process fair: buyers are served in arrival order at the room, not in whatever order the queue happens to deliver.
3. Purchase intent: API Gateway checks the token and sends the request directly to SQS (Amazon Simple Queue Service) through a service integration (a built-in API Gateway connector that calls another AWS service's API directly, with no Lambda function in between). It returns 202 Accepted with a client-generated reservation id. There is no Lambda on this path, so there is nothing to cold-start.
4. Workers: an SQS event source mapping (the piece of Lambda configuration that polls the queue and invokes the reservation function with each batch it reads) invokes the reservation function. Each worker does two things in one database transaction, keyed by the reservation id: first it tries to INSERT a row into a reservations table with reservation_id as the primary key; if that insert hits a duplicate-key conflict, this reservation id has already been processed (the message is a redelivery), so the worker stops there and does nothing further. Only on a successful insert does it run the conditional decrement (UPDATE inventory SET remaining = remaining - 1 WHERE remaining > 0). Because the primary-key constraint on reservation_id lets that insert succeed at most once, the decrement behind it can also run at most once per reservation, so a redelivered message cannot sell a second seat.
5. Status: a small function reads the status table. This is the one synchronous function that sees peak traffic.

Warm-up strategy

Size the status function. Two independent checks apply, and you provision to the larger. First, Little's law (a queueing-theory identity: the average number of requests in flight equals the arrival rate multiplied by how long each one takes to handle) gives the actual concurrency, i.e. how many execution environments are doing work at once: concurrency = RPS x duration. Second, Lambda enforces a request-rate ceiling on the alias equal to 10x whatever concurrency you provision or reserve for it (AWS states this directly for provisioned concurrency: the per-alias request-per-second quota is "10 x allocated provisioned concurrency"). That is not a literal cap on what a single environment can physically do. AWS's own conceptual scaling guide warns against that exact framing: "it's incorrect to say that each Lambda execution environment can handle only a maximum of 10 requests per second." It is a budget tied to the unit count you configure, so under-provisioning starves throughput before it starves headcount. As a conservative upper bound, assume all 10,000 RPS could be status polls, each taking an assumed 50 ms (a single lightweight status-table read): Little's law alone says 500 environments would be busy at once, but provisioning only 500 units caps this alias's own request rate at 10 x 500 = 5,000 RPS, half of what is needed, so the number to provision is the larger of the two:

Provision 1,100 (+10% buffer) on the function's alias, using a scheduled Application Auto Scaling action from T-15 minutes (15 minutes before the on-sale's published start time) to T+30 minutes, then scale back down.

Quota first. The default Regional concurrency quota is 1,000, which is below 1,100. Request an increase weeks ahead, to around 3,000, so the rest of the account still has room during the event.

No "ping" warmers. A scheduled ping keeps one environment warm. It does nothing for 1,000 concurrent requests.

Warm the non-Lambda parts too. Pre-scale the database's capacity. Confirm API Gateway's account throttle. Cache the event page at the CDN before the on-sale.

Make the on-demand overflow cheaper. Keep the package small, create SDK clients at top level, and keep the handler minimal.

Predictive scaling

• Scheduled events (almost all on-sales): the calendar is the forecast. Use scheduled provisioned concurrency, scheduled DB scaling, and a pre-set waiting-room admission rate.
• Sizing inputs: registered interest, previous on-sales for similar demand, and the live waiting-room queue length at T-10 minutes. That last signal is what you use to adjust the admission rate and pre-warm level before the doors open.
• Target tracking on ProvisionedConcurrencyUtilization handles slow drift only. It needs about 3 minutes of sustained load before it acts, so it will miss the first minute, which is the minute that matters.
• Unscheduled spikes (a surprise drop): the waiting room absorbs them. Lambda's on-demand scaling (up to 1,000 new environments immediately, then about 100 per second per function) covers what gets through.

Queue-based throttling

The accept path takes the spike. The worker path runs at a speed you choose:

baseline_rps, multiplier = 200, 50
peak_rps = baseline_rps * multiplier

# Status API (the only synchronous Lambda on the hot path).
# Two independent checks: Little's law for actual concurrency (RPS x duration), and the
# platform's request-rate ceiling, which is 10x whatever concurrency you provision/reserve
# for this alias (not a literal per-environment cap: AWS's scaling guide is explicit that
# framing is wrong). Provision to the larger of the two.
status_s = 0.050
concurrency_needed = peak_rps * status_s
rate_ceiling_floor = peak_rps / 10   # must provision >= this many units so 10x it clears peak_rps
status_units = max(concurrency_needed, rate_ceiling_floor)
print(f"peak {peak_rps} RPS; status-API concurrency (Little's law) = {concurrency_needed:.0f}; "
      f"units needed to clear the 10x rate ceiling = {rate_ceiling_floor:.0f}; provision {status_units:.0f}")

# Reservation workers behind the queue, capped to what the inventory DB sustains
db_writes_per_s, worker_s = 800, 0.040
print(f"worker concurrency cap = {db_writes_per_s} x {worker_s} = {db_writes_per_s * worker_s:.0f}")

tickets, attempts = 50_000, peak_rps * 60
print(f"attempts in a 60 s spike: {attempts:,}; tickets: {tickets:,}; "
      f"sell-out time at {db_writes_per_s}/s: {tickets / db_writes_per_s:.1f} s")

pc, gb, price, window_s = 1_100, 1.0, 0.0000041667, 45 * 60
print(f"provisioned concurrency {pc} x {gb} GB for 45 min: ${pc * gb * window_s * price:.2f}")

slo, reqs = 0.999, peak_rps * 60 * 30
print(f"error budget for {reqs:,} requests at {slo:.1%} success: {reqs * (1 - slo):,.0f} failed requests")

peak 10000 RPS; status-API concurrency (Little's law) = 500; units needed to clear the 10x rate ceiling = 1000; provision 1000
worker concurrency cap = 800 x 0.04 = 32
attempts in a 60 s spike: 600,000; tickets: 50,000; sell-out time at 800/s: 62.5 s
provisioned concurrency 1100 x 1.0 GB for 45 min: $12.38
error budget for 18,000,000 requests at 99.9% success: 18,000 failed requests

• Worker cap = 32. Set it as MaximumConcurrency on the event source mapping, and set reserved concurrency to at least that. The poller then never invokes more workers than the database can serve, which protects the database without anyone writing throttling code.
• Sell-out takes about 62.5 s at 800 reservations per second. After that, a "sold out" flag (config the waiting room and API read) stops admissions, and queued messages are answered "sold out" cheaply. The 600,000-attempt figure is an upper bound if every request in the minute were a purchase. The point is that you must not process 600,000 database writes to sell 50,000 seats.
• Queue hygiene. Visibility timeout at least 6x the function timeout (the time a received message stays hidden before it is redelivered). A dead-letter queue (DLQ, where messages go after repeated failures) with maxReceiveCount of at least 5. Partial batch failure reporting (ReportBatchItemFailures: the function tells the poller exactly which messages in the batch failed, so only those are retried), so one bad message does not retry nine good ones.

Evaluating SLA risk

1. Write per-path SLOs. Accept: p99 under 300 ms at 99.9% success. Confirmation: p95 (95th percentile) under 120 s. Oversell: zero, as an invariant, not a percentage. At 99.9% over a 30-minute peak (18,000,000 requests), the error budget is 18,000 failed requests. That is the number to compare against load-test results.
2. Audit the refusal point of every hop:

3. Load-test at 1.5x forecast (15,000 RPS) with production configuration. Run one test right after a fresh deploy, so you see real cold starts. Add a database failover in the middle of the run.
4. Failure-mode table and kill switches.
   • DB slows: the queue grows, and the confirmation SLO is at risk while the accept SLO is not. Lower the admission rate.
   • Status function throttles: clients back off with jitter (a small random delay added to each retry, so they do not all retry at the same instant).
   • Waiting room fails: API Gateway throttling is the fallback.
     Each lever is a config change you can make in seconds, rehearsed in a game day (a scheduled drill where the team deliberately triggers a failure, such as the database failover in step 3 above, to prove the response actually works).

Cost

Pre-warming 1,100 x 1 GB for 45 minutes costs about $12.38 (from the script above). The same 1,100 environments left on for a full month (730 hours) would cost 1,100 x 1.0 GB x 730 x 3,600 s x $0.0000041667/GB-s, about $12,045 a month. Event-window scheduling is the whole cost story.

Pitfalls

• A synchronous function plus a database write on the accept path. At 10,000 RPS that becomes thousands of database connections and cold starts, exactly when you can least afford them.
• Warmer pings instead of provisioned concurrency.
• Trusting target tracking to react in time.
• Checking Lambda quotas and forgetting API Gateway, the database, or the CDN configuration.
• Treating the queue as an infinite buffer. Buyers are waiting. Tell them their position, and stop admitting once the backlog alone can sell out the inventory.