Serverless and Function-as-a-Service Architecture Questions

Direct answer

For an 800 MB artifact that changes with the code, package the function as a container image: it lifts the size limit from 250 MB to 10 GB, needs no private network, and Lambda loads image contents on demand from a cache, so cold starts track what init actually reads. Choose Amazon EFS (Elastic File System, a shared network file system) when the artifact changes independently of the code or is shared by several functions. Avoid downloading from S3 at every cold start: the arithmetic below shows 800 MB alone takes over 10 seconds per cold start. Move to a managed always-on endpoint outside Lambda when traffic is steady enough that you pay for idle capacity anyway, when you need a GPU, or when no cold start is acceptable.

First, a correction to the premise: /tmp on Lambda is configurable from 512 MB up to 10,240 MB, so 800 MB can fit there (extra ephemeral storage is billed per GB-second). The real problems are getting 800 MB there on every cold start, and the memory needed if the artifact is then loaded into RAM.

The constraints that decide it

• Package limits. A zip deployment, including layers, may be at most 250 MB unzipped; a container image may be up to 10 GB uncompressed.
• Init time limit. On-demand Init (module-level set-up) gets 10 seconds; if it overruns, Lambda retries Init inside the first invocation, bounded by the function timeout.
• Network bandwidth. Lambda's documented network bandwidth per execution environment is 625 Mbps (megabits per second, 10^6 bits). Anything fetched over the network at cold start is bounded by it.
• Memory. If the artifact must sit in RAM, configured memory must hold it plus the runtime and working space, and you pay for that memory on every invocation.

Worked example: how long does 800 MB take to fetch?

Lambda's "MB" is 2^20 bytes, so 800 MB = 800 x 1,048,576 x 8 = 6,710,886,400 bits. At the 625 Mbps ceiling:

That is the best case, before any decompression or parsing, and it already exceeds the 10-second Init cap. Every new environment pays it, so a burst that needs 50 new environments downloads 50 x 800 MB = 40,000 MB.

The four options compared

flowchart TD
    A[800 MB artifact] --> B{Changes with code releases?}
    B -- yes --> C{Traffic steady or GPU needed?}
    B -- no, updated separately or shared --> E[EFS mount, read lazily]
    C -- no, spiky --> D[Container image]
    C -- yes --> F[Managed always-on endpoint]

Recommendation and what would change it

Default: container image. It is the only option that removes the size limit without adding a network dependency at start-up that you have to operate. Put the artifact in an early image layer that rarely changes and your code in the last layer, so ordinary code changes don't re-upload the artifact.

• Switch to EFS if the artifact updates on its own schedule (for example daily data refreshes) and rebuilding an image each time is unacceptable, or several functions share it. Structure access to read only what each request needs, and provision throughput for peak concurrency.
• Switch to a managed endpoint once sustained load makes always-on replicas cheaper than per-invocation billing, or when the artifact needs a GPU or must stay in memory without per-environment duplication.
• Use S3 download only for rare background jobs where a slow cold start does not matter, and then fetch during the invocation (subject to the function timeout) rather than squeezing it into the 10-second Init.

Pitfalls

• RAM, not just disk. If the 800 MB is fully deserialized, meaning converted from the file's on-disk bytes into in-memory objects your code works with directly, you may need well over 1 GB configured, paid on every invocation. Memory-mapping keeps resident memory closer to what is actually touched.
• EFS throughput exhaustion. Many environments starting at once can exhaust burst capacity, the reserve of extra throughput EFS's bursting mode grants based on file-system size and lets you spend down temporarily; the symptom is init times that stretch only during scale-outs.
• Inactive container functions. A container-image function not invoked for weeks becomes inactive; the next call fails while Lambda re-optimizes the image. Scheduled invocations or monitoring the function state avoids surprise.

Direct answer

Serverless systems are usually built by wiring managed services together with events, not by writing servers that poll. The common patterns are:

• event notification (a service announces "something happened")
• queue-based load leveling (a queue sits between producer and consumer)
• publish/subscribe fan-out (one event, many independent consumers)
• orchestration (a workflow engine drives the steps)
• the claim-check pattern (pass a pointer to large data, not the data)
• dead-letter handling (failures go somewhere you can see them)

For "a file lands, so run a job" I would wire it like this:

1. The bucket emits an Object Created event to an event router (Amazon EventBridge, a managed service that receives events from many sources and routes each one to whichever targets match its filtering rules).
2. A rule filtered on the input prefix (the folder-like portion of the object key that events are matched against, such as incoming/) sends it to a queue.
3. A small function starts a workflow (Step Functions Standard, one of two execution modes: Standard keeps a full execution history and can run for up to a year; Express, mentioned below, is cheaper and higher-throughput but keeps no long history, and suits short executions better) whose name is derived from the object's identity.
4. The workflow runs the job.

Every hop is at-least-once (each event may be delivered more than once, though never fewer), so reliability comes from queues with dead-letter queues and alarms. "Not twice" comes from a deterministic execution name. A second start with the same name does not start a second job.

The patterns, briefly

The pipeline

flowchart LR
  up[Upload to incoming/] --> bucket[(Object storage bucket)]
  bucket -->|Object Created event| router[EventBridge rule: prefix incoming/]
  router --> q[Queue]
  q --> starter[Starter function]
  starter -->|StartExecution, deterministic name| wf[Workflow: Step Functions Standard]
  wf --> job[Batch or data-processing job]
  wf --> done[Completion event]
  q -.->|after 5 failures| dlq[DLQ + alarm]

• Why EventBridge rather than a direct bucket-to-function trigger. Rules can filter on prefix and suffix, send one event to several targets, and archive and replay events. A direct S3-to-Lambda notification also works, but it is an asynchronous invoke (Lambda queues the event internally and retries it on failure without the caller waiting for a response) with Lambda's own retry queue (a separate, less visible retry mechanism than a queue you configure and can alarm on yourself), and you get fewer controls.
• Why a queue before the starter. The queue gives a buffer, a retry policy you control, a DLQ, and a visible metric (age of oldest message) to alarm on.
• Why a workflow rather than running the job inside the function. A Lambda invocation lasts at most 15 minutes. A Standard workflow can wait on a job for far longer, and it records every step, so retries and failures are visible.
• Multipart uploads (large files uploaded to object storage in separate chunks that are only assembled into the final object once every part has arrived) emit one event, when the upload completes, so the job never sees a half-written file.

Making the trigger reliable

• Assume duplicates and delays. S3 documents its event notifications as delivered at least once, usually within seconds but sometimes a minute or longer. The event router and the queue are also at-least-once.
• Every async hop gets a DLQ and an alarm. Alarm on DLQ depth above 0, and on the age of the oldest message in the main queue.
• Reconcile. A scheduled sweep compares objects under incoming/ with the set of files the workflow has processed. It re-triggers anything missing. This catches the rare lost event, a misconfigured rule, or an outage longer than the retry windows.
• Avoid recursion. If the job writes output back into the same bucket, write it under a different prefix (or bucket) than the rule listens on. Otherwise every output file triggers another job.

Making sure the job fires once per file

Derive the workflow's execution name from the object's identity: bucket + key + version id (enable versioning on the bucket first: with it on, every overwrite of the same key is kept as a separate, uniquely identified version instead of replacing the object in place, and it is that per-write version id, not the key alone, that changes on a genuine re-upload). For a Standard workflow, a second StartExecution (the API call that starts a workflow execution) with the same name and input returns the existing execution while it is running, and returns ExecutionAlreadyExists once it has closed. Either way, no second job starts. Names can be reused 90 days after an execution closes.

import hashlib

def execution_name(bucket, key, version_id):
    # Same object version -> same name; a re-upload (new version) -> new name.
    digest = hashlib.sha256(f"{bucket}/{key}#{version_id}".encode()).hexdigest()
    return f"ingest-{digest[:40]}"          # 47 chars, within the 80-char limit, no banned characters

class FakeStandardWorkflow:
    """Models documented StartExecution semantics for STANDARD workflows."""
    def __init__(self):
        self.executions = {}                 # name -> (input, status)
    def start(self, name, payload):
        if name in self.executions:
            prev_input, status = self.executions[name]
            if status == "RUNNING" and prev_input == payload:
                return "same execution returned (idempotent)"
            raise RuntimeError("ExecutionAlreadyExists")
        self.executions[name] = (payload, "RUNNING")
        return "started"

def handle(event, sfn):
    d = event["detail"]
    name = execution_name(d["bucket"]["name"], d["object"]["key"], d["object"]["version-id"])
    payload = f'{d["bucket"]["name"]}/{d["object"]["key"]}'
    try:
        return name, sfn.start(name, payload)
    except RuntimeError as e:
        return name, f"{e}: already handled, acknowledge the message"

sfn = FakeStandardWorkflow()
evt = {"detail": {"bucket": {"name": "raw-landing"},
                  "object": {"key": "incoming/2026-09-27/orders.csv", "version-id": "v1"}}}
print(handle(evt, sfn))                                   # first delivery
print(handle(evt, sfn)[1])                                # duplicate while running
sfn.executions[execution_name("raw-landing", "incoming/2026-09-27/orders.csv", "v1")] = (
    "raw-landing/incoming/2026-09-27/orders.csv", "SUCCEEDED")
print(handle(evt, sfn)[1])                                # duplicate after it finished
evt["detail"]["object"]["version-id"] = "v2"
print(handle(evt, sfn)[1])                                # genuine re-upload

('ingest-f5772e2d0a15754e5cd838f6fdd74a26602f48ad', 'started')
same execution returned (idempotent)
ExecutionAlreadyExists: already handled, acknowledge the message
started

• Duplicate while the job is running: the same execution is returned.
• Duplicate after it finished: ExecutionAlreadyExists, which the starter treats as "already handled" and acknowledges.
• A genuine re-upload (new version id) gets a new name and correctly starts a new job.

Two edges to handle deliberately:

• A retry after a FAILED execution needs a new name (for example, append an attempt number after checking the old execution's status). Otherwise ExecutionAlreadyExists blocks the legitimate retry.
• The job itself should also be idempotent. It writes its output to a deterministic path and overwrites, so a manual re-run can't double-count.

Pitfalls

• Hashing the event id instead of the object identity. Duplicates carry different event ids, so the dedupe never matches.
• Dropping the version id and deduping on key alone. A legitimate re-upload of orders.csv would be silently ignored.
• No DLQ alarm. Failed files wait unseen until someone asks where the report is.

Direct answer

A stateless function keeps nothing it depends on between invocations: everything it needs arrives in the event or is read from an external store, and everything worth keeping is written back out. That is the default in serverless, because the platform runs many copies in parallel, routes each request to whichever copy is free, and destroys copies whenever it likes. Holding state inside the function is justified only as a disposable optimization: a warm cache of read-mostly data or a reused connection, where losing it costs a slower request but never a wrong answer. Anything that must be correct or durable (sessions, running totals, workflow progress) lives outside the function, and you pick the store by the latency and consistency you need.

Why "in-memory state" breaks in FaaS

Three properties of the platform, each enough on its own:

1. Many copies. At 50 concurrent requests there are about 50 separate execution environments (sandboxes), each with its own memory. A counter in one is invisible to the others.
2. No affinity. A user's second request can land on a different environment than their first, so an in-memory session is simply missing.
3. No lifetime guarantee. Environments are frozen when idle, recycled every few hours even under load, and reset after a crash. Memory contents vanish without warning.

The simulation below routes 1,000 click events to environments the way a platform might (any environment, new ones added under load), and has the platform recycle one environment part-way through. Each environment keeps a local counter; the "external" counter stands in for an atomic counter in a database (one whose increments cannot partially overlap: two simultaneous +1s always land as +2, never silently lose one).

import random

random.seed(7)
EVENTS = 1000
envs = {}                 # env_id -> in-memory click count (what a "stateful" function would keep)
external_total = 0        # stands in for an atomic counter in an external store
next_env = 0

def pick_env():
    global next_env
    # the platform routes to any idle environment, and adds new ones under load
    if not envs or random.random() < 0.01:
        envs[next_env] = 0
        next_env += 1
    return random.choice(list(envs))

for i in range(EVENTS):
    e = pick_env()
    envs[e] += 1              # in-memory aggregation
    external_total += 1       # externalized aggregation
    if i == 600:              # the platform recycles one environment mid-stream
        victim = min(envs)
        lost = envs.pop(victim)

print("environments created:", next_env)
print("any single environment's count:", envs[min(envs)])
print("sum of surviving in-memory counts:", sum(envs.values()))
print("counts lost when env", victim, "was recycled:", lost)
print("external counter:", external_total)

Output:

environments created: 15
any single environment's count: 163
sum of surviving in-memory counts: 792
counts lost when env 0 was recycled: 208
external counter: 1000

No single environment knows the true total (163 is one environment's view), the in-memory sum is short by the 208 clicks held in the recycled environment, and only the externalized counter is right.

When holding state is justified

The test: if this state disappeared right now, would the next request be slower, or wrong? Slower is acceptable for an in-function cache. Wrong means it belongs outside.

Where externalized state lives, and what each option costs you

Worked example: a click-counting API

Requirement: count clicks per article per minute, 2,000 clicks per second at peak, dashboard reads the counts. Assume a lightweight handler, about 15 ms per click (just an increment call): by Little's Law (the number of requests in flight equals the arrival rate times the time each spends being handled), 2,000/s x 0.015s = 30, which is where the "~30 environments in flight" below comes from.

• Wrong: each function keeps counts[article] += 1 in memory and writes it out "every minute". With ~30 environments in flight (the arithmetic above), 30 partial counts race to overwrite each other, and a recycled environment loses its last minute.
• Right, simple: each click performs an atomic increment on a key such as article#42#2026-09-27T10:15 in DynamoDB. 2,000 writes per second is fine for the table as a whole, but a single viral article sends all its writes to one key; spread that hot key (a single partition key taking disproportionate traffic) across N sub-keys (#shard0 to #shardN) and sum them on read.
• Right, at larger scale: put clicks on a stream (a managed, ordered, replayable log of events that many independent readers can consume, such as Amazon Kinesis or Apache Kafka) instead of writing straight to the table. A stream is split into shards, in this context a partition of the stream that one reader handles at a time (a different sense of "shard" from the DynamoDB sub-keys just above, which split a hot row, not a stream); let one consumer, a process reading a shard in order, aggregate clicks per minute and write the result once. Aggregation in a stream consumer is safe because the stream, not the function's memory, is the source of truth: if a consumer crashes partway through a minute, a new one picks up the shard from the last committed position and can be replayed, meaning it re-reads events the stream already has instead of losing them.
• The justified warm state: each function keeps a 60-second in-memory cache of the article metadata it needs to validate clicks. Losing it costs one extra read.

Trade-offs and pitfalls

• Latency versus correctness. Every external read or write adds a network hop. Cache reads locally when stale data is acceptable; never cache anything you then use to make an authoritative decision (balances, inventory).
• Globals leak across requests. Module-level variables survive between invocations in the same environment, so per-request data stored there can bleed into the next user's request. Keep request data local.
• /tmp is memory-adjacent state too. Files there survive between warm invocations and vanish on recycle. Same rule: cache only.
• Retries make external writes happen twice. Event sources deliver at least once, so a retried invocation can double-increment. Where exactness matters, record a per-event ID with a conditional write (a database write that only succeeds if a stated condition holds, here "this event ID is not already recorded") and skip duplicates.

Direct answer

At a sustained 100,000 events per second I would build this as Kinesis Data Streams (provisioned shards) feeding a Lambda consumer that enriches and pre-aggregates in batches, with a per-shard checkpoint committed atomically alongside the aggregates, plus a Firehose copy of the raw stream into S3 for replay. The two decisions that matter most are both about batching: producers pack many events into each stream record, and Lambda is invoked once per batch of about 1,600 events rather than once per event. Invoking once per event would cost about $52,560 a month in request fees alone. The batched design runs at about $10,900 a month, and the raw archive (not the compute) is the biggest line item.

Quick glossary for the rest of the answer:

• Serverless / FaaS (functions as a service): you upload a function, the cloud runs as many copies as traffic needs and bills per invocation and per millisecond. AWS Lambda is the example here.
• Kinesis Data Streams (KDS): AWS's managed, ordered, replayable log (similar in spirit to Kafka, Apache Kafka: the most common self-hosted alternative doing the same ordered-log job). It is split into shards; each shard accepts up to 1 MB/s or 1,000 records/s of writes.
• Event source mapping (ESM): the Lambda-managed poller that reads a shard and invokes your function with a batch of records.
• At-least-once delivery: every record is delivered, but some may be delivered twice (after a retry). The pipeline must tolerate duplicates.
• Idempotent: processing the same input twice leaves the same result as processing it once.
• Amazon Data Firehose: a managed service that buffers a stream and writes large files to S3. S3 is object storage; Parquet is a columnar file format; Athena is a serverless SQL engine that queries files in S3.
• DynamoDB: AWS's managed key-value database.

Requirements I am designing to

Architecture

flowchart LR
  P[Devices and gateways] -->|packed 24 KB records| K[Kinesis stream 125 shards]
  K --> L[Lambda consumer: enrich and pre-aggregate]
  L -->|one transaction per batch| D[(DynamoDB: per-shard minute aggregates and checkpoint)]
  L -.->|poison records| Q[SQS on-failure queue]
  D --> R[Rollup job every minute]
  R --> S3A[(S3 aggregates as Parquet, queried by Athena)]
  K --> F[Firehose]
  F --> S3R[(S3 raw archive for replay)]

1. Ingestion and buffering

• Shard count. 100 MB/s divided by 1 MB/s per shard is 100 shards by bytes. With 24 events packed per record, the record rate is 100,000 / 24 = 4,167 records/s, far below the 100,000 records/s limit of 100 shards. Bytes bind, so I provision 125 shards (25% headroom for skew and bursts).
• Why pack events. Unpacked, 100,000 records/s would need 100 shards just for the record-count limit, and every downstream fee charged per record (Kinesis PUT payload units, which bill each record in 25 KB chunks; Firehose's 5 KB billing increment) would be charged 100,000 times a second. Packing is done by the gateway or by the Kinesis Producer Library's (KPL, an AWS client library that batches and compresses records before sending them) aggregation feature.
• Partition key. Device ID, hashed, so load spreads evenly across shards. A partition key concentrated on a few tenants creates a hot shard (one shard at its 1 MB/s ceiling while others idle), which shows up as write throttling.
• Provisioned vs on-demand. On-demand mode (no shard planning, billed per GB) costs about $42,048/month at this volume versus about $1,522 provisioned (arithmetic below). For flat, known 100k/s load, provisioned wins by more than 20x. I would flip to on-demand only for a new stream whose traffic I cannot yet predict.
• Buffering. The stream itself is the buffer: default retention is 24 hours, so a consumer outage of a few hours loses nothing; the backlog is replayed when the consumer recovers.

2. Processing concurrency

• The Kinesis ESM runs one concurrent batch per shard by default. ParallelizationFactor (1 to 10) raises that, but it breaks per-shard ordering (ordering is then only per partition key), and my deduplication below relies on per-shard order. So I keep the factor at 1 and scale by adding shards instead.
• Maximum concurrency is therefore 125 execution environments, well inside Lambda's default regional quota of 1,000 concurrent executions. I set reserved concurrency (a per-function concurrency allotment that is both a guaranteed floor and a hard ceiling) of about 150 on this function so it can never starve other functions, and so nothing else can starve it.
• Batching settings: BatchSize 10,000 (the maximum), MaximumBatchingWindowInSeconds 2. Each shard receives 100,000 / 125 = 800 events/s, so each 2 s batch carries about 1,600 events (about 67 packed records, about 1.6 MB, safely under the 6 MB invocation payload limit even after base64 encoding inflates it by a third).
• That gives 125 / 2 = 62.5 invocations/s. If a batch takes about 120 ms, average concurrency is 62.5 x 0.12 = 7.5 environments. Cold starts (the extra latency when a new execution environment boots) are irrelevant here: environments stay warm under constant load, and a few hundred ms of extra delay on a 2 s batch changes nothing.
• Enrichment (joining each event to device metadata) uses a reference table loaded into memory at initialization and refreshed every few minutes, never a per-event database call. 100,000 lookups/s against a database would itself be a larger system than the pipeline.

3. Idempotency: exactly-once aggregates on at-least-once delivery

Kinesis plus Lambda is at-least-once. The ESM retries a whole batch if the function errors or times out, so if the function wrote aggregates and then timed out before returning, the retry would add the same events again. For aggregates (counters), duplicates silently inflate numbers.

The fix: store a per-shard checkpoint (the highest sequence number already counted) in the same atomic write as the aggregate increments. Each Kinesis record has a sequence number that increases within a shard. On every invocation the function:

1. Reads the checkpoint for its shard (cached in memory, re-read if the conditional write in step 3, a database write that only takes effect if the value has not changed since it was last read, fails).
2. Drops records whose sequence number is at or below the checkpoint (already counted).
3. Commits, in one DynamoDB TransactWriteItems call (an all-or-nothing multi-item write): increment the (shard, minute) aggregate items, and set the checkpoint to the batch's last sequence number, with a condition that the checkpoint still equals the value just read.

Either both the counts and the checkpoint land or neither does, so a retry of any subset of the batch finds its records already covered and adds nothing.

A small simulation proves it. It models 4 shards of 5,000 records, with a 10% chance each attempt crashes before writing and a 10% chance it crashes after writing, and halves the batch after each failure (a simplified model of BisectBatchOnFunctionError):

import random
from collections import Counter

def make_stream(num_shards=4, per_shard=5000, seed=7):
    rng = random.Random(seed)
    shards = {}
    for s in range(num_shards):
        recs = []
        for seq in range(1, per_shard + 1):
            minute = (seq - 1) * 60 // per_shard       # events spread over 60 minutes
            group = rng.choice(["sensor-a", "sensor-b", "sensor-c"])
            recs.append((seq, minute, group))
        shards[s] = recs
    return shards

class Store:
    """Stands in for the database: aggregate counters plus a per-shard checkpoint.
    commit() is atomic, like one TransactWriteItems call."""
    def __init__(self):
        self.agg = Counter()
        self.hwm = {}
    def commit(self, shard, deltas, new_hwm):
        self.agg.update(deltas)
        if new_hwm is not None:
            self.hwm[shard] = new_hwm

def handler(store, shard, batch, use_checkpoint):
    last_done = store.hwm.get(shard, 0) if use_checkpoint else 0
    fresh = [r for r in batch if r[0] > last_done]
    deltas = Counter((shard, minute, group) for _, minute, group in fresh)
    new_hwm = max(last_done, batch[-1][0]) if use_checkpoint else None  # never move backwards
    store.commit(shard, deltas, new_hwm)

def run(use_checkpoint, batch_size=500, p_fail_before=0.10, p_fail_after=0.10, seed=42):
    shards = make_stream()
    rng = random.Random(seed)
    store = Store()
    for shard, recs in shards.items():
        pos, size = 0, batch_size
        while pos < len(recs):
            batch = recs[pos:pos + size]
            roll = rng.random()
            if roll < p_fail_before:                 # crash before the write: nothing committed
                size = max(1, len(batch) // 2)       # simplified bisect-on-error
                continue
            handler(store, shard, batch, use_checkpoint)
            if roll < p_fail_before + p_fail_after:  # write committed, then timeout: batch retried
                size = max(1, len(batch) // 2)
                continue
            pos += len(batch)
            size = batch_size
    truth = Counter((s, m, g) for s, recs in shards.items() for _, m, g in recs)
    return truth, store.agg

for mode in (False, True):
    truth, got = run(use_checkpoint=mode)
    label = "with per-shard checkpoint" if mode else "naive (no checkpoint)   "
    print(f"{label}: true events={sum(truth.values())}, counted={sum(got.values())}, "
          f"every (shard, minute, group) cell exact={truth == got}")

Output:

naive (no checkpoint)   : true events=20000, counted=22250, every (shard, minute, group) cell exact=False
with per-shard checkpoint: true events=20000, counted=20000, every (shard, minute, group) cell exact=True

The naive version over-counts by 11.25%. The max(...) on the checkpoint line is load-bearing: my first version set the checkpoint to the retried half-batch's last sequence number, which moved it backwards after a bisect and still double counted (21,125). A checkpoint must only ever move forward.

Two things this does not cover, stated so nobody assumes otherwise: duplicates created by the producer (a device retrying a PUT gets a new sequence number) need an event ID and a dedupe step, and the raw archive in S3 is at-least-once, so queries over raw events should deduplicate by event ID.

4. Error handling

The DLQ (dead-letter queue) idea here is the on-failure destination: a place failed work goes so it is not lost and does not block the shard.

5. Storage choice

• Aggregates: DynamoDB holds the live per-shard, per-minute partial counts (small, write-heavy, needs atomic conditional writes). A rollup (a job that combines many small partial records into one summarized one) Lambda run every minute by EventBridge Scheduler (AWS's managed cron) sums the 125 shard items for a closed minute and writes one Parquet object per minute to S3 with a deterministic key such as aggregates/dt=2026-09-27/hh=14/mm=05.parquet. Rewriting the same key is naturally idempotent. Analysts query with Athena. A small daily compaction merges per-minute files into hourly ones to avoid a small-file problem (many tiny files make each file's fixed per-file read overhead dominate, slowing every downstream query).
• Raw events: Firehose reads directly from the stream and writes compressed files to S3, so enrichment or aggregation logic can be replayed after a bug fix.
• What would change it: if dashboards need sub-second freshness or high-concurrency interactive queries, I would write aggregates to a real-time OLAP (online analytical processing) database such as ClickHouse or Apache Druid instead of S3 plus Athena.

6. Observability

• Consumer lag: Lambda's IteratorAge metric (how old the last record in each batch was when the batch was sent to the function). This is the single most important alarm: rising iterator age means processing is falling behind. Alarm if it exceeds 60 s for 5 minutes.
• Stream health: Kinesis WriteProvisionedThroughputExceeded (a hot or under-provisioned shard) and GetRecords.IteratorAgeMilliseconds.
• Function health: Errors, Throttles, Duration (p99, meaning the 99th percentile), ConcurrentExecutions.
• Business correctness: a reconciliation job (one that independently checks two derived numbers against each other and flags any mismatch) comparing hourly raw-archive event counts with aggregated counts, alarming on a difference above 0.1%. This catches silent double counting or silent drops that no infrastructure metric will show.
• Structured logs with shard ID and sequence range per batch, so any aggregate can be traced to the records behind it.

7. Cost profile

List prices for us-east-1 as published on the AWS pricing pages; the per-event CPU cost is an assumption to be measured in a load test. DynamoDB is billed in write request units (WRU): under on-demand pricing a normal write costs 1 WRU per KB, and (as the pitfall below explains) a transactional write costs double that per KB:

import math
# us-east-1 list prices (verify on the pricing pages before relying on them)
LAMBDA_GBS = 0.0000166667      # $ per GB-second, x86, first tier
LAMBDA_REQ = 0.20 / 1e6        # $ per request
KDS_SHARD_HR = 0.015           # provisioned shard-hour
KDS_PUT_UNIT = 0.014 / 1e6     # per 25 KB PUT payload unit
KDS_OD_IN, KDS_OD_OUT = 0.08, 0.04   # on-demand $/GB ingested, retrieved
FH_GB = 0.029                  # Firehose ingestion $/GB, billed in 5 KB increments
DDB_WRU = 0.625 / 1e6          # on-demand write request unit

SEC_MONTH = 730 * 3600         # AWS bills a month as 730 hours
events_s, event_kb = 100_000, 1.0
events_per_record = 24         # producers pack ~24 KB records
shards = 125                   # 100 MB/s needs 100 shards at 1 MB/s each, plus 25% headroom

records_s = events_s / events_per_record
mb_s = events_s * event_kb / 1000
gb_month = mb_s / 1000 * SEC_MONTH

kds_prov = shards * 730 * KDS_SHARD_HR + records_s * SEC_MONTH * KDS_PUT_UNIT
kds_od = gb_month * (KDS_OD_IN + 2 * KDS_OD_OUT)   # two consumers: Lambda and Firehose

window_s, mem_gb = 2, 1.0
inv_s = shards / window_s
events_per_inv = events_s / inv_s
duration_s = 0.040 + events_per_inv * 0.00005     # ASSUMED: 40 ms fixed + 0.05 ms per event
lam = inv_s * SEC_MONTH * (LAMBDA_REQ + duration_s * mem_gb * LAMBDA_GBS)
per_event_invoke_requests = events_s * SEC_MONTH * LAMBDA_REQ

wru_per_inv = 2 * 1 + 2 * 4 * 1.05   # txn: 1 KB checkpoint + ~1.05 x 4 KB aggregate item, 2 WRU per KB
ddb = inv_s * SEC_MONTH * wru_per_inv * DDB_WRU

fh_billed_kb = math.ceil(events_per_record * event_kb / 5) * 5
fh = records_s * fh_billed_kb / 1e6 * SEC_MONTH * FH_GB

print(f"records/s={records_s:.0f}  data={mb_s:.0f} MB/s  GB/month={gb_month:,.0f}")
print(f"Kinesis provisioned={kds_prov:,.0f}  on-demand={kds_od:,.0f}")
print(f"Lambda: {inv_s:.1f} inv/s, {events_per_inv:.0f} events/inv, {duration_s*1000:.0f} ms, "
      f"avg concurrency={inv_s*duration_s:.1f}, cost={lam:,.0f}")
print(f"Lambda request fee alone if invoked once per event={per_event_invoke_requests:,.0f}")
print(f"DynamoDB aggregates={ddb:,.0f}")
print(f"Firehose raw archive (billed {fh_billed_kb} KB/record)={fh:,.0f}")
print(f"TOTAL (provisioned Kinesis)={kds_prov + lam + ddb + fh:,.0f} $/month")

Output (dollars per month):

records/s=4167  data=100 MB/s  GB/month=262,800
Kinesis provisioned=1,522  on-demand=42,048
Lambda: 62.5 inv/s, 1600 events/inv, 120 ms, avg concurrency=7.5, cost=361
Lambda request fee alone if invoked once per event=52,560
DynamoDB aggregates=1,068
Firehose raw archive (billed 25 KB/record)=7,939
TOTAL (provisioned Kinesis)=10,890 $/month

The ~4 KB aggregate-item size is an assumption for this example: it is enough room for counts and sums across roughly a few dozen (device group, metric) key combinations packed into one item; size it against your own key cardinality. The ~1.05 multiplier on that item accounts for the occasional batch that straddles a minute boundary and touches two minute items. S3 storage and Athena query charges are excluded because they depend on retention and query volume.

What the numbers say:

• Compute is the cheap part ($361). Lambda cost is dominated by how you batch, not by how fast the code is.
• The raw archive is 73% of the bill. Its value (replay, ad hoc analysis) should be confirmed with the data consumers. If a 7-day replay window is enough, extending the stream's retention instead of archiving everything could remove most of it.
• Firehose bills each record rounded up to 5 KB, so 1 KB unpacked records would be billed at 5x their size. Packing records to just under a 5 KB multiple matters.

Trade-offs and pitfalls

• Is serverless right at this volume? The load is flat and high, which is where serverless is weakest on unit price. It still wins here because the managed pieces (stream, poller, retries, scaling to 125 parallel consumers, no servers to patch) cost less in engineering time than running and patching your own Kafka and stream-processing cluster, while the Lambda line itself is only a few hundred dollars. If enrichment grew heavy, say 2 ms of CPU per event instead of 0.05 ms, two things break. First, one consumer per shard can then process at most 1 / 0.002 = 500 events/s, but each shard receives 800, so the stream falls steadily behind unless you add shards: 100,000 events/s / 500 events/s per shard needs at least 200 shards by that arithmetic (up from 125), before headroom. Second, compute becomes 100,000 x 0.002 = 200 GB-seconds every second at 1 GB, which is 200 x 2,628,000 s/month x $0.0000166667/GB-s, about $8,760 a month in Lambda duration alone. At that point a long-running container consumer (for example Apache Flink, an open-source stream-processing engine that keeps one continuously running process instead of Lambda's start-stop-per-batch model, on a managed service) is the better call, because steady, CPU-heavy work is exactly where per-millisecond billing loses.
• Per-event invocation is the classic mistake: 262.8 billion requests a month.
• ParallelizationFactor as the first scaling lever silently breaks the per-shard ordering the checkpoint relies on. Scale with shards.
• Aggregating in the function's memory across invocations is unsafe: environments are recycled at any time and two environments can serve the same shard over time. State lives in the database or in Lambda's managed tumbling-window state (a built-in feature that carries a small amount of state between consecutive batches from the same shard, for windowed aggregation, which AWS documents as at-least-once and capped at 1 MB per shard, so it does not remove the double-count problem).
• Transactions are not free: each item written in a DynamoDB transaction costs 2 write request units (WRU) per KB instead of 1. That is why there is one transaction per batch, not per event.

Direct answer

Don't ask Lambda to absorb a 50x spike synchronously. Let components that never cold-start take the spike:

• a CDN (content delivery network) serves the event pages
• a virtual waiting room admits buyers at a controlled rate
• API Gateway writes purchase requests straight into a queue, with no function in between

Keep the only synchronous function on the hot path small, and pre-warm it with scheduled provisioned concurrency that starts before the doors open. Let queue workers drain purchases at the rate the inventory database can safely take. Ticket on-sales have a published start time, so "predictive scaling" is mostly calendar-driven, not a forecasting model. Evaluate SLA (service-level agreement) risk in four steps: write SLOs (service-level objectives, the internal targets behind the customer-facing SLA) per path, find the refusal point of every component in the chain, load-test above forecast, and pre-decide the kill switches (config toggles you can flip in seconds to shed load or change behavior, prepared and tested before the event rather than invented during the incident).

Requirements I am designing to

• Baseline 200 RPS (requests per second), peaking at 50x = 10,000 RPS for a few minutes around the on-sale.
• 50,000 tickets. Never oversell. That is a correctness invariant, not a latency goal.
• Accepting a purchase attempt: p99 (99th percentile) under 300 ms. Confirmation to the buyer within about 2 minutes.
• Pay for extra capacity only around the event.

Architecture

flowchart LR
  buyer[Buyers] --> cdn[CDN: event page and assets]
  buyer --> room[Waiting room: signed admission tokens]
  room --> gw[API Gateway]
  gw -->|direct SendMessage| q[Purchase queue]
  gw --> status[Status function, pre-warmed]
  q --> workers[Reservation workers, max concurrency 32]
  workers --> inv[(Inventory DB)]
  workers --> res[(Reservation status table)]
  status --> res
  q -.->|after 5 failed receives| dlq[Dead-letter queue]

1. Browse: served from the CDN cache. No function runs.
2. Admission: the waiting room hands out signed, expiring tokens at a chosen rate. From the buyer's side this looks like a queue page: they land on it, see a position or estimated wait, and are redirected into the real purchase flow once admitted with a token. It is the first throttle, and it is also what makes the process fair: buyers are served in arrival order at the room, not in whatever order the queue happens to deliver.
3. Purchase intent: API Gateway checks the token and sends the request directly to SQS (Amazon Simple Queue Service) through a service integration (a built-in API Gateway connector that calls another AWS service's API directly, with no Lambda function in between). It returns 202 Accepted with a client-generated reservation id. There is no Lambda on this path, so there is nothing to cold-start.
4. Workers: an SQS event source mapping (the piece of Lambda configuration that polls the queue and invokes the reservation function with each batch it reads) invokes the reservation function. Each worker does two things in one database transaction, keyed by the reservation id: first it tries to INSERT a row into a reservations table with reservation_id as the primary key; if that insert hits a duplicate-key conflict, this reservation id has already been processed (the message is a redelivery), so the worker stops there and does nothing further. Only on a successful insert does it run the conditional decrement (UPDATE inventory SET remaining = remaining - 1 WHERE remaining > 0). Because the primary-key constraint on reservation_id lets that insert succeed at most once, the decrement behind it can also run at most once per reservation, so a redelivered message cannot sell a second seat.
5. Status: a small function reads the status table. This is the one synchronous function that sees peak traffic.

Warm-up strategy

Size the status function. Two independent checks apply, and you provision to the larger. First, Little's law (a queueing-theory identity: the average number of requests in flight equals the arrival rate multiplied by how long each one takes to handle) gives the actual concurrency, i.e. how many execution environments are doing work at once: concurrency = RPS x duration. Second, Lambda enforces a request-rate ceiling on the alias equal to 10x whatever concurrency you provision or reserve for it (AWS states this directly for provisioned concurrency: the per-alias request-per-second quota is "10 x allocated provisioned concurrency"). That is not a literal cap on what a single environment can physically do. AWS's own conceptual scaling guide warns against that exact framing: "it's incorrect to say that each Lambda execution environment can handle only a maximum of 10 requests per second." It is a budget tied to the unit count you configure, so under-provisioning starves throughput before it starves headcount. As a conservative upper bound, assume all 10,000 RPS could be status polls, each taking an assumed 50 ms (a single lightweight status-table read): Little's law alone says 500 environments would be busy at once, but provisioning only 500 units caps this alias's own request rate at 10 x 500 = 5,000 RPS, half of what is needed, so the number to provision is the larger of the two:

Provision 1,100 (+10% buffer) on the function's alias, using a scheduled Application Auto Scaling action from T-15 minutes (15 minutes before the on-sale's published start time) to T+30 minutes, then scale back down.

Quota first. The default Regional concurrency quota is 1,000, which is below 1,100. Request an increase weeks ahead, to around 3,000, so the rest of the account still has room during the event.

No "ping" warmers. A scheduled ping keeps one environment warm. It does nothing for 1,000 concurrent requests.

Warm the non-Lambda parts too. Pre-scale the database's capacity. Confirm API Gateway's account throttle. Cache the event page at the CDN before the on-sale.

Make the on-demand overflow cheaper. Keep the package small, create SDK clients at top level, and keep the handler minimal.

Predictive scaling

• Scheduled events (almost all on-sales): the calendar is the forecast. Use scheduled provisioned concurrency, scheduled DB scaling, and a pre-set waiting-room admission rate.
• Sizing inputs: registered interest, previous on-sales for similar demand, and the live waiting-room queue length at T-10 minutes. That last signal is what you use to adjust the admission rate and pre-warm level before the doors open.
• Target tracking on ProvisionedConcurrencyUtilization handles slow drift only. It needs about 3 minutes of sustained load before it acts, so it will miss the first minute, which is the minute that matters.
• Unscheduled spikes (a surprise drop): the waiting room absorbs them. Lambda's on-demand scaling (up to 1,000 new environments immediately, then about 100 per second per function) covers what gets through.

Queue-based throttling

The accept path takes the spike. The worker path runs at a speed you choose:

baseline_rps, multiplier = 200, 50
peak_rps = baseline_rps * multiplier

# Status API (the only synchronous Lambda on the hot path).
# Two independent checks: Little's law for actual concurrency (RPS x duration), and the
# platform's request-rate ceiling, which is 10x whatever concurrency you provision/reserve
# for this alias (not a literal per-environment cap: AWS's scaling guide is explicit that
# framing is wrong). Provision to the larger of the two.
status_s = 0.050
concurrency_needed = peak_rps * status_s
rate_ceiling_floor = peak_rps / 10   # must provision >= this many units so 10x it clears peak_rps
status_units = max(concurrency_needed, rate_ceiling_floor)
print(f"peak {peak_rps} RPS; status-API concurrency (Little's law) = {concurrency_needed:.0f}; "
      f"units needed to clear the 10x rate ceiling = {rate_ceiling_floor:.0f}; provision {status_units:.0f}")

# Reservation workers behind the queue, capped to what the inventory DB sustains
db_writes_per_s, worker_s = 800, 0.040
print(f"worker concurrency cap = {db_writes_per_s} x {worker_s} = {db_writes_per_s * worker_s:.0f}")

tickets, attempts = 50_000, peak_rps * 60
print(f"attempts in a 60 s spike: {attempts:,}; tickets: {tickets:,}; "
      f"sell-out time at {db_writes_per_s}/s: {tickets / db_writes_per_s:.1f} s")

pc, gb, price, window_s = 1_100, 1.0, 0.0000041667, 45 * 60
print(f"provisioned concurrency {pc} x {gb} GB for 45 min: ${pc * gb * window_s * price:.2f}")

slo, reqs = 0.999, peak_rps * 60 * 30
print(f"error budget for {reqs:,} requests at {slo:.1%} success: {reqs * (1 - slo):,.0f} failed requests")

peak 10000 RPS; status-API concurrency (Little's law) = 500; units needed to clear the 10x rate ceiling = 1000; provision 1000
worker concurrency cap = 800 x 0.04 = 32
attempts in a 60 s spike: 600,000; tickets: 50,000; sell-out time at 800/s: 62.5 s
provisioned concurrency 1100 x 1.0 GB for 45 min: $12.38
error budget for 18,000,000 requests at 99.9% success: 18,000 failed requests

• Worker cap = 32. Set it as MaximumConcurrency on the event source mapping, and set reserved concurrency to at least that. The poller then never invokes more workers than the database can serve, which protects the database without anyone writing throttling code.
• Sell-out takes about 62.5 s at 800 reservations per second. After that, a "sold out" flag (config the waiting room and API read) stops admissions, and queued messages are answered "sold out" cheaply. The 600,000-attempt figure is an upper bound if every request in the minute were a purchase. The point is that you must not process 600,000 database writes to sell 50,000 seats.
• Queue hygiene. Visibility timeout at least 6x the function timeout (the time a received message stays hidden before it is redelivered). A dead-letter queue (DLQ, where messages go after repeated failures) with maxReceiveCount of at least 5. Partial batch failure reporting (ReportBatchItemFailures: the function tells the poller exactly which messages in the batch failed, so only those are retried), so one bad message does not retry nine good ones.

Evaluating SLA risk

1. Write per-path SLOs. Accept: p99 under 300 ms at 99.9% success. Confirmation: p95 (95th percentile) under 120 s. Oversell: zero, as an invariant, not a percentage. At 99.9% over a 30-minute peak (18,000,000 requests), the error budget is 18,000 failed requests. That is the number to compare against load-test results.
2. Audit the refusal point of every hop:

3. Load-test at 1.5x forecast (15,000 RPS) with production configuration. Run one test right after a fresh deploy, so you see real cold starts. Add a database failover in the middle of the run.
4. Failure-mode table and kill switches.
   • DB slows: the queue grows, and the confirmation SLO is at risk while the accept SLO is not. Lower the admission rate.
   • Status function throttles: clients back off with jitter (a small random delay added to each retry, so they do not all retry at the same instant).
   • Waiting room fails: API Gateway throttling is the fallback.
     Each lever is a config change you can make in seconds, rehearsed in a game day (a scheduled drill where the team deliberately triggers a failure, such as the database failover in step 3 above, to prove the response actually works).

Cost

Pre-warming 1,100 x 1 GB for 45 minutes costs about $12.38 (from the script above). The same 1,100 environments left on for a full month (730 hours) would cost 1,100 x 1.0 GB x 730 x 3,600 s x $0.0000041667/GB-s, about $12,045 a month. Event-window scheduling is the whole cost story.

Pitfalls

• A synchronous function plus a database write on the accept path. At 10,000 RPS that becomes thousands of database connections and cold starts, exactly when you can least afford them.
• Warmer pings instead of provisioned concurrency.
• Trusting target tracking to react in time.
• Checking Lambda quotas and forgetting API Gateway, the database, or the CDN configuration.
• Treating the queue as an infinite buffer. Buyers are waiting. Tell them their position, and stop admitting once the backlog alone can sell out the inventory.