Serverless and Function-as-a-Service Architecture Questions

Direct answer

First confirm the spikes really are cold starts: compare p99 (the 99th-percentile latency, the value only 1% of requests exceed) for invocations whose logs carry an Init Duration against those that don't. If they are, three practical strategies, from cheapest to most expensive: (1) make initialization cheaper (smaller package, fewer and lazier imports, more memory for more CPU); (2) restore from a snapshot instead of initializing (AWS Lambda SnapStart); (3) pre-initialize environments so users never hit a cold one (provisioned concurrency, or minimum instances on other platforms). Do (1) always, then pick (2) or (3) based on runtime support and how strict the latency target is.

Why cold starts land exactly in p99

AWS reports cold starts are typically under 1% of invocations. p99 is the boundary of the slowest 1%. If 0.5% of requests are cold, the slowest 1% is half cold and half slow-warm, and p99 sits at the edge of the warm tail. If 1.5% are cold (low traffic, frequent deploys, bursts that force new environments), the p99 value itself is a cold request. A modest rise in cold-start rate can therefore move p99 from roughly "slow warm request" to "full cold start". This is why the fix list starts with cutting the cold-start cost, and why reducing how often they happen also counts.

Strategy 1: make initialization cheaper

Mechanism. Cold-start time is roughly code download + runtime start + your init code. Shrinking any term shrinks the spike.

• Import only what you use (a single SDK service client rather than the whole SDK); remove unused dependencies, test fixtures and docs from the package.
• Lazy-load heavy objects needed only on some code paths.
• Raise memory: Lambda allocates CPU in proportion to memory (1,769 MB equals one full vCPU, one virtual CPU core, a unit of allocated compute), so CPU-bound init (parsing, class loading, JIT warm-up, where JIT is just-in-time compilation of hot code) runs faster.

Trade-offs.

Strategy 2: snapshot restore (Lambda SnapStart)

Mechanism. When you publish a function version, Lambda runs Init once, takes a snapshot of the initialized environment's memory and disk (a Firecracker microVM snapshot, Firecracker being the lightweight virtual machine technology Lambda runs on), and caches it. New environments resume from the snapshot instead of re-running Init.

Trade-offs.

Strategy 3: pre-initialized capacity (provisioned concurrency)

Mechanism. You tell Lambda to keep N environments initialized for a version or alias at all times. Requests up to N concurrent never see a cold start; above N, they spill over (run on ordinary on-demand environments instead of the pre-warmed ones) to normal on-demand environments (and can be cold). The equivalents elsewhere are minimum instances on Google Cloud Run and always-ready instances on Azure Functions.

Trade-offs, with numbers. Price in US East (N. Virginia): $0.0000041667 per GB-second for keeping capacity provisioned, and duration drops to $0.0000097222 per GB-second (from $0.0000166667) for work that runs on it.

• 10 environments at 1 GB for a 30-day month: 10 GB x 2,592,000 s = 25,920,000 GB-s x $0.0000041667 = $108.00 per month, even with no traffic.
• If those environments serve 5 million requests of 80 ms at 1 GB (400,000 GB-s), duration costs $3.89 instead of $6.67 on-demand. The $108 floor dominates unless utilization is high.

Worked example: choosing

A checkout API on Node.js with a 250 ms p50 (median latency: half of requests finish faster than this) and 1.8 s p99, 40 requests/s in business hours, 2 requests/s at night; cold-starting requests show ~1.5 s Init Duration.

1. Trim the bundle and replace a whole-SDK import with single clients: Init Duration falls from ~1.5 s to, say, ~600 ms (measure it; don't assume).
2. SnapStart is not available for Node.js, so for the remaining spikes use provisioned concurrency. By Little's law (concurrency equals arrival rate times how long each request takes; here the 250 ms p50 stands in for that per-request time), 40 req/s x 0.25 s means about 10 requests are in flight at once, so provision ~12 (headroom) during business hours via a schedule and ~2 at night. Priced at the same $0.0000041667 per GB-second as above (1 GB, roughly 12 business hours and 12 night hours a day): 12 environments busy 12 h/day plus 2 environments the other 12 h/day, over a 30-day month, is about $75.60/month, cheaper than holding a flat 10 environments 24/7 ($108, from the pricing above) precisely because it drops to 2 overnight.
3. Re-check p99 and the spillover metric after a week.

Pitfalls

• "Keep-warm" pings (a scheduled event calling the function every few minutes) keep one environment warm. A burst that needs 20 environments still cold-starts 19. Provisioned concurrency is the supported version of this idea.
• Blaming cold starts for everything. Slow downstream calls, network set-up for functions attached to a VPC (Virtual Private Cloud, a private network), or large payloads also show up in p99. Split latency by cold versus warm before paying for a fix.
• Deploy frequency matters. Every deployment replaces all environments, so ten deploys a day means ten waves of cold starts.

Direct answer

I would not try to make a serverless system fully portable. That costs a lot and throws away most of what serverless offers. Instead I keep the business logic portable and let the edges be native: put domain code behind a thin handler adapter, ship functions as standard container images, keep data in open formats, and choose portable tools only where a move would otherwise mean a rewrite (mainly workflow orchestration). Then I write down an exit plan with a cost estimate, so lock-in is a known, priced risk instead of a vague fear.

Terms:

• Vendor lock-in: the cost (time, money, risk) of moving off a provider. It is never zero; the goal is to keep it proportional to the value you get.
• Serverless / FaaS (functions as a service): provider-run functions triggered by events and billed per invocation, for example AWS Lambda, Azure Functions, Google Cloud Run functions.
• IaC (infrastructure as code): declaring cloud resources in files that a tool applies, for example Terraform or AWS CloudFormation.
• Workflow orchestrator: a service that runs multi-step processes with retries and state, for example AWS Step Functions.

Where lock-in actually lives

Rank the layers by how expensive they are to move, because that is where portability effort pays off:

The four levers, and my choice for each

1. Portable packaging

• Package functions as OCI container images (OCI, the Open Container Initiative image format that Docker produces). Lambda, Google Cloud Run and Azure Functions (on hosting plans that support custom containers) all run container images, and the same image runs locally for tests.
• For HTTP services, the open-source AWS Lambda Web Adapter lets an ordinary web app (Flask, FastAPI, Express, Spring Boot) run on Lambda unchanged, so the same image also runs on a container platform such as Fargate (AWS's serverless container runtime) or Kubernetes (an open-source system for running and scaling containers across a cluster of machines).
• Cost: depending on image size, container images can have slower cold starts than small zip packages (the platform has to fetch and start a larger artifact), so keep images slim, and measure.
• Choice: container images for anything non-trivial; zip packages only for tiny glue functions where a rewrite would take an afternoon anyway.

2. Workflow orchestration

• Step Functions is excellent (visual, managed, deeply integrated with AWS services) and the definitions are AWS-only.
• A portable alternative is Temporal, an open-source workflow engine where workflows are ordinary code; it runs self-hosted or as a managed cloud service and on any provider. The cost is that you operate a cluster (or pay for the managed offering), and you lose Step Functions' built-in, no-code integrations with many AWS services.
• Choice: Step Functions for short, glue-like workflows (a handful of states, mostly calling AWS services), where a rewrite would be small. Temporal (or equivalent code-first engine) when workflows are the core of the business (long-running order or claims processes with dozens of steps), because rewriting those is the single largest item in any exit plan.

3. IaC portability

• An important nuance candidates often miss: Terraform (or its open-source fork OpenTofu) makes the tool and the workflow portable, not the resources. An aws_lambda_function resource does not become an Azure function by changing the provider. What you gain is one language, one plan/apply workflow (write the desired resources, preview the changes the tool would make, then apply them) and one state model (the tool's own record of what it has already created, so it knows what to change versus create) across clouds, so the team's skills transfer.
• AWS CDK (a framework that generates CloudFormation from TypeScript or Python) and CloudFormation are AWS-only.
• Choice: Terraform/OpenTofu if multi-cloud is a real possibility or the organization already uses it; otherwise use whatever the team is fastest with, and accept that IaC gets rewritten on a move regardless. It is rarely the expensive part.

4. Abstraction layers between code and platform

• Ports and adapters (hexagonal architecture, named because the domain sits in the middle of the diagram with an adapter hanging off each face): domain logic depends on interfaces you own (OrderRepository, EventPublisher); a thin adapter per platform implements them. The handler just parses the provider's event, calls the domain, and formats the response. Concretely:

# handler (adapter): one version per platform, knows only the event shape
def lambda_handler(event, context):
    order_id = event["pathParameters"]["orderId"]
    order = create_order(order_id, OrderRepository(), EventPublisher())
    return {"statusCode": 200, "body": order.to_json()}

# domain (portable): knows nothing about Lambda, API Gateway, or AWS
def create_order(order_id, repo, publisher):
    order = repo.save(order_id)
    publisher.publish("order.created", order)
    return order

Move to a container: write a new container_handler(request) that unpacks an HTTP request instead of a Lambda event and calls the same create_order. create_order itself, and its tests, do not change.

• Standard event envelope: wrap events in CloudEvents (a CNCF, Cloud Native Computing Foundation, specification for describing events in a common format) so consumers do not depend on each provider's payload shape.
• Where to stop: do not wrap managed services in lowest-common-denominator abstractions (a "generic queue" that hides SQS's (Amazon Simple Queue Service, AWS's managed message queue) visibility timeout, how long a delivered message is hidden from other consumers before it is treated as failed and redelivered, and FIFO, first-in first-out, groups). You lose the features you chose the service for, and you still have to rewrite semantics on a move because two queues never behave identically.
• Choice: adapters around your own code, always. Abstraction over managed services, only for the two or three you would realistically swap.

Worked example: an exit-cost inventory

A team runs 30 Lambda functions, 3 Step Functions workflows (8, 12 and 40 states), 4 DynamoDB tables, SQS and SNS (AWS's managed queue and pub/sub topic services), and API Gateway (AWS's managed HTTP front door). An exit plan lists each item with a move strategy and a rough effort, estimated by the team:

Those figures are the team's own planning estimate, not a measurement, and total about 62 engineer-days. The inventory makes the decision concrete: two items, the 40-state workflow (about 15 days) and the DynamoDB migration (about 20 days), are 35 of those 62 days, 56%, most of the exit cost. That is where portability spending pays: moving that one workflow to a code-first engine and keeping the data access behind a repository interface, while leaving the small workflows and glue native.

Weighing the cost of staying portable

Trade-offs and pitfalls

• "Multi-cloud from day one" usually doubles the work for a move that never happens. Portability should be proportional to a real probability of moving (regulation, acquisition, price leverage, a customer requirement).
• Treating Terraform as portability. It is a skills and workflow benefit, not a resource-migration tool.
• Ignoring data gravity, the tendency for code and services to cluster near a large dataset because moving the data, not the code, is the expensive part of a move. Code moves in weeks; terabytes of data in a proprietary data model are the real anchor. Open formats (Parquet, a column-oriented file format for structured data that many query engines can read directly, in object storage) and a repository interface do more for portability than any compute abstraction.
• Forgetting the exit plan is a living document. Revisit it when a new proprietary service is adopted, because each one adds a line to the exit cost.

Direct answer

Each concurrent function environment usually opens its own database connection and keeps it across warm invocations. At 1,000 concurrent functions that is about 1,000 connections, against a relational database configured for a few hundred. (Examples below use AWS: Amazon RDS is the managed relational database service, Aurora its cloud-native engine, and a VPC, Virtual Private Cloud, is the private network both run in.) The fix is a connection pooler between the functions and the database: it accepts many cheap client connections and multiplexes them onto a small, fixed pool of real database connections. Pair it with a cap on function concurrency, so the queue at the pooler cannot grow without bound; size that cap to the assumed peak concurrency (1,000 in the worked example below), not to the pool's connection count. The pooler already handles multiplexing connections; the concurrency cap's job is only to stop a runaway retry storm or a bug from spinning up unbounded environments and overwhelming the pooler's own queue.

• On AWS the managed option is RDS Proxy.
• The self-managed option is PgBouncer (PostgreSQL) or ProxySQL (MySQL) running on containers.
• Connectionless alternatives are HTTP data APIs (for example the Aurora Data API, which runs SQL over an HTTPS call with no persistent connection at all) and serverless-native database drivers (for example the Neon or PlanetScale serverless drivers, which speak a connection protocol over HTTP or WebSocket instead of a raw TCP connection).

Size the pool from the database's real work (Little's law), not from function concurrency. I would use RDS Proxy for RDS or Aurora, plus reserved concurrency as a hard ceiling. I would switch to PgBouncer only if session pinning or cost made the managed proxy a poor fit.

Why serverless exhausts connections

• One connection per environment. Environments don't share memory, so an in-process pool of size 10 per function makes things worse, not better. It just multiplies connections.
• Frozen environments hold their connections. Between invocations an environment is frozen but its TCP connection stays open until the environment is reclaimed or the database's idle timeout kills it. The database sees many idle "zombie" connections.
• Connection storms. A burst creates hundreds of new environments at once, and each performs a TCP + TLS + authentication handshake. PostgreSQL also starts a backend process (it forks a dedicated OS process for each connection, rather than using lightweight threads) per connection. The database spends its CPU on logins, not queries.

The options

Sizing the pool

Assumptions:

• max_connections = 500 on the database
• peak function concurrency 1,000
• 3,000 queries per second averaging 4 ms each

db_max_connections = 500      # set in the DB parameter group (assumption for this example)
lambda_peak_concurrency = 1_000
queries_per_s, avg_query_s = 3_000, 0.004
busy = queries_per_s * avg_query_s                 # Little's law: connections in use on average
burst_factor = 3                                   # assumed ratio of short-burst to average load
needed = busy * burst_factor
with_headroom = needed * 1.3                       # 30% headroom above peak usage
max_pct = 20
pool_cap = db_max_connections * max_pct // 100
print(f"direct connections at peak: {lambda_peak_concurrency} vs max_connections {db_max_connections}")
print(f"average busy DB connections = {queries_per_s} x {avg_query_s} = {busy:.0f}")
print(f"burst need = {busy:.0f} x {burst_factor} = {needed:.0f}; with 30% headroom = {with_headroom:.1f}")
print(f"MaxConnectionsPercent {max_pct} -> pool cap {pool_cap}; client:DB multiplexing {lambda_peak_concurrency / pool_cap:.0f}:1; "
      f"left for admin, migrations and other apps: {db_max_connections - pool_cap}")

direct connections at peak: 1000 vs max_connections 500
average busy DB connections = 3000 x 0.004 = 12
burst need = 12 x 3 = 36; with 30% headroom = 46.8
MaxConnectionsPercent 20 -> pool cap 100; client:DB multiplexing 10:1; left for admin, migrations and other apps: 400

• Only about 12 connections are busy on average. A query holds a connection for 4 ms, and at 3,000 per second the average number in use is 3,000 x 0.004 = 12. That is Little's law, and it is why a pool far smaller than 1,000 works.
• Burst allowance x3 = 36, plus the 30% headroom AWS recommends above peak usage = about 47. MaxConnectionsPercent = 20 (20% of 500 = 100) covers that comfortably. It is also AWS's minimum recommended value for instance classes of db.t3.medium (a small/mid-sized RDS instance class; the letters and number describe the instance family and size, similar to EC2 instance types) and larger. It leaves 400 connections for migrations, admin sessions and other applications. The proxy also reserves a few connections for itself, so read the effective cap from the MaxDatabaseConnectionsAllowed metric.
• Upper bound from the database, not the proxy. A database with N vCPUs can only run about N queries truly in parallel. A pool far above a small multiple of the core count just moves the queue from the proxy into the database, where it costs more.
• Borrow timeout. When all 100 are busy, clients wait at the proxy. ConnectionBorrowTimeout defaults to 120 s. That is longer than any API latency budget, so lower it (for example to 5 s) and fail fast instead of piling up waiting functions.

Placement and latency

• Same Region and VPC as the database, with proxy endpoints in each of the database's Availability Zones. Functions run in private subnets (network segments with no direct route to the public internet) of that VPC. Readers use the proxy's read-only endpoint (a separate address that routes to read replicas instead of the primary, the one instance that accepts writes).
• The added hop is one in-VPC network round trip per query. Measure it by load-testing with and without the proxy rather than assuming a figure. It usually matters less than what it removes: connection setup on cold starts, and connection errors during bursts.
• Cold versus warm.
  • Open the connection once per environment (at top level or lazily on first use) and reuse it on warm invocations.
  • A cold start still pays TCP + TLS + authentication, but to the proxy, which is much cheaper than a database backend starting up.
  • If you use IAM (AWS Identity and Access Management) database authentication, generate the token once and reuse it. Tokens are valid for 15 minutes.
  • Close nothing per invocation. Opening and closing per request brings back the storm.

Pitfalls

• Assuming the proxy removes the need for a concurrency cap. Without a cap, the queue simply moves to the proxy's borrow wait.
• Pinning. Watch DatabaseConnectionsCurrentlySessionPinned. Move identical per-connection SET statements into the proxy's initialization query, and avoid session-level state in handler code.
• Pool sized to function concurrency ("1,000 functions, so 1,000 connections") instead of to query concurrency.
• Failover (the database promoting a standby to primary after the original primary becomes unavailable). The proxy keeps client connections open while it reconnects to the new primary. Test that your queries get errors and retries, not hangs, during a failover drill.

Direct answer

A FaaS platform (functions-as-a-service: you upload a function and the provider runs it once per event) scales by creating more execution environments. An execution environment is a small isolated sandbox with your runtime and code loaded, and it handles one request at a time. So 600 requests in flight at the same moment need 600 environments. When a request arrives and no idle environment exists, Lambda builds a new one first. That is a cold start. Growth is bounded by two different limits:

• a per-function scaling rate: how fast new environments can be created
• a Regional account concurrency quota: how many can exist at once, shared by every function in the account

Reserved concurrency gives a function its own slice of that quota and caps it there. Provisioned concurrency keeps environments already initialized, so requests that land on them skip the cold start. For a latency-sensitive endpoint, the cold starts and throttles all arrive in the same first seconds of a burst. The fix is to pre-warm the endpoint with provisioned concurrency sized to the burst and make sure the quota has headroom before the burst starts.

How a new environment gets provisioned

1. Routing. A request arrives. If an idle, already-initialized ("warm") environment exists, Lambda sends the request there. No setup cost.
2. Init phase (the cold start). If there is no idle environment, Lambda creates one: it loads your code package, starts the language runtime and any extensions, then runs your top-level code. That is everything outside the handler: imports, SDK clients, configuration loading.
3. Invoke phase. The handler runs.
4. Freeze and reuse. After the response, the environment is frozen and kept for later requests. How long it is kept is unspecified. It can be reclaimed at any time, so you cannot rely on it staying warm.

Concurrency means the number of requests being processed at the same instant, and it is also the number of execution environments Lambda needs at that instant. You can compute it with Little's law: concurrency = arrival rate (RPS) x the average time each request spends in the system (duration in seconds). That gives you environments directly; there is no hard per-environment throughput ceiling. AWS's own conceptual scaling guide warns against exactly the framing that sounds intuitive here: "it's incorrect to say that each Lambda execution environment can handle only a maximum of 10 requests per second. Instead of observing the load on any individual execution environment, Lambda only considers overall concurrency and overall requests per second when calculating your quotas." A single warm environment reused back to back for a fast function easily clears 10 requests per second in practice (a 20 ms handler run continuously on one environment serves 50 requests per second, not 10).

What Lambda actually enforces alongside the concurrency quota is a separate REQUEST-RATE quota: RPS must stay at or below 10 times whichever concurrency quota is in force (10,000 RPS against the default 1,000-unit account quota; 10 times a function's reserved concurrency if it has one; 10 times its provisioned concurrency for the units running on it). This is a policy ceiling tied to the quota number, not a physical property of any one environment. For a function whose average duration is at or above 100 ms, Little's law alone already keeps the request rate inside that 10x ceiling, so it never binds. It matters specifically for the sub-100-ms case: a 20 ms function serving 30,000 RPS needs a concurrency, and so an environment count, of only 600, comfortably under a 1,000-unit account quota, but its request rate of 30,000 is three times the 10,000 RPS the default quota allows, so it throttles on the rate ceiling even though the environment count looks fine.

with a second, independent check for functions under about 100 ms average duration:

The limits

The effect of a throttle depends on who called the function:

• Synchronous callers (API Gateway, a direct invoke) get the error immediately. Nothing retries for you.
• Asynchronous sources (S3 notifications, SNS, Amazon Simple Notification Service, a managed pub/sub messaging service) go through Lambda's internal queue, which retries throttled events for up to 6 hours by default.
• Queue pollers (Amazon SQS, Simple Queue Service) leave the messages in the queue to be retried.

Reserved versus provisioned concurrency

If both are set, provisioned concurrency cannot exceed reserved concurrency. Setting reserved concurrency to 0 is the documented way to stop a function completely.

Worked example

Assume an endpoint whose handler takes 120 ms. It normally sees 100 RPS with 12 environments warm, and the account quota is the default 1,000.

# Environments needed = RPS x duration (Little's law: this alone gives the concurrency,
# i.e. the environment count). The separate 10x-quota request-rate ceiling only binds
# for average durations under about 100 ms, which this 120 ms example never reaches.
duration_s = 0.120      # assumed average handler duration for the endpoint
bucket = 1000           # scaling rate: up to 1,000 new environments per function...
refill_per_s = 100      # ...refilled continuously (1,000 per 10 s)
warm_now = 12

def report(rps, account_limit):
    need = rps * duration_s
    cold = max(0, min(need, account_limit) - warm_now)
    wait = max(0, cold - bucket) / refill_per_s
    short = max(0, need - account_limit)
    print(f"{rps:>6} RPS, quota {account_limit:>5}: need {need:>5.0f}; new (cold) environments {cold:>5.0f}; "
          f"seconds beyond the first 1,000 new: {wait:>4.1f}; throttled capacity gap: {short:.0f}")

for rps in (100, 5_000, 15_000):
    report(rps, 1_000)          # default Regional concurrency quota
report(15_000, 3_000)           # after a quota increase

pc_units, mem_gb, price_pc = 600, 1.0, 0.0000041667   # USD per GB-second, us-east-1, x86
hour = pc_units * mem_gb * 3600 * price_pc
print(f"provisioned concurrency {pc_units} x {mem_gb} GB: ${hour:.2f}/hour, ${hour * 730:,.0f}/month if always on")

   100 RPS, quota  1000: need    12; new (cold) environments     0; seconds beyond the first 1,000 new:  0.0; throttled capacity gap: 0
  5000 RPS, quota  1000: need   600; new (cold) environments   588; seconds beyond the first 1,000 new:  0.0; throttled capacity gap: 0
 15000 RPS, quota  1000: need  1800; new (cold) environments   988; seconds beyond the first 1,000 new:  0.0; throttled capacity gap: 800
 15000 RPS, quota  3000: need  1800; new (cold) environments  1788; seconds beyond the first 1,000 new:  7.9; throttled capacity gap: 0
provisioned concurrency 600 x 1.0 GB: $9.00/hour, $6,570/month if always on

What the numbers mean:

• Burst to 5,000 RPS. You need 600 environments. The scaling rate is not the problem: its token bucket starts full at 1,000 (see above), so all 588 new environments can be created immediately from that reserve, with no need to wait for the refill. The problem is that 588 requests each wait for a cold start at the same moment, and that is what p99 latency (the latency 99% of requests beat) will show. Init time depends on runtime and package size. Read it from the Init Duration Lambda logs rather than guessing.
• Burst to 15,000 RPS on the default quota. You need 1,800 environments but only 1,000 can exist. About 800/1,800, or 44%, of the load is throttled.
• Same burst after raising the quota to 3,000. The first 1,000 new environments appear immediately. The remaining 788 arrive at the refill rate of about 100 per second, so over about 7.9 seconds. Requests above capacity during those seconds are still throttled.
• Cost of pre-warming 600 x 1 GB: $9.00 an hour, or about $6,570 a month if it is left on all the time. That is why you schedule it rather than leave it on.

What this means for a latency-sensitive endpoint

1. Provisioned concurrency on the alias the API actually invokes (never on $LATEST, which cannot hold provisioned concurrency). Size it to the expected burst plus a buffer. Lambda's own guidance is +10%, so 600 becomes 660. If the burst is predictable, raise it on a schedule.
2. Reserved concurrency at or above that number. This stops other functions in the account from starving the endpoint. Without it, a noisy batch job (an unrelated function in the same account that suddenly scales hard, for a backfill or a retry storm) in the same Region can use up the shared 1,000.
3. Quota headroom requested ahead of time. Check the layers in front of Lambda too. API Gateway (AWS's managed HTTP front door) has its own default account throttle of 10,000 RPS per Region.
4. Make cold starts cheaper for the overflow. Keep the package small, create SDK clients at top level so warm invocations reuse them, and defer rarely used clients. For supported runtimes, snapshot restore (Lambda SnapStart) is a cheaper alternative to keeping environments warm.
5. For bursts beyond anything you pre-warmed, decide the behaviour explicitly. Either return a fast 429 and have clients retry with jittered backoff (each retry waits longer than the last, with a bit of randomness mixed in so many clients do not all retry at the same instant), or, if the work does not need an immediate answer, accept it into a queue.

Pitfalls

• Invoking $LATEST or the wrong alias. Provisioned concurrency sits unused, you still pay for it, and cold starts continue.
• Treating reserved concurrency as a performance feature. It warms nothing.
• Expecting target-tracking auto scaling (a controller that watches a metric and adds or removes capacity to hold it near a target value) to catch a spike. Application Auto Scaling on ProvisionedConcurrencyUtilization needs the load to last about 3 minutes (three data points) before it adds capacity. It handles slow drift, not a sudden burst.
• Forgetting the separate request-rate ceiling for sub-100-ms functions. Little's law alone gives the right environment count even for a fast function (a 20 ms function at 30,000 RPS needs only 600 environments), but request rate can still hit the platform's 10x-of-quota ceiling before concurrency does: at the default 1,000-unit quota that ceiling is 10,000 RPS, so the 30,000-RPS case above throttles despite having plenty of environment headroom.
• Scale-out has side effects downstream. 600 new environments can mean 600 new database connections. Plan the downstream protection together with the scaling.

Direct answer

For an 800 MB artifact that changes with the code, package the function as a container image: it lifts the size limit from 250 MB to 10 GB, needs no private network, and Lambda loads image contents on demand from a cache, so cold starts track what init actually reads. Choose Amazon EFS (Elastic File System, a shared network file system) when the artifact changes independently of the code or is shared by several functions. Avoid downloading from S3 at every cold start: the arithmetic below shows 800 MB alone takes over 10 seconds per cold start. Move to a managed always-on endpoint outside Lambda when traffic is steady enough that you pay for idle capacity anyway, when you need a GPU, or when no cold start is acceptable.

First, a correction to the premise: /tmp on Lambda is configurable from 512 MB up to 10,240 MB, so 800 MB can fit there (extra ephemeral storage is billed per GB-second). The real problems are getting 800 MB there on every cold start, and the memory needed if the artifact is then loaded into RAM.

The constraints that decide it

• Package limits. A zip deployment, including layers, may be at most 250 MB unzipped; a container image may be up to 10 GB uncompressed.
• Init time limit. On-demand Init (module-level set-up) gets 10 seconds; if it overruns, Lambda retries Init inside the first invocation, bounded by the function timeout.
• Network bandwidth. Lambda's documented network bandwidth per execution environment is 625 Mbps (megabits per second, 10^6 bits). Anything fetched over the network at cold start is bounded by it.
• Memory. If the artifact must sit in RAM, configured memory must hold it plus the runtime and working space, and you pay for that memory on every invocation.

Worked example: how long does 800 MB take to fetch?

Lambda's "MB" is 2^20 bytes, so 800 MB = 800 x 1,048,576 x 8 = 6,710,886,400 bits. At the 625 Mbps ceiling:

That is the best case, before any decompression or parsing, and it already exceeds the 10-second Init cap. Every new environment pays it, so a burst that needs 50 new environments downloads 50 x 800 MB = 40,000 MB.

The four options compared

flowchart TD
    A[800 MB artifact] --> B{Changes with code releases?}
    B -- yes --> C{Traffic steady or GPU needed?}
    B -- no, updated separately or shared --> E[EFS mount, read lazily]
    C -- no, spiky --> D[Container image]
    C -- yes --> F[Managed always-on endpoint]

Recommendation and what would change it

Default: container image. It is the only option that removes the size limit without adding a network dependency at start-up that you have to operate. Put the artifact in an early image layer that rarely changes and your code in the last layer, so ordinary code changes don't re-upload the artifact.

• Switch to EFS if the artifact updates on its own schedule (for example daily data refreshes) and rebuilding an image each time is unacceptable, or several functions share it. Structure access to read only what each request needs, and provision throughput for peak concurrency.
• Switch to a managed endpoint once sustained load makes always-on replicas cheaper than per-invocation billing, or when the artifact needs a GPU or must stay in memory without per-environment duplication.
• Use S3 download only for rare background jobs where a slow cold start does not matter, and then fetch during the invocation (subject to the function timeout) rather than squeezing it into the 10-second Init.

Pitfalls

• RAM, not just disk. If the 800 MB is fully deserialized, meaning converted from the file's on-disk bytes into in-memory objects your code works with directly, you may need well over 1 GB configured, paid on every invocation. Memory-mapping keeps resident memory closer to what is actually touched.
• EFS throughput exhaustion. Many environments starting at once can exhaust burst capacity, the reserve of extra throughput EFS's bursting mode grants based on file-system size and lets you spend down temporarily; the symptom is init times that stretch only during scale-outs.
• Inactive container functions. A container-image function not invoked for weeks becomes inactive; the next call fails while Lambda re-optimizes the image. Scheduled invocations or monitoring the function state avoids surprise.