Amazon Senior Solutions Architect Interview Preparation Guide

Direct answer

I would stop new over-permissioning at the source with organization-level guardrails and paved-road defaults, then shrink existing excess using usage data, so that least privilege is the easy default rather than a periodic cleanup. Developers keep self-service inside a safe boundary, so security stops being a ticket queue.

Mechanisms, in the order I would roll them out

1. Organization guardrails (SCPs, service control policies: a permission ceiling no account below can exceed): deny the things nobody legitimately needs, such as disabling audit logging, creating long-lived access keys for users, or using unapproved regions.
2. Permission boundaries for delegated role creation. Teams may create their own roles, but only up to a maximum set of permissions, so self-service cannot become privilege escalation. A permission boundary is a policy attached to a role that sets its ceiling without granting anything: effective access is only what both the role's own policies and the boundary allow. For example, with a boundary that allows only s3:*, dynamodb:* and sqs:*, a team role that was given iam:* and s3:* ends up with S3 access only, because IAM actions fall outside the ceiling.
3. Paved-road templates. A paved road is the platform team's pre-approved, ready-made way to build and deploy a service, so the secure choice needs no extra effort. The standard infrastructure-as-code module (a reusable template that defines cloud resources as code) for a new service creates its own role with only listed actions on named resources and short-lived credentials. The easiest path is the safe path.
4. Policy checks in the pipeline. A change that adds a wildcard action or grants new access beyond an approved baseline fails the pull request (cloud providers offer policy-validation and policy-comparison checks; AWS IAM Access Analyzer provides both).
5. Right-sizing from usage. Unused-access findings (roles, keys and permissions not used within a window you set, such as 90 days) open automatic pull requests to remove them. A typical before and after, using illustrative names:

Before: Action "s3:*"                          Resource "*"
After:  Action ["s3:GetObject","s3:PutObject"]  Resource "arn:aws:s3:::orders-uploads/*"

The access logs showed the service only ever read and wrote objects in one bucket, so everything else could go.
6. Exceptions expire. Broad access is granted with an owner and an end date, enforced by the platform, rather than reviewed by hand.

Patterns across service types

Keeping it valid over time

Policies live in version control; the pipeline blocks regressions; unused-access findings flow to owners automatically; and a small sample audit (not a full review) checks the automation is working.

Not slowing developers

Measure request-to-working-permission time. A boundary that lets a team self-serve in minutes removes the incentive to ask for admin "just in case".

Existing creep

Do not try a big-bang cleanup. Fix the most dangerous first: roles with administrator-equivalent rights and cross-account trust (a role that principals in another cloud account are allowed to assume), then the unused ones, with usage data proving removal is safe.

Direct answer

For 200 million monthly active users (MAU) and 10,000 writes/sec, a pure fan-out-on-write pushes every new post into every follower's feed at write time, buying very low read latency at the cost of massive write amplification and storage. Pure fan-out-on-read defers that work to feed-view time, keeping writes cheap but making every read do more work. A celebrity account with millions of followers breaks the pure push model outright, which is why the practical answer is a hybrid: push for ordinary accounts, pull (or a separate merge step) for very high-fan-out accounts.

Structured elaboration

The decision criterion is the read:write ratio implied by a given account's follower count, not a single global choice: an account followed by 200 people generates trivial fan-out and huge read-latency benefit from push; an account followed by millions generates enormous fan-out for a benefit (marginally faster reads for those followers) that pull-at-read can approximate at read time instead.

Worked example

At 10,000 writes/sec, assume (illustrative, pinned input) an average of 300 followers per post for non-celebrity accounts:
$\text{fan-out ops/s} = 10{,}000 \text{ writes/s} \times 300 \text{ avg followers} = 3{,}000{,}000 \text{ inbox writes/s}$
That is the write-amplification cost a pure push model pays continuously just for ordinary accounts.

Now take one celebrity post going to 5 million followers, and assume (illustrative) a fan-out cluster capable of sustaining 500,000 inbox writes/s:
$\text{time to fan out one celebrity post} = \frac{5{,}000{,}000 \text{ followers}}{500{,}000 \text{ writes/s cluster capacity}} = 10 \text{ s}$
A single celebrity post would take roughly 10 seconds to fully propagate through push fan-out, and that's before accounting for every other post competing for the same fan-out capacity at the same time. This is the concrete reason celebrity accounts change the answer: pushing their posts synchronously into millions of inboxes is not just expensive, it measurably delays delivery to everyone else sharing that fan-out capacity.

Trade-offs & pitfalls

• Treating fan-out-on-write and fan-out-on-read as a single global choice, rather than a per-account decision keyed on follower count, is the most common shallow answer.
• A hybrid design still needs a merge step at read time for celebrity posts, so pull-style merge logic doesn't disappear; it just gets scoped to a small fraction of accounts instead of all of them.
• Async, idempotent fan-out pipelines are required regardless of strategy, because retries and partial failures are certain at this scale; a synchronous fan-out-on-write implementation is a reliability risk independent of the storage trade-off.
• Caching the celebrity's own recent posts aggressively (rather than fanning them out) reduces the read-time merge cost without reintroducing full push fan-out.

A centralized log pipeline for EC2, ECS (Elastic Container Service)/EKS (Elastic Kubernetes Service), and Lambda across accounts routes everything through Amazon CloudWatch Logs first, since every one of those compute types can write there natively, subscribes each log group to Amazon Kinesis Data Firehose, and lets Firehose fan out to Amazon S3 for durable, cheap long-term storage and to Amazon OpenSearch Service for searchable, near-real-time indexing. Keep the security team's access and developers' access separated by index and IAM policy, not by pipeline: everyone's logs flow through the same infrastructure, but who can query what is enforced at the OpenSearch and S3 layer. Alert fatigue gets controlled by being deliberate about what actually pages someone, not by how much data flows through the pipeline.

Pipeline

flowchart LR
    EC2["EC2 (CloudWatch agent)"] --> CWL["CloudWatch Logs\nlog groups per service/env"]
    ECS["ECS/EKS (log driver / Fluent Bit)"] --> CWL
    Lambda["Lambda (native)"] --> CWL
    CWL -- subscription filter --> Firehose["Kinesis Data Firehose"]
    Firehose --> Enrich["Transform Lambda\nadd metadata, dedupe id"]
    Enrich --> S3["S3\ndurable, lifecycle to Glacier"]
    Enrich --> OS["OpenSearch Service\nindexed, searchable"]
    OS --> SecIdx["security-* indices\nrestricted access"]
    OS --> DevIdx["app-* indices\ndeveloper read access"]
    CTacct["CloudTrail S3 (audit account)"] -.cross-account read via assumed role.-> SecIdx

Components

• CloudWatch Logs is the landing zone for every compute type: EC2 through the CloudWatch agent, ECS/EKS through the awslogs log driver or a Fluent Bit sidecar or daemonset, Lambda natively. Keep log group retention short, seven to thirty days, since this is the "live tail" copy, not the archive.
• A subscription filter on each log group streams new log events to a shared Kinesis Data Firehose delivery stream, the fan-out point: Firehose writes to both S3 (the durable backup, partitioned by date and service) and OpenSearch (the searchable index), with an optional processing Lambda in between to enrich each record with service name, task or pod ID, and a deterministic event ID for de-duplication.
• OpenSearch indices are named and separated by data sensitivity, for example security-<service>-YYYY.MM.DD versus app-<service>-YYYY.MM.DD. Index Lifecycle Management rolls indices from hot to warm to cold and eventually deletes them, while S3 lifecycle rules move the same data to Glacier and expire it on a longer horizon for compliance retention.

Scoping access: security team vs. developers

• Enforce the split at the OpenSearch layer with fine-grained access control (role mappings tied to index name patterns) rather than trying to filter at ingestion time: security gets read access to security-* indices plus broad read across app-* for investigation, while each developer gets read access scoped only to their own service's app-<service>-* indices.
• For cross-account CloudTrail visibility specifically, this application-log pipeline is not the audit trail of record. CloudTrail should already be centralizing to its own S3 bucket in a dedicated logging account; the security team reads that bucket through a cross-account audit IAM role rather than relying on this pipeline to carry CloudTrail data. Keep the two conceptually separate even if they share infrastructure patterns, since mixing audit logs into the same OpenSearch domain as application debug logs makes both harder to govern.
• IAM policies on the OpenSearch domain and the S3 bucket should mirror the index-level split, so a developer's IAM role literally cannot read the security-only S3 prefixes even if they guessed the index name.

Avoiding alert fatigue

• Alert on symptoms the business cares about, an elevated server-error rate, a security-relevant API call, a pod crash loop, not on every line a service logs at error level; a noisy dependency that logs a retry as an error should not page anyone.
• For ECS/EKS specifically, container restarts and task replacements are routine during deploys and Auto Scaling events. Alert on a rate of restarts over a window, for example five restarts in ten minutes, instead of any single restart event, since a single-event alert fires constantly during normal deploys and trains the on-call rotation to ignore pages.
• Tune OpenSearch-based alerting, or CloudWatch Logs metric filters feeding CloudWatch Alarms, to use sustained thresholds over a rolling window rather than instantaneous spikes, and route low-urgency findings to a dashboard or ticket instead of a page.

Worked example

A company runs a mix of forty EC2 instances, an EKS cluster, and fifteen Lambda functions across three AWS accounts. All of it streams to CloudWatch Logs, subscribed to one Firehose delivery stream per account, delivering to a shared OpenSearch domain in a central logging account with security-* and app-<service>-* index patterns, plus S3 for the durable copy. Security's IAM role can query security-* and read-only across all app-*; each developer's IAM role can only query their own service's app-<service>-* indices, enforced by OpenSearch fine-grained access control mapped to their IAM role. When an EKS pod crash-loops during a bad deploy, a CloudWatch Alarm on "pod restarts greater than five in ten minutes" pages the on-call engineer, while the single restarts that happen during every normal rolling deploy generate no alert at all, because the threshold is a rate, not a count of one.

Trade-offs and pitfalls

• Sending every log line to OpenSearch is expensive and creates index mapping bloat. Sample or filter verbose debug logs before they reach Firehose, and rely on the S3 copy, cheaper and still queryable through Amazon Athena, which runs SQL queries directly against data in S3, for anything that does not need real-time search.
• CloudWatch Logs subscription filters into Firehose deliver at-least-once, not exactly-once. Without a deterministic event ID and an upsert into OpenSearch using that ID as the document ID, a Firehose retry after a transient failure creates duplicate log entries.
• A shared OpenSearch domain across teams is operationally simpler than one domain per team, but a noisy service's log volume can degrade query performance for everyone else on the same domain. Watch shard count and cluster resource pressure as adoption grows.
• Do not conflate this application-log pipeline with the CloudTrail audit pipeline just because they share Firehose and S3 patterns. They have different retention, access, and integrity requirements, and CloudTrail needs tamper-evidence and log file validation that an application-log pipeline typically does not.

The design combines two layers that solve different failure modes: multi-AZ redundancy within a primary region to survive the common case (a host, rack, or single-AZ failure) with near-zero disruption, and a warm standby in a second region, kept current via continuous replication, to survive the rare but severe case (a full regional outage). Trying to hit 99.99% and a 5-minute RPO (recovery point objective: the maximum data, measured in time since the last durable copy, you can afford to lose) with only one of those layers doesn't work: multi-AZ alone doesn't survive a regional disaster, and cross-region alone (without in-region redundancy) means every AZ blip forces an unnecessary cross-region failover.

Architecture

flowchart TB
  subgraph Primary["Primary region (us-east-1)"]
    LB1[Load Balancer]
    AZ1[App tier - AZ1]
    AZ2[App tier - AZ2]
    AZ3[App tier - AZ3]
    DBP[(Primary DB\nsynchronous multi-AZ)]
    LB1 --> AZ1 & AZ2 & AZ3
    AZ1 & AZ2 & AZ3 --> DBP
  end
  subgraph DR["DR region (eu-west-1)"]
    LB2[Load Balancer - standby]
    AZS[App tier - warm, scaled down]
    DBS[(Standby DB replica)]
    LB2 --> AZS --> DBS
  end
  DBP -- "async replication\n(CDC / log shipping)" --> DBS
  DNS[DNS / Global routing] --> LB1
  DNS -. failover .-> LB2

Redundancy strategy. Inside the primary region, the app tier runs across three AZs behind a load-balanced, health-checked pool (an AZ failure just removes that AZ's capacity from rotation), and the database runs synchronous or semi-synchronous multi-AZ replication so an AZ failure promotes a same-region replica with no data loss. This layer alone handles the overwhelming majority of real infrastructure failures.

Database replication approach. In-region: synchronous, to keep in-region failover at RPO≈0. Cross-region to the DR site: asynchronous (via change-data-capture or log shipping), because synchronous cross-region commits would add tens of milliseconds of write latency for every transaction just to protect against an event, a full regional outage, that's far rarer than an AZ blip. The async lag is what has to stay under the 5-minute RPO budget, so it's monitored continuously with an alert threshold well below 5 minutes (for example, paging at 60 seconds of lag) so operators have room to react before the budget is actually at risk.

Failover mechanics. In-region (AZ failure): automatic, health-check-driven, sub-minute, no human involved. Cross-region (regional failure): health checks on the primary region trigger an automated runbook that promotes the DR replica to primary, scales the warm (already-running, just smaller) app tier in the DR region up to full capacity, and updates DNS/global routing to point at the DR region's load balancer. The DR app tier is warm, not cold, specifically so this promotion is a scale-up operation, not a from-scratch provision, which is what keeps the cross-region RTO (recovery time objective: how long restoring service takes after a failure) in the minutes range instead of the tens-of-minutes range a cold standby would need.

Worked example: why the SLA is bounded by the DR path, not the multi-AZ path

Pin illustrative per-AZ availability at 99.9% for a single AZ's app-tier capacity. With three independent AZs, the app tier only fails when all three fail simultaneously:

That's a downtime budget of about $525{,}600 \times 10^{-9} \approx 0.0005$ minutes/year from independent AZ failures alone, far inside the 99.99% target's 52.56-minute/year budget. The conclusion this points to: independent-component math says multi-AZ redundancy alone should comfortably clear 99.99%, so in practice the SLA is bounded by things that math doesn't model, correlated failures, a bad deploy, a full regional outage, human error, not by an AZ dying in isolation. That's precisely why the DR region exists: it's insurance against the failure modes the availability formula can't see, not an incremental nudge on an already-tiny independent-failure number.

Trade-offs and pitfalls

The most common wrong turn is treating the independent-AZ math above as if it were the whole SLA story and skipping the cross-region layer as "redundant redundancy"; it's protecting against a different, uncorrelated failure class entirely. A second pitfall is under-provisioning the warm standby: if "warm" means a token instance that can't actually absorb full production load once promoted, the design has a theoretical RTO that doesn't survive contact with a real regional failover, so the standby's scaled capacity needs to be load-tested, not just health-checked. A concrete way this shows up: if the actual mandate is something like "cut our current 4-hour cold-standby RTO down to 15 minutes on a fixed budget," the honest trade-off conversation is that a full active-active design gets there fastest but costs the most and adds multi-writer complexity, while a well-automated warm standby (this design) gets to a low-single-digit-minutes RTO for a fraction of the cost, which is usually the better fit unless the traffic is latency-sensitive and truly global. Finally, none of this replaces DR drills: an automated runbook that's never been exercised against a real regional cutover is the same "theoretical RTO" problem as an untested pilot-light tier, just at higher stakes.

Direct answer

Show the moment your evidence, not your title or persistence, changed what leadership decided to do, and be precise about what specifically shifted (a roadmap priority, a budget line, a technical approach) as a direct result of what you brought them. The strongest version has a clear before (what leadership planned to do) and after (what they did instead because of your input).

How to build the case

• Lead with evidence, not opinion: pair a quantitative signal (usage data, error rates, funnel drop-off) with a qualitative one (user quotes, incident detail, direct observation), one alone is easier to dismiss.
• Address the standing objection directly: name the reason leadership was leaning the other way (cost, timeline, competing priority) and show how you specifically answered it, rather than only restating your own case louder.
• De-risk the ask: a prototype, pilot, or small experiment that shows early signal before asking for the full commitment makes the change easier to approve than a request based on projection alone.
• This scales: the same shape (evidence, a direct answer to the standing objection, a way to de-risk the ask) sits behind a smaller "changed the sprint plan" story and a larger "got executive sponsorship for a multi-month investment" story, only the size of the audience and the ask differs.

Worked example (skeleton)

Situation: leadership was planning to prioritize new-feature marketing pushes; I believed drop-off in an early funnel step was costing more than those pushes would gain.
Task: make the case to reprioritize.
Action: I pulled the funnel data (drop-off at that step was roughly double the next-worst step), ran five quick user sessions that surfaced a specific trust concern at that exact point, and built a lightweight prototype of a fix rather than only describing it. I brought a one-page brief to the planning review and addressed the standing objection directly: "this doesn't have to compete with the marketing work, it's a two-day fix we can land first."
Result: leadership moved the fix ahead of the marketing work for that sprint. After it shipped, completion at that funnel step rose from 48 out of 100 sessions to 66 out of 100 over the following two weeks, measured from the same analytics view used to make the original case.

Trade-offs and pitfalls

• Bringing only a strong opinion with no evidence, or data with no answer to the specific objection leadership actually has, both tend to stall rather than change the decision.
• Overselling the size of the shift: if the "direction change" was really a minor scheduling tweak, calling it a strategic pivot invites a skeptical follow-up you can't support.
• Taking sole credit when the decision was genuinely a group call; name who else weighed in and what your specific contribution was to the outcome.

Direct answer

Layer the cache so most of the 10,000 requests per second (RPS) never reach the database at all: an edge content delivery network (CDN) for public, cacheable responses, a regional distributed cache (Redis or Memcached) as the authoritative read cache per region, an optional small local in-process cache for the very hottest keys, and database read replicas underneath everything as the last line of defense for the reads that still get through. The 95th-percentile (P95) latency target of 50 milliseconds is achievable because each layer above the database answers in single-digit milliseconds; the design work is mostly about which layer owns which slice of the traffic, how eviction and time-to-live (TTL) are set per data shape, and how the system behaves when a cache entry expires or a whole region briefly loses its cache.

Topology

flowchart LR
    U[Client] --> CDN[Edge CDN]
    CDN -->|miss| Regional[Regional Redis cluster x3 regions]
    Regional -->|miss| App[App server: local hot-key cache]
    App -->|miss, singleflight| DB[(Primary DB)]
    App -->|read-heavy fallback| Replica[(Read replicas)]
    DB --> Regional
    Regional --> CDN

A 3-region topology, one regional Redis cluster per region, is the natural shape once traffic is global: each region's application servers read from their own regional Redis cluster first, so a cross-region round trip only happens on a true miss. Cache-warming keeps this useful even right after a deploy or a regional failover: instead of a cold Redis cluster taking the full brunt of traffic, a warm-up job pre-populates the known hot key set from a snapshot or from the surviving region before that region takes production traffic. The failover mechanics themselves (how a region is declared unhealthy, how traffic is rerouted, what replication mode feeds the standby) belong to the availability and disaster-recovery side of the system; what matters here is only that cache-warming is the piece that keeps a freshly-promoted region from serving a wall of cache misses.

Database read replicas sit at the bottom of the hierarchy, not because they are unimportant, but because by design most traffic should never reach them: they exist to absorb the read load that the cache layers above did not catch, keeping the primary database free for writes.

Sizing the read/write mix and the key-popularity shape

The exact read/write ratio depends on the dataset, and it changes both the TTL strategy and the stampede-protection budget:

• An inventory-style catalog might run close to 10,000 reads/s against 100 writes/s, a roughly 99:1 read-heavy mix, where staleness is cheap because writes are rare.
• A more write-active product catalog might run closer to a 95/5 read/write split, where the TTL-length trade-off is tighter: a long TTL serves more reads from cache, but a 5% write rate means more of what's cached is out of date at any moment, so the acceptable TTL shrinks compared to the 99:1 case.

Traffic is also rarely uniform across the catalog. With 10 million products, a Zipfian distribution where roughly 95% of requests hit the top 10% of products is typical for product-detail pages: this is the number that justifies keeping a small, aggressively-warmed hot-key set (the top 10%) rather than trying to cache the long tail with the same priority.

Eviction and TTL strategy

• Use least-recently-used (LRU) eviction in the regional cache with a memory cap, since it is a reasonable default when access patterns are Zipfian: the hot 10% naturally stays resident.
• Tier TTLs by how often data actually changes: a product whose price or stock updates roughly per-minute needs a TTL well under a minute, or event-driven invalidation on write, rather than sitting on a multi-minute TTL that would show stale stock counts. Slower-changing fields (description, images) can carry a much longer TTL.
• Pair short-TTL data with a background cache-priming job for the known hot set, so the top 10% of products refresh proactively instead of every expiration turning into a cold miss during peak traffic.

Cache stampede protection

When a hot key expires, do not let every one of its concurrent readers hit the database at once. Use request coalescing (a "singleflight" pattern): the first request after a miss acquires a per-key lock and performs the real fetch, while concurrent requests for the same key wait for that result instead of issuing their own database queries. Combine this with serving the previous, slightly stale value while the refresh happens in the background rather than making every waiter block, which is what keeps the P95 latency target intact even during a miss.

Masking a slow downstream dependency

Suppose part of the product-detail response depends on a personalization or pricing microservice with 200-500ms of latency, far too slow to hit on every request at a 50ms P95 target. Caching absorbs this by storing that dependency's response keyed by product, so only the first request (or the background refresh) pays the 200-500ms cost. Cache-key design has to account for authenticated versus anonymous traffic here: an anonymous (logged-out) request can share one cache entry per product, but an authenticated request that gets a personalized result cannot be keyed by raw user ID without effectively defeating the cache (one entry per user, near-zero reuse). A workable middle ground is keying authenticated responses by product plus a coarse user segment (for example, a pricing tier or locale) rather than by individual user identity, which keeps the cache's hit rate high while still respecting personalization.

Fragment caching for personalized pages

A product page usually mixes shared page components, images, description, review counts, that are identical for everyone, with personalized widgets, like "recommended for you", that are not. Caching the whole page as one unit forces a choice between caching nothing (because of the personalized part) or caching stale personalization (because of the shared part). Fragment (partial) caching avoids that trade-off: cache the shared shell as one fragment with a longer TTL, and render or fetch the personalized widgets separately with their own short-TTL or no-cache treatment, so the expensive-to-personalize slice does not force the whole page out of the cache.

Instrumentation

Track, per layer (CDN, regional, application): hit ratio, origin request rate, P95 and P99 (99th-percentile) latency, count of stampede/lock-contention events, and TTL distribution against actual data-change frequency. Alert on a sudden hit-ratio drop or an origin request-rate spike, since both are early warning signs that a cache layer stopped doing its job, whether from a bad deploy, an expired warm-up job, or a traffic shape shift away from the assumed Zipfian pattern.

Trade-offs and pitfalls

Serving a stale value during a background refresh trades a small amount of freshness for a large amount of latency and database protection, which is almost always the right trade at this scale, but it has to be a deliberate choice with a bounded staleness window, not an accident of a missing invalidation path. The most common design mistake is applying one TTL policy to the whole product object instead of splitting it by field volatility (price and stock versus description and images), which either stales the fast-moving fields or needlessly re-fetches the slow-moving ones. A second common mistake is keying personalized cache entries by raw user identity, which looks correct in a demo with few users and quietly collapses the cache hit rate once real traffic and real personalization variety show up.

This is a rate-limiter, but the thing being limited is release APPROVALS rather than requests, and the limit itself should shrink as either the global or the service-level error budget burns down.

Structured elaboration

Data model: a per-service record tracks {service_id, hard_cap_per_hour, budget_remaining_pct, rolling_approval_count, window_start_timestamp}, refreshed from the same error-budget system that computes SLO burn; a single global record tracks the analogous {global_hard_cap_per_hour, global_budget_remaining_pct, rolling_global_approval_count} shared across all services; and every approval decision (including overrides) is written to an append-only approval-log table {service_id, timestamp, requested, allowed, cooldown, manual_override, justification} so the audit trail exists independent of the current-state records, which only hold the latest rolling counts. Two caps apply simultaneously: a hard per-service cap (protects that service from too-frequent changes) and a hard global cap (protects shared deploy infrastructure and blast radius across all services); the function must always honor the tighter of the two. On top of the hard caps, each cap is scaled down as its corresponding budget burns: above 50% remaining budget the full hard cap applies, and below 50% it ramps linearly to zero at 0% remaining, so approvals get progressively harder to obtain exactly when the service or the whole fleet is already unhealthy. A manual override exists for genuine emergencies (a critical security patch), but it must still respect the hard caps (never grant more than the absolute ceiling) so a panicked override can't itself become the cause of an outage.

Worked example (executed; python3, verified against 5 scenarios)

import math

def approve_release(service_id, requested_approvals_per_hour, global_budget_remaining_pct,
                     service_budget_remaining_pct, global_cap_per_hour=20,
                     service_cap_per_hour=5, manual_override=False):
    def budget_scaled_cap(hard_cap, remaining_pct):
        if remaining_pct >= 50:
            return hard_cap
        return math.floor(hard_cap * (remaining_pct / 50.0))

    if manual_override:
        return min(requested_approvals_per_hour, global_cap_per_hour, service_cap_per_hour), 0

    g_cap = budget_scaled_cap(global_cap_per_hour, global_budget_remaining_pct)
    s_cap = budget_scaled_cap(service_cap_per_hour, service_budget_remaining_pct)
    allowed = max(0, min(requested_approvals_per_hour, g_cap, s_cap))
    binding = min(g_cap, s_cap)
    if binding <= 0:
        cooldown = 60
    elif requested_approvals_per_hour > binding:
        cooldown = math.ceil(60 / max(binding, 1))
    else:
        cooldown = 0
    return allowed, cooldown

Confirmed behavior: a request for 3 approvals/hour with both budgets healthy passes through unchanged (allowed=3, cooldown=0). A burst request of 10/hour against a healthy 5/hour service cap is capped at 5 with a 12-minute suggested cooldown between bursts. When the service budget alone drops to 10% remaining, the effective service cap shrinks to 1/hour even though the request and the global budget are unchanged, correctly localizing the throttle to the unhealthy service. A manual override at 5%/5% budget still caps at 5 (the hard service cap), not the requested 10, so the override cannot exceed the absolute ceiling.

Trade-offs and pitfalls

A pure per-hour cap can be gamed by bursting right at the top of every hour boundary; a production version should use a sliding window (or a token-bucket refill) rather than a fixed hourly reset. The linear budget-scaling ramp is a simplification; teams often prefer a steeper (e.g. quadratic) ramp near zero so the last sliver of budget is protected much more aggressively than the middle of the range. Logging every override with its justification is not optional: a bypass mechanism with no audit trail is exactly the kind of control that gets silently abused under deadline pressure.

Direct answer

Don't choose between a managed time-series database (TSDB) and an in-memory cache for high-frequency telemetry: use both, in a layered architecture, because they solve different halves of the requirement. The cache (Redis or similar) serves the most recent data point per series with sub-millisecond latency for dashboards and alerting, and the TSDB durably stores the full-resolution history and serves the short-term analytical queries (trends, aggregations over the last hour or day) that a cache was never designed to answer well.

Why neither one alone is enough

An in-memory cache alone is excellent at "what is the latest value for this sensor," but has no native concept of a time series: computing "the average over the last 15 minutes" means the application reading back a window of raw values and aggregating them itself, and a cache-only design either has to hold significant history in memory (expensive, and gone on restart unless persistence is configured) or push that history somewhere else anyway. A managed TSDB alone is built for exactly that history-and-aggregation problem, but even a well-tuned one adds latency for the single most common telemetry query (the current value), because it's optimized for range and aggregate queries, not point lookups on the newest row.

Proposed architecture

This is the same latency-versus-query-flexibility trade-off that shows up across database selection generally: the cache buys latency by deliberately supporting fewer query shapes, and the TSDB buys query flexibility by accepting higher latency than an in-memory lookup.

What this gets right, and where it can go wrong

The combination gives the best of both stated requirements (latency and query flexibility) precisely because neither system is asked to do the other's job: alerting and live dashboards read the cache and never touch the TSDB's query planner, while trend analysis and reporting read the TSDB and never contend with the cache's memory budget. The risk this design introduces is consistency between the two: if the fan-out write to the cache succeeds but the TSDB write fails (or vice versa), the "latest value" and "historical record" can briefly disagree. For most telemetry use cases that's an acceptable trade (a dashboard being a few seconds stale during a rare write failure is far less costly than adding synchronous coordination between the two writes), but it should be a deliberate decision, with monitoring on write failures to either side, not an unexamined gap.

Trade-offs and pitfalls

• Don't let the cache become an unintentional second system of record. If the cache is configured to persist and hold significant history, the team now has two places storing overlapping data with two different consistency and durability models, without having decided that on purpose.
• Downsampling policy needs to be decided before storage costs become a problem, not after. High-frequency telemetry at full resolution grows fast; deciding how long raw-resolution data is kept versus rolled up is part of the architecture, not an afterthought.
• A dual-write (cache and TSDB) without a plan for partial failure will eventually produce a real inconsistency. Decide explicitly what happens if one write succeeds and the other doesn't (retry, alert, accept and reconcile later), rather than assuming it won't happen.
• If the workload's real requirement turns out to be complex real-time aggregation (not just "latest value" and "recent trend"), a stream-processing layer (Kafka Streams/Flink) in front of both stores may be a better fit than pushing that logic into either the cache or the TSDB directly.

Direct answer

Monitor at two levels that answer two different questions: per-shard metrics tell you which specific shard is unhealthy, and cluster-wide aggregates tell you whether the cluster as a whole is meeting its service-level objective (SLO, the target level of service the team has committed to). Alert on the aggregate for paging (nobody should be paged because one shard out of two hundred has a blip), and keep the per-shard view for diagnosis once a page fires. Thresholds should be set relative to each shard's own healthy baseline, not one global number, since shards can have legitimately different steady-state load.

Metrics to collect

Per-shard:

• Query latency, p50/p95/p99 (95th and 99th percentile response time, the tail behavior that matters more than the average for user-facing impact)
• Queries per second (QPS) and the read/write split
• Replication lag to that shard's replicas (for example Postgres's pg_stat_replication.replay_lag, the delay between a write committing on the primary and becoming visible on a given replica)
• Disk utilization (both space used and IO throughput/IOPS, input/output operations per second, since a shard can be IO-saturated well before it runs out of space)
• Connection count against the configured limit
• Lock wait time and deadlock rate (how often two transactions end up blocking each other in a cycle, forcing the database to kill one so the other can proceed)

Cluster-wide:

• Aggregate QPS and error rate across all shards
• Number of shards currently unhealthy (below their own baseline) as a single number, so a dashboard shows "3 of 200 shards degraded" instead of 200 individual panels
• Rebalance/resharding operations in flight and their progress
• Cross-shard query fan-out latency (the scatter-gather pattern from a query that has to touch multiple shards is only as fast as its slowest shard, so this is a distinct signal from any single shard's own latency)

Thresholds, severity, and what should page

What should happen automatically versus what pages a human

• Automatic, no page: a single shard's connection pool (its fixed set of reusable, already-open database connections) exhausting temporarily under a burst (if the application has retry-with-backoff, this self-heals); routine rebalance operations proceeding on schedule.
• Page, with a runbook attached, not just an alert: replication lag critical (runbook: check for a long-running transaction blocking apply, check network throughput between primary and replica, decide whether to fail over or wait); disk critical (runbook: confirm whether growth is organic or a leak/bug, trigger an emergency resize or archive-and-purge); unhealthy-shard-count critical (runbook: check for a common cause across the affected shards before treating them as independent incidents).
• A runbook is not optional documentation, it is the alert's payoff. An alert that pages someone with no linked, concrete next action just moves the "what do I do now" research into the middle of the incident instead of before it.

Trade-offs and pitfalls

• Alerting on raw per-shard thresholds instead of each shard's own baseline produces both false pages (a shard that's always a bit hotter than average) and missed pages (a shard whose baseline is unusually low, where a real problem never crosses a one-size-fits-all number).
• Too many per-shard alerts wired directly to paging is how on-call burns out. Fan noisy per-shard signals into the aggregate unhealthy-shard-count metric for paging, and keep the per-shard dashboards for triage after the page, not as 200 separate alert rules.
• A metric with no owner decays. Review thresholds periodically against actual incident history: a threshold that never fires might be too loose to catch real problems, and one that fires constantly and gets silenced is worse than no alert at all, since it trains people to ignore it.

Direct answer

I would first decide whether the deal justifies trying at all, then compress by cutting scope rather than adding people or demanding overtime, and negotiate with the customer a written, pilot-sized acceptance for day 28 with a dated plan for the rest. A pilot is a limited first release used for real work on a small scale, to prove the value before the full build. Squeezing 12 weeks of work into 4 needs three times the throughput (12 / 4 = 3), which people and overtime do not honestly deliver. What can change is what we promise to have by day 28.

Structured elaboration

Step 1: decide whether to compress. Ask:

1. Is the four-week condition firm, or an opening position? Ask Sales to find out what the customer is really trying to do by then (a board demo, a budget cycle, a competing vendor).
2. What is the deal worth, and what does the team stop doing for four weeks to make room?
3. Is the capability reusable by other customers, or a one-off that becomes a support burden?
   If the deal is real, the value is high, and the capability is reusable, I try. If it is a one-off for a modest deal, I decline the date and offer a later one.

Step 2 (worked example): find the slice that fits four weeks. Break the 12 weeks into components, then ask what the customer must see on day 28.

Every row's slice plus deferred equals its full size, and the columns total 12, 4 and 8.

Why those amounts: the customer's day-28 success is their own data flowing through their own rules, so data import gets 2 of its 3 team-weeks (their real file formats, not every format), configuration gets 1 of 3 (their first two rules, not a general rules engine), and reporting gets 0.5 (the one monthly report they named). Single sign-on and audit log are not needed to see the workflow work, so they get 0. Hardening gets 0.5 so the pilot does not fall over on their first real file.

Step 3: why not just add people or work longer. New engineers take weeks to ramp up and slow the existing team while they do, so a late addition often delays a short project. Overtime for four weeks trades quality for speed, and the defects land in front of the customer we are trying to impress.

Step 4: negotiate with the customer. Ask what must be true on day 28 for them to call this a success; usually it is their own workflow working on their own data, not every feature. Propose the slice as a pilot, and agree in writing:

• Accepted when: their sample files import, their first two rules run, and the monthly report comes out correct, with no critical defects open.
• Explicitly not in the pilot: single sign-on, audit log, full-scale load.
• Their dependencies: sample data and test users by day 5; slip there moves the date.
• The rest: dated, with a firm schedule re-baselined (re-estimated and re-agreed) at the end of week 4 once real use shows the true effort.

What I would say to the customer: "We can have your own data and your first two rules working, with the monthly report, on day 28. Single sign-on and the audit log come in the next phase, and I will give you a firm date for them at the end of week 4, once we have seen your real files. What would you need to see on day 28 to call it a success?"

Step 5: internal discipline. Sales agrees not to promise beyond the written scope. I re-estimate at the end of week 1; if even the slice looks like it will miss, the customer hears in week 2, not week 4.

What would change my call

If the customer's security team will not accept a pilot without single sign-on, the slice grows by 2 weeks and no longer fits, so I would say so and propose a later date or a different pilot setup. If the displaced work is a committed reliability fix, I would decline.

Trade-offs and pitfalls

• A pilot promised as "basically the full thing" is how a four-week deal becomes a twelve-week argument.
• Do not hide the cut scope in the contract; surface it in the first conversation.
• Track what the team stopped doing, and tell its owners.