Scalable System Design and Architecture Questions

Focuses on designing systems that scale to large numbers of users and high load while balancing consistency, availability, and operational concerns. Candidates should discuss application programming interface design, data flow, load distribution and load balancing, caching strategies, database selection and partitioning, indexing, replication, consistency models, microservices boundaries, synchronous and asynchronous communication patterns, message and event architectures, deployment and rollout strategies, and measures for capacity planning and performance optimization. Answers should include trade off analysis, expected bottlenecks, quantification where possible, and rationale for architecture decisions.

EasyTechnical

76 practiced

Describe how you would design rate limiting and throttling for a public API exposed to third-party clients. Cover per-user, per-API-key, and per-IP limits, burst handling, backpressure signals, graceful degradation strategies, and how you would surface quota usage to clients (headers, dashboards).

Sample Answer

**Approach overview (from an Engineering Manager perspective)** I’d set a clear, layered rate-limiting strategy implemented as a distributed service (e.g., Redis + rate-limiter microservice) using token-bucket for burst control and sliding-window or fixed-window with jitter for steady-state fairness. This balances developer velocity, SLOs, and operational simplicity.

**Per-scope limits**- Per-API-key: primary quota tied to client plan (tokens/sec + monthly quota). Enforced at edge for billing/plan isolation. - Per-user: finer-grain limits inside multi-tenant keys to protect noisy users. - Per-IP: last line of defense for abusive traffic or anonymous clients. Use consistent hashing/sharding for counters to scale and avoid single points.

**Burst handling & algorithms**- Token bucket allows bursts up to bucket size, refilling at rate R. Use sliding window for fairness on high concurrency.- Implement soft- and hard-limits: soft triggers alerts/telemetry, hard returns 429.

**Backpressure & client signals**- Return 429 Too Many Requests with Retry-After header when throttled. Include headers: - X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, Retry-After- For streaming or long-running endpoints expose advisory headers and websocket/webhook alerts for near-quota.

**Graceful degradation**- Prioritize critical endpoints (auth, payments) with reserved capacity. - Degrade non-essential features (analytics, batch jobs) or switch to degraded responses (cached data). - Queue low-priority requests with bounded queues and exponential backoff for retries.

**Surface quota usage**- Client-side: headers above + clear error body with guidance. - Self-service dashboard: per-key and per-user usage, historical graphs, forecasted burn rate, billing implications, and alerting config. - Webhooks/alerts for nearing quota and automated plan-upgrade links.

**Operational & team considerations**- Define SLOs and quotas in collaboration with Product and Billing. - Instrument metrics (rate per key/user/IP, throttle rates, latency) and alert on anomalies. - Run client SDKs and docs to encourage exponential backoff and expose best practices.

This design protects platform availability, gives clients actionable feedback, and provides product hooks for monetization and support.

HardTechnical

81 practiced

Given a time series (per-minute) of requests-per-second for a service, write Python code to compute a rolling capacity plan: for each minute compute required instances assuming each instance serves 500 rps at the 95th percentile, add 20% headroom, and account for a 2-minute warm-up by smoothing required instance increases. Explain how this output would feed an autoscaler policy.

Sample Answer

**Approach (brief)** - For each minute t compute the 95th percentile of RPS over a recent window (configurable, e.g., last 60 minutes) to be conservative about bursts. - Compute raw instances = ceil( (p95_rps / 500) * 1.20 ) to include 20% headroom. - Enforce a 2-minute warm-up smoothing: when required instances increase, ramp up linearly over 2 minutes (limit per-minute increase). Decreases are allowed immediately (or could be rate-limited separately). - Feed result as desired_capacity timeline to an autoscaler (target capacity + annotations about proactive vs reactive).

Code (Python):

python

import math
import numpy as np

def rolling_capacity_plan(rps_series, window_minutes=60, inst_capacity=500, headroom=0.20, warmup_minutes=2):
    """
    rps_series: list or 1D array of per-minute RPS samples (length T)
    returns: list of integer instances required per minute (length T)
    """
    T = len(rps_series)
    required = [0]*T
    prev = 0
    for t in range(T):
        start = max(0, t - window_minutes + 1)
        window = rps_series[start:t+1]
        p95 = float(np.percentile(window, 95))
        raw = math.ceil((p95 / inst_capacity) * (1 + headroom))
        # smoothing: allow increase of at most delta_per_min each minute
        if raw <= prev:
            # allow immediate scale down (or optionally rate-limit)
            prev = raw
        else:
            delta = raw - prev
            max_step = math.ceil(delta / max(1, warmup_minutes))  # linear ramp over warmup_minutes
            prev = prev + max_step
            if prev > raw:
                prev = raw
        required[t] = prev
    return required

Key concepts and reasoning- Using a sliding-window p95 captures tail demand rather than noisy instantaneous values. Window size is configurable; larger windows smooth more. - 20% headroom protects against small bursts and scheduling jitter. - Warm-up smoothing prevents sudden spikes in desired instances which would cause flapping or overshoot; linear ramp over 2 minutes approximates instance boot time.

Complexity- O(T * W) if percentiles computed naively; with numpy percentile on slices it's roughly O(T * W). For large T/W, maintain a streaming quantile estimator (t-digest) for O(T log k).

Edge cases & improvements- If per-minute series is sparse or has microsecond bursts, prefer higher-resolution data or use streaming quantile (t-digest). - Consider separate rate limits for scale-down to avoid oscillation. - Integrate health checks and instance readiness before counting toward capacity.

How this feeds an autoscaler policy- The produced timeline is the desired_capacity signal. Autoscaler should use it as target, but enforce platform/operator constraints: cooldowns, min/max instances, verification of instance readiness, and override if predictive model signals imminent spike. Annotate decisions (proactive vs reactive) and expose metrics (p95, raw target, smoothed target) for audit and tuning.

EasyTechnical

63 practiced

Explain rolling, blue-green, and canary deployment strategies. As an Engineering Manager, describe how you'd select a deployment strategy for a service with stringent uptime and frequent releases, including rollback procedures, automated health gates, and monitoring signals that determine whether a rollout should be continued or aborted.

MediumTechnical

136 practiced

Implement a simplified consistent hashing ring in your choice of language (Python, Java, or Go) with methods to add_node(node_id), remove_node(node_id), and get_node_for_key(key). After implementation, explain how you'd extend it to support virtual nodes and handle node failures in a production distributed cache cluster.

Sample Answer

**Approach (brief)** Use a sorted map of hashes -> node_id. For key lookup, hash key and find the first ring entry >= hash (wrap to start). add/remove update map.

**Code (Python)**

python

import bisect, hashlib

def _hash(val):
    return int(hashlib.md5(val.encode()).hexdigest(), 16)

class ConsistentHashRing:
    def __init__(self):
        self.ring = []           # sorted list of hashes
        self.nodes = {}          # hash -> node_id

    def add_node(self, node_id):
        h = _hash(node_id)
        if h in self.nodes: return
        bisect.insort(self.ring, h)
        self.nodes[h] = node_id

    def remove_node(self, node_id):
        h = _hash(node_id)
        if h in self.nodes:
            idx = bisect.bisect_left(self.ring, h)
            self.ring.pop(idx)
            del self.nodes[h]

    def get_node_for_key(self, key):
        if not self.ring: return None
        h = _hash(key)
        idx = bisect.bisect_right(self.ring, h) % len(self.ring)
        return self.nodes[self.ring[idx]]

**Complexity** - add/remove: O(log N) for insert/remove into sorted list (bisect + shift). - lookup: O(log N).

**Extensions for production & manager perspective** - Virtual nodes: create multiple hashed replicas per physical node (node_id#i) to improve balance; tune replica count per load tests. - Node failures: detect via health checks + heartbeat; on failure, mark node drained and remove its vnodes from ring; client-side retry and read-repair. - Operational: automate rebalancing windows, monitor hot keys, add metrics (key distribution, load per node), run chaos tests. - Team practices: write deterministic hashing tests, include migration playbooks, and capacity planning to choose vnode counts and failure domains.

MediumTechnical

63 practiced

Your cloud bill for a high-throughput service grew 4x in a quarter. As the EM, outline a structured investigation to find the cost drivers and a prioritized remediation plan. Include rightsizing, autoscaling adjustments, caching improvements, spot instances vs reserved, workload scheduling, and how to measure cost and availability impacts for each change.

Sample Answer

**Situation & goal**I would lead a fast, data-driven investigation to identify why spend grew 4x and deliver a prioritized remediation plan that preserves availability.

**Investigation plan (steps)**1. Gather facts (24–72h): pull billing by SKU, product tag, account, region; query CloudCost reports, Kubernetes node/pod metrics, APM traces, and request rates.2. Triage top hitters: rank services by spend delta, CPU/Memory, I/O, network egress, and instance hours.3. Root-cause analysis per top candidate: correlate traffic patterns, deployment changes, autoscaling events, CI/CD rolls, and retention/backup jobs.4. Validate anomalies with team interviews and deployment logs.

**Prioritized remediation (quick wins → medium → strategic)**- Quick wins - Rightsize oversized instances and idle resources (stop orphan volumes, unattached NICs). - Turn off non-prod environments outside business hours. - Convert obvious long-running dev VMs to spot or suspend.- Medium - Tune autoscaling policies: use target tracking (p95 latency) vs fixed thresholds; add cooldowns; scale on request queue/CPU/latency as appropriate. - Introduce caching (CDN, Redis) for high-read paths; set TTLs and cache keys. - Shift batch/ETL to off-peak windows; use cheaper instance types during windows.- Strategic - Move steady-state workloads to reserved/committed use discounts; use spot for fault-tolerant workers with fallbacks. - Evaluate architectural changes (streaming vs polling, serverless where appropriate).

**Measurement & rollout**- Define KPIs: cost per request, cost per 1M requests, p95 latency, error rate, availability (SLA), and deployment success rate.- Use canary experiments: implement change in a single region or 5–10% traffic slice; monitor cost deltas and SLOs for 1–2 billing cycles.- Expected validation: show cost reduction delta with confidence intervals and no SLO regressions; rollback on >5% availability degradation or >10% latency regression.- Reporting: weekly dashboard with cost drivers, actions taken, savings realized, and next steps.

I would coordinate with finance, SRE, and product to balance savings with business risk and formalize a continuous cost governance process.

Unlock Full Question Bank

Get access to hundreds of Scalable System Design and Architecture interview questions and detailed answers.

Join thousands of developers preparing for their dream job.