Real World Scenario Based Decision Making Questions

Applying infrastructure knowledge to realistic business scenarios: handling traffic spikes, migrations from on-premises to cloud, optimizing costs during resource constraints, responding to security incidents, and managing infrastructure during rapid growth. Making trade-off decisions when constraints conflict.

EasyTechnical

40 practiced

Write a Python function decide_scale(samples: List[int]) -> str that inspects a list of CPU percentage samples (one per minute). Return 'scale_up' if the last 3 consecutive samples are >75, return 'scale_down' if the last 5 consecutive samples are <30, otherwise return 'no_change'. Provide example inputs and outputs, and ensure O(n) time. Example: samples = [60, 78, 80, 82] -> 'scale_up'.

Sample Answer

Approach: scan the samples from the end and count consecutive values that meet the thresholds. Stop as soon as a condition is met or when values break the streak. This is O(n) time and O(1) extra space.

python

from typing import List

def decide_scale(samples: List[int]) -> str:
    """
    Inspect CPU percentage samples (one per minute).
    - Return 'scale_up' if the last 3 consecutive samples are > 75
    - Return 'scale_down' if the last 5 consecutive samples are < 30
    - Otherwise return 'no_change'
    """
    if not samples:
        return 'no_change'

    # Check scale_up: need last 3 consecutive > 75
    up_count = 0
    for v in reversed(samples):
        if v > 75:
            up_count += 1
            if up_count >= 3:
                return 'scale_up'
        else:
            break

    # Check scale_down: need last 5 consecutive < 30
    down_count = 0
    for v in reversed(samples):
        if v < 30:
            down_count += 1
            if down_count >= 5:
                return 'scale_down'
        else:
            break

    return 'no_change'

Key points:- We scan from the end and break when a streak is interrupted, so worst-case we touch all elements once: O(n) time, O(1) space.- Edge cases: empty list -> 'no_change'; lists shorter than required lengths cannot trigger those actions.

Examples:- [60, 78, 80, 82] -> 'scale_up'- [25, 20, 28, 26, 29] -> 'scale_down'- [80, 70, 80, 80] -> 'no_change' (only two consecutive >75 at end)

MediumSystem Design

39 practiced

Design an observability pipeline for logs, metrics, and traces for a microservices platform processing ~100k events/sec. Cover ingestion, labeling/enrichment, sampling, storage choices (hot vs cold), alerting architecture, and cost controls. Suggest open-source or managed components and justify trade-offs.

Sample Answer

Requirements & constraints:- Ingest logs, metrics, traces for microservices at ~100k events/sec.- Low-latency alerting for SLOs, searchable logs/traces, cost-effective long-term storage.- Prefer hybrid open-source + managed options.

High-level architecture:- Agents on hosts: Vector or Fluent Bit (lightweight) for logs + traces; Prometheus exporters + node_exporter for metrics.- Ingestion buffer/stream: Kafka or Pulsar for durable, backpressure-aware buffering.- Processing pipeline: Stream processors (Vector, Fluentd, or Kafka Streams) for labeling, enrichment, parsing, schema normalization, and sampling.- Storage: - Hot tier (fast queries / alerts): - Metrics: Cortex/Thanos (Prometheus remote-write) or managed Prometheus (Grafana Cloud / Cortex). Use TSDB with short retention (15–30d). - Traces: Tempo or Jaeger with single-node store + Elasticsearch backend for indexed trace search OR Tempo + ClickHouse for high-volume; short retention (7–14d). - Logs: Loki (label-based, cheap index) or Elasticsearch for richer search; hot retention 7–14d. - Cold tier (long-term, cheaper): Object storage (S3/GS/MinIO) with Parquet/ORC for logs and traces; metrics aggregated rollups in S3 and long-term Prometheus snapshots.- Query/UX: Grafana connecting to metrics, traces (Tempo), logs (Loki/ES).

Labeling & enrichment:- Do at edge when possible: add service, environment, deployment id, trace_id correlation, request metadata.- Central enrichment in stream processors for heavyweight joins (user profile, account tier) to avoid agent bloat.- Enforce schema and cardinality limits during enrichment.

Sampling strategy:- Traces: Adaptive tail-based sampling (via Kafka Streams/Vector/KSampler) to keep all error/slow traces and sample successful requests (e.g., 1-5%) to control volume.- Logs: Apply dynamic sampling and redaction; full capture for error-level logs, sampled for debug/info.- Metrics: No sampling; use aggregation and downsampling for resolution retention trade-offs.

Alerting architecture:- Real-time metrics alerts via Prometheus + Alertmanager (or Cortex Alertmanager federated). Use Grafana for dashboards and escalation.- Trace/log-based alerts via streaming anomaly detectors (e.g., Prometheus rules or Kafka Streams detecting error spikes) pushing to Alertmanager.- Alerting best practices: SLO-driven alerts (page on burn-rate), deduplication, suppression windows, escalation policies.

Cost controls & trade-offs:- Use Kafka for buffering to decouple producers/consumers; managed Kafka (Confluent/MSK) reduces ops cost but increases license/cloud cost.- Loki + labels is cheaper for logs than Elasticsearch; ES offers richer querying at higher cost.- Object storage for cold data reduces cost massively; convert to columnar format for analytics.- Enforce cardinality limits upstream — high-cardinality labels (user_id, request_id) are main cost drivers.- Downsample high-frequency metrics for long-term retention; aggregate traces/logs to event summaries.- Autoscale consumers and use spot instances for batch cold-processing jobs.- Monitor observability costs as an SLO: track cost per million events and alert on anomalies.

Trade-offs summary:- Managed components reduce operational overhead (recommended for teams without SRE capacity) but cost more.- Loki + Tempo + Cortex/Thanos offers good cost/scale balance; Elasticsearch/Jaeger gives richer features at higher cost.- Tail-based sampling preserves signal but requires streaming compute; simple rate sampling is cheaper but loses rare failures.

This design balances reliability, queryability, and cost by buffering, targeted enrichment, adaptive sampling, and hot/cold tiering.

EasyTechnical

48 practiced

Explain differences between a Layer 4 Network Load Balancer (NLB) and a Layer 7 Application Load Balancer (ALB). Cover TLS termination, connection handling (TCP/HTTP), sticky sessions, WebSocket support, path/host-based routing, and considerations for source-IP preservation. Recommend which to use for a microservices REST API vs a WebSocket chat service.

Sample Answer

Layer 4 NLB vs Layer 7 ALB — concise comparison:

- TLS termination: - NLB: Typically passes TCP (including TLS) through to backends; TLS termination happens on the target unless you terminate on target group (or use TLS listener with passthrough). Good when you want end-to-end TLS and backend certificate handling. - ALB: Can terminate TLS at the load balancer (offloading crypto), optionally re-encrypt to backend. Easier certificate management and HTTP features.

- Connection handling (TCP/HTTP): - NLB: Operates at TCP/UDP level; high-performance, low-latency, preserves raw connections. - ALB: Understands HTTP/HTTPS; can inspect headers, methods, and payload-level routing.

- Sticky sessions: - NLB: Can support source-IP or network-level affinity but not HTTP-cookie-based sticky sessions. - ALB: Supports cookie-based stickiness (application cookies or generated ones).

- WebSocket support: - NLB: Works (raw TCP) if backend handles WebSocket handshake; preserves connection semantics. - ALB: Supports WebSockets over HTTP(S) and handles upgrades; convenient when terminating TLS at ALB.

- Path/host-based routing: - NLB: No. Routes are per IP/port only. - ALB: Yes — rich path, host, header, method-based routing for microservices.

- Source IP preservation: - NLB: Preserves client source IP by default — useful for IP-based auth/rate-limiting. - ALB: Terminates connection so backend sees LB IP; original client IP is forwarded via X-Forwarded-For header (apps must read that).

Recommendation:- Microservices REST API: Use ALB — TLS termination, path/host-based routing, cookie/session features, richer HTTP-aware features (WAF, redirects).- WebSocket chat service: Use NLB if you need true source-IP preservation and minimal latency or if you want end-to-end TLS to backend; use ALB if you prefer to terminate TLS at LB and still get managed WebSocket upgrades. Generally, NLB for raw WebSocket performance and IP preservation; ALB if you want HTTP-layer features alongside WebSockets.

EasyTechnical

32 practiced

You observe anomalous outbound traffic from a production VM suggesting possible data exfiltration. Describe the immediate containment and assessment steps you would take within the first hour, prioritizing actions to protect customer data, preserve forensic evidence, and maintain critical services. Indicate which actions are reversible and which will cause downtime.

Sample Answer

First 0–10 minutes — contain without destroying evidence- Put the VM into a network-isolated VLAN or apply a host-level firewall rule to block suspicious outbound IPs/ports (reversible; minimal downtime if rule targets only malicious flows). - Take a live snapshot of the VM disk and a memory dump (critical for forensics; reversible in sense you don’t modify production state but will consume storage; does not cause service downtime). - Preserve logs: export system, application, network (flow) logs and centralize them to immutable storage.

10–30 minutes — assess scope and protect data- Identify process/user initiating traffic (SSH sessions, service accounts) via ps/ss/lsof and correlate with logs (non-destructive). - Quarantine credentials: rotate API keys/service-account credentials referenced by the VM, revoke tokens if possible (reversible but may cause dependent services to fail — potential downtime). - Block external destinations at edge firewall/IDS to stop exfiltration (reversible; may cause partial downtime for legitimate outbound flows).

30–60 minutes — escalate, communicate, and stabilize- Notify security/incident response, on-call manager, and stakeholders with summary and actions taken. - If VM runs critical customer-facing service and containment caused outage, consider failover: spin up a clean instance from known-good image and route traffic (causes planned downtime during cutover but restores service). - Continue forensic collection: network captures (pcap), hash snapshots, document timestamps and commands — ensure chain of custody.

Which actions cause downtime vs reversible- Non-downtime and reversible: firewall rules targeting malicious IPs, snapshots, log exports, forensics captures. - Potential downtime but reversible: revoking/rotating creds, blocking broad outbound access. - Causes downtime (unless mitigated by failover): isolating VM from network entirely or stopping services; recoverable by restoring snapshot or reconfiguring network.

Rationale: prioritize stopping data flow first with minimal disruption, preserve volatile evidence early, then perform scoped investigation and credential containment, escalating to service failover only if needed to maintain customer-facing availability.

HardTechnical

47 practiced

Your metrics bill is exploding due to high cardinality labels. Propose a concrete plan to reduce observability costs while preserving SLOs: include label normalization and removal, aggregation/rollups, sampling strategies, histograms vs gauges, and application instrumentation changes. Provide examples of acceptable transformations and their trade-offs.

Sample Answer

Requirements & goal:- Reduce metrics cost caused by high-cardinality labels while preserving SLO observability and alert fidelity (latency/error-rate SLOs). Target: cut active time-series by X% (e.g., 70%) without blind spots.

Plan (phased, measurable):1. Inventory & prioritize- Query current TS count by metric and label (prometheus_series_total, remote_write samples). Rank metrics by cost × importance to SLOs.- Identify SLO-critical metrics (latency p99, error rate, throughput).

2. Label normalization & removal (application changes)- Remove unnecessary dynamic labels (user_id, request_id, session_id). Example: transform "user_id=12345" → drop label for metrics; keep as logs/traces.- Normalize high-cardinality enums: "region=us-west-2a" → "region=us-west" (strip az). Trade-off: loses granularity for debugging but keeps region-level visibility.- Bucket free-form strings: "endpoint=/api/v1/users/123/profile" → "endpoint=/api/v1/users/:id/profile" or only record "endpoint_group=user-profile". Trade-off: loses per-id metrics but greatly reduces series.

3. Aggregation / rollups (ingest/remote write or in scrape targets)- Emit pre-aggregated counters/gauges from app for common dimensions (service, region, status) instead of all combinations.- Use downsampling/rollup in the metrics backend: raw 1s → 1m for non-SLO metrics. Keep high-resolution for SLO-critical signals only.- Example: Keep per-instance CPU gauge at 1m; aggregate to service-level for dashboards.

4. Sampling & cardinality caps- For non-SLO traces/metrics (debug counters), sample 1/N requests and multiply by N at ingestion. Example: record a sampled error_count with sample_rate label; multiply in processing for estimation. Trade-off: variance increases; not suitable for alerting thresholds.

5. Histograms vs gauges- Use histograms/exponential histograms for latency distributions instead of many fixed-bucket gauges per percentile. Prometheus histograms let you compute quantiles without generating per-quantile series. Example: replace p50/p90/p99 gauges with a single request_duration_seconds histogram. Trade-off: computing accurate extreme tails requires proper buckets or EH support.- Use summaries only when client-side quantiles are required (but they create per-instance series — avoid if cardinality high).

6. Instrumentation best practices- Instrument counter increments with stable low-cardinality labels only (service, env, region, status_class).- Expose exemplars (trace IDs) only on error/high-latency samples to link traces without adding series.- Provide config flags to enable debug-level high-cardinality metrics only for short windows.

7. Governance & automation- Enforce a cardinality review in PRs for new metrics. Block addition of labels above allowed cardinality (e.g., >50 values).- Alert when new metric explosion occurs (rate of series growth).- Maintain a whitelist of SLO-critical high-card metrics with business sign-off.

Examples of acceptable transformations:- endpoint=/users/123 -> endpoint_group=/users/:id (good)- az=us-west-2a -> zone=us-west (good)- user_id=12345 -> drop label (good; use trace/logs for per-user analysis)- percentiles exported as gauges per-instance -> replace with histogram (good)

Trade-offs summary:- Removing labels reduces debugging granularity; mitigate with traces/logs/exemplars.- Sampling and aggregation introduce estimation error and latency in detection; avoid for hard SLO thresholds.- Histograms require correct bucket design; mis-bucketing can hide tail behavior.

Metrics & success criteria:- Reduce total series by target percentage.- No increase in SLO alert false negatives; tightly monitor alert latency/false positives post-change.- Roll back or temporarily re-enable higher-card metrics for incident triage.

Implementation timeline:- Week 1: inventory + blocking of non-critical high-card metrics- Week 2–3: application instrument changes for top 5 cost metrics- Week 4: backend rollups and sampling configs- Continuous: governance, alerts, and refinement.

This plan balances cost reduction and SLO fidelity by keeping high-resolution for critical signals, moving detailed per-entity data to traces/logs, and using aggregation/histograms for distributions.

Unlock Full Question Bank

Get access to hundreds of Real World Scenario Based Decision Making interview questions and detailed answers.

Join thousands of developers preparing for their dream job.