Threat Modeling and Secure System Design Questions

Applying threat modeling and structured problem solving to secure system design. Candidates should be able to decompose complex security challenges by identifying business context, critical assets, threat actors, attack surfaces, and compliance requirements. Topics include threat modeling methodologies, attacker capability and motivation analysis, risk assessment and prioritization, selection of mitigations and compensating controls, and evaluation of trade offs among security, usability, cost, and performance. Candidates should also be able to produce implementation and monitoring plans that address scalability and maintainability and to clearly explain and justify design choices and residual risk to stakeholders.

MediumTechnical

28 practiced

Construct an attack tree for a payment authorization endpoint in an e-commerce system. Include branches for credential compromise, API abuse, man-in-the-middle, database tampering, and supply-chain compromise. For each major branch provide at least two leaf nodes describing specific attack steps and propose mitigations targeted at the highest-probability leaf nodes.

Sample Answer

Root: Payment Authorization Endpoint (POST /authorize)

1) Credential compromise- Leaf A: Phished/stolen user credentials used to submit payment requests.- Leaf B: Compromised service account/API key used by attacker to call endpoint.Mitigations (high-probability: stolen user creds, compromised API keys):- Enforce MFA for user accounts and require device fingerprinting/behavioral risk scoring for high-value txns.- Rotate and store service credentials in vault (HashiCorp Vault/AWS Secrets Manager), enforce short-lived tokens (OIDC, STS).- Monitor: alert on anomalous geo/IP velocity, sudden new device, many failed logins; automatic step-up auth or block.

2) API abuse- Leaf A: Credential stuffing / brute-force to guess payment tokens.- Leaf B: Mass automated requests to drain business logic (replay or race conditions).Mitigations (high-probability: automated abuse):- Rate limiting and per-credential/per-IP throttles + global quotas; apply progressive delays.- Use bot detection (CAPTCHA, device signals) and API gateway with WAF rules; circuit-breakers to protect downstream services.- Instrument metrics: request rates, error rate spikes, auth failures; auto-scale and triage via runbooks.

3) Man-in-the-middle (MITM)- Leaf A: TLS downgrade/strip leading to interception.- Leaf B: Compromised client CA/rogue proxy intercepting traffic.Mitigations:- Enforce TLS 1.2+ with HSTS, certificate pinning for official clients, and strict ciphers.- Use mTLS between internal services; monitor cert transparency logs and automations to detect unusual certs.

4) Database tampering- Leaf A: Unauthorized DB write altering payment statuses (rollback/refund fraud).- Leaf B: Privilege escalation leading to data exfiltration and replay of old transactions.Mitigations (high-probability: privilege misuse):- Apply least privilege, role-based access, and immutable ledger pattern for payments (append-only transactions).- Enable DB auditing, write-ahead integrity checks, signed transaction records, and alert on out-of-pattern writes.- Backup and point-in-time recovery with anomaly detection on restore.

5) Supply-chain compromise- Leaf A: Malicious dependency injected into payment library.- Leaf B: CI/CD pipeline credentials leaked allowing attacker to push malicious code/config.Mitigations (high-probability: CI/CD creds leak):- Sign and verify artifacts (sigstore), use SBOMs, scan dependencies, pin versions.- Harden CI: short-lived runner credentials, segregated build accounts, require PR review and signed commits for releases.- Monitor build logs and deployment patterns; enforce canary rollouts and automated rollback on anomalies.

SRE-specific additions:- Define SLOs for authorization latency/availability; create playbooks for scenarios (fraud flood, credential compromise).- Centralized telemetry: correlate auth logs, WAF, vault access, CI events; run periodic chaos tests (failure injection) and threat-drills.

MediumTechnical

27 practiced

Given the table schema: access_logs(user_id varchar, ip inet, event varchar, ts timestamptz), write a PostgreSQL query that identifies IP addresses that attempted connections to more than 50 distinct user_ids within any 10-minute window (indicating credential stuffing or scanning). Use window functions or time-bucketing in your solution and state any assumptions.

Sample Answer

Approach: for each log row treat its timestamp as the start of a 10‑minute sliding window, count distinct user_ids seen from the same IP in that window, and surface IPs where the distinct user count > 50. This uses a correlated subquery (simple, works in plain PostgreSQL). For large data consider time-partitioning, indexes, or TimescaleDB/hop_windows for efficiency.

Assumptions:- ts is indexed and in UTC (timestamptz)- We consider half-open windows [ts, ts + 10 minutes)- Memory and performance acceptable for correlated subqueries or use more scalable approaches if data is huge

Query:

sql

-- Find IPs that, within any 10-minute sliding window, touched >50 distinct user_ids
SELECT DISTINCT ip
FROM (
  SELECT a1.ip,
         a1.ts AS window_start,
         (
           SELECT COUNT(DISTINCT a2.user_id)
           FROM access_logs a2
           WHERE a2.ip = a1.ip
             AND a2.ts >= a1.ts
             AND a2.ts < a1.ts + interval '10 minutes'
         ) AS distinct_users
  FROM access_logs a1
) t
WHERE distinct_users > 50;

Key points and optimizations:- Create indexes: CREATE INDEX ON access_logs (ip, ts);- For very large volumes use time-partitioning or materialized summary tables (e.g., pre-aggregate per minute) or TimescaleDB’s hop_windows to avoid O(n^2) behavior.- Edge cases: simultaneous spikes at exact boundary (half-open window handles consistently), botnets distributing requests across many IPs (aggregate by /24 if needed).

HardSystem Design

30 practiced

Design a DDoS protection architecture for a public API that normally handles 500k RPS but must be prepared for short bursts up to 5M RPS. Include edge filtering (CDN/WAF), autoscaling and rate-limiting strategies, upstream scrubbing options, caching strategies to reduce load, and SRE playbooks and escalation paths for extreme events.

Sample Answer

Requirements & constraints:- Normal: 500k RPS, burstable to 5M RPS for short periods- Low latency API, public endpoints, must maintain availability and protect backend integrity and stateful services

High-level architecture:Edge CDN/WAF (global anycast) → Rate-limit + token bucket at edge → Verified requests → API gateway / ingress (with autoscaling + local rate limits) → Backend services (stateless, scaled behind queues) → Origin DB / stateful systems (protected by write-throttles & circuit breakers)

Edge filtering- Use multi-CDN anycast (Cloudflare/Akamai/FASTly + backup providers) to absorb volumetric attack.- WAF rules: block known bad IPs, geo-block options, signature-based rules, OWASP CRS, challenge (JS/CAPTCHA) for suspicious flows.- Behavioral detection: challenge on anomalous request patterns (high error rates, unusual UA, missing headers).- Edge TCP/UDP scrubbing for SYN/UDP floods via provider DDoS scrubbing centers.

Autoscaling & rate limiting- API gateway enforces hierarchical limits: - Global per-IP / per-subnet soft limit (token bucket, burstable) - Per-API-key or per-user quotas (strict) - Service-level circuit breakers (drop or 429 when downstream latency/queue depth crosses thresholds)- Autoscale stateless workers aggressively with fast scale-out policies (scale by CPU/queue depth + predictive scaling using traffic forecasting). Keep warm pools and pre-warmed containers to reduce cold start latency for burst to 5M RPS.- Scale write-durable components conservatively; route excess to durable queues (Kafka/SQS) and return 202 Accepted.

Upstream scrubbing options- Contract with scrubbing providers (Cloudflare Spectrum / provider scrubbing) and have BGP failover / traffic steering to scrubbing centers during volumetric attacks.- Implement geo-routing and blackhole only as last resort (announce more specific route to scrubbing or null-route abusive prefixes).

Caching strategies- Maximize CDN caching for GETs and idempotent responses; use Cache-Control, stale-while-revalidate, and origin shielding to collapse requests.- Edge key normalization (strip non-essential headers, normalize query strings) to improve hit rate.- Use tiered caching: CDN edge → regional POPs → origin cache (Redis/memcached) for semi-dynamic responses; short TTLs with background refresh and stale-while-revalidate.- Cache authorization tokens where safe (short TTL) and implement signed URLs for private resources.

Observability & SRE playbook- Monitoring: end-to-end: RPS, 5xx/4xx, p95/p99 latency, queue depths, CPU, Pod autoscaler metrics, CDN hit ratio, WAF/edge alerts, BGP route changes.- Alerts: tiered thresholds (warning at 30% over baseline, critical at >200% sustained; DDoS incident threshold at >1M RPS or sudden >5x spike).- Playbook summary: 1. Detect: automated detection from CDN & telemetry; open incident and route to on-call. 2. Triage: confirm pattern (volumetric vs application-layer). Check CDN dashboards, WAF logs, origin metrics. 3. Mitigate at edge: - Enable strict WAF rules and JS challenge - Increase scrubbing/protection with provider (activate scrubbing) - Tighten global rate limits, drop abusive IP ranges, enable geo-block if attack concentrated 4. Protect backend: - Enable circuit breakers, increase 429 responses, enable queuing for writes - Scale stateless pool and enable warm pools 5. Escalate: involve network/BGP team to announce route to scrubbing; contact CDN/ISP TAM; exec if customer-facing SLA at risk 6. Post-incident: RCA, update rules, add new signatures, update SLOs, and rehearse tabletop for next event

Operational considerations & trade-offs- Aggressive edge blocking risks false positives: use phased escalation (challenge → rate-limit → block).- Caching increases hit ratio but must respect freshness and auth constraints.- Autoscaling too slow for 10x bursts: use predictive/pre-warmed capacity and banked burst credits with providers.- Contractual: SLAs with CDN/scrubber for activation time and capacity guarantees.

This design prioritizes absorbing volumetric attacks at the edge, shifting work to caches and queues, protecting stateful systems with throttles and circuit breakers, and giving SREs a clear mitigation and escalation playbook.

EasyTechnical

31 practiced

Define 'defense-in-depth' and provide three concrete examples showing how an SRE team would implement defense-in-depth across network, host, and application layers. For each example explain what layer it protects and why layering these controls reduces overall risk.

Sample Answer

Defense-in-depth is a security principle that uses multiple, redundant layers of controls across the stack so that if one control fails or is bypassed, others still mitigate or detect the threat. For an SRE team, it means combining network, host, and application controls to protect availability and integrity.

1) Network: Perimeter and microsegmentation + network ACLs and service mesh mTLS- What: Use firewalls/NGFW at the edge, internal VLANs or Kubernetes NetworkPolicies, and mTLS via a service mesh.- Why it protects: Blocks unauthorized traffic at the perimeter and limits lateral movement inside the cluster; mTLS authenticates services.- Risk reduction: Even if an edge rule is misconfigured, internal segmentation and service identity prevent broad compromise.

2) Host: Hardened images, host-based firewall, and EDR + automated patching- What: Build minimal golden images, enable iptables/ufw, run EDR/agent monitoring and automated security patch pipelines.- Why it protects: Reduces attack surface, blocks suspicious connections on-host, and detects/prevents execution of malicious processes.- Risk reduction: If network controls miss an exploit, host hardening and EDR can stop or detect it before it affects service integrity.

3) Application: Input validation, authz/authn, rate limiting, and telemetry- What: Enforce strong authentication (OAuth), role-based authorization, input sanitization, per-API rate limits, and detailed logging/alerts.- Why it protects: Prevents common app-layer attacks (SQLi, auth bypass) and limits abuse; telemetry enables rapid detection.- Risk reduction: Even if a host is compromised or a network tunnel exists, app-level checks and rate limits reduce impact and allow fast remediation.

Layering these controls creates overlapping protections: preventive (network/host hardening), detective (EDR/logging), and responsive (rate limits/automated rollback). That diversity lowers the probability of a single point of failure causing a major incident and speeds detection and recovery, improving overall reliability.

MediumTechnical

30 practiced

Design monitoring signals and an alerting strategy to detect privilege escalation activity in a Kubernetes cluster (examples: sudden creation of clusterroles, escalations of service accounts, unexpected binding changes, or suspicious kubectl exec usage). Include which audit logs and metrics to collect, alert thresholds, and a short mitigation playbook to follow when alerts fire.

Sample Answer

Situation: We need monitoring signals and an alerting strategy that reliably surface privilege-escalation activity in a Kubernetes cluster (new cluster-wide RBAC objects, suspicious ServiceAccount use, unexpected binding changes, excessive kubectl execs) with low false positives and clear runbook steps for responders.

Monitoring signals to collect- Kubernetes audit logs (high-fidelity apiserver audit): capture RequestURI, user.username, user.groups, sourceIPs, verbs, objectRef (kind/name/namespace), responseStatus, requestObject. Ensure audit policy records requestObject for create/update/delete on ClusterRole, ClusterRoleBinding, RoleBinding, ServiceAccount, Secret, TokenRequest, Pods (exec/attach/portforward).- Cloud provider IAM logs (GCP/AWS/Azure) for IAM role/grant changes and token exchanges.- Prometheus metrics & events: - kube_apiserver_request_total by user/verb/resource - kube_audit_event_count (if using audit exporter) by event_type/resource - Falco/OpenPolicyAgent/Gatekeeper alerts (runtime suspicious syscalls / policy denies) - Kubernetes Events for RoleBinding/ClusterRoleBinding/ServiceAccount creation - Application logs that perform execs (if sidecars/logging in place)- Runtime telemetry: container exec events (Falco rule), kubelet logs, and audit webhook sinks to SIEM (ELK/Splunk).

Alert rules and thresholds (priority tiers)- P0 (Immediate/High): Creation or update of ClusterRole or ClusterRoleBinding (resource=ClusterRole|ClusterRoleBinding) where user is not in allowed admin set OR created by short-lived credential. Trigger immediately with details.- P0: Binding that grants cluster-admin or equivalent to a ServiceAccount, user, or group outside known owners. Immediate.- P1 (Elevated): >3 ClusterRole/ClusterRoleBinding/RoleBinding creations or updates in 5 minutes from same user/IP — possible scripted attack.- P1: TokenRequest or ServiceAccount token creation for non-system ServiceAccount from unexpected namespace, or large number of TokenRequests (>10 in 10m).- P2 (Notice): >5 kubectl exec/attach events by same ServiceAccount or user across different pods in 5 minutes — anomaly for most apps. Lower threshold for stateless infra namespaces.- P2: Unexpected role binding modifications to privileged roles (change in subjects) — alert but allow human triage.- All alerts should include: who (user.username, groups), sourceIP, requestObject snapshot, timestamp, audit log link, and correlated recent related events.

Why these thresholds- Immediate alerts for creation/binding of cluster-level privileged objects because impact is high even at a single event.- Rate-based thresholds detect automated bulk changes or lateral movement.- Exec/attach thresholds tuned per-namespace (dev vs prod) to avoid false positives.

Enrichment & correlation- Enrich alerts with identity context (SSO source, MFA state, recent login), Pod owner labels, CI/CD pipelines (detect known automation users), geographic/source-IP reputation, and recent Kubernetes events from same user/IP.- Correlate with cloud IAM changes and CI pushes within last 30 minutes.

Short mitigation playbook (runbook)1. Triage (first 10 minutes) - Acknowledge alert, capture audit IDs, and gather context: kubectl get events, kubectl get clusterrole/clusterrolebinding -o yaml for the named object, list recent TokenRequests and pods accessed by that user/serviceaccount. - Identify whether the actor is an automation identity (CI/CD) or human (SSO). Check login times and sourceIP reputation.2. Containment (10–30 minutes) - If binding or ClusterRole is newly created and unauthorized: kubectl delete <object> (ClusterRole/ClusterRoleBinding/RoleBinding) OR patch to remove privileged subjects. - Revoke tokens/credentials: - For ServiceAccount: create a patch to remove secrets/token, rotate secrets, delete ServiceAccount secrets, or annotate to disable. If using projected tokens, revoke at issuer (OIDC). - For user credentials: disable the user in identity provider (SSO) or revoke API keys. - Isolate affected workloads: cordon/evict pods if necessary, or disable network access using NetworkPolicy.3. Investigation (30–120 minutes) - Search audit logs for lateral movement: other privilege-creation events, execs, CronJobs, new pods with hostPath or privileged true. - Snapshot cluster state (kubectl get all --all-namespaces), export critical logs to secure storage, collect Pod images and running commands. - Check CI/CD job runs, recent git commits, and automation tokens used.4. Remediation (hours) - Rotate compromised secrets, service account keys, and any cloud credentials associated with the user. - Revert unauthorized RBAC objects or apply least-privilege replacements. - Patch vulnerable admission controls: enforce OPA/Gatekeeper policies preventing cluster-admin binding, require owner annotation for binding changes, block creation of privileged ClusterRoles except via approved pipeline.5. Recovery & Lessons (1–3 days) - Restore any deleted/rolled back objects only after validation. - Post-incident: update audit policies, tighten thresholds, add Falco/Gatekeeper rules, run tabletop with stakeholders. - Report to security and compliance teams, and if required, follow disclosure policies.

Detection hardening & prevention- Enforce admission controls: deny ClusterRole/ClusterRoleBinding creation except from approved service accounts/pipelines (Gatekeeper).- Require MFA for admin operations and short-lived credentials.- Use dedicated admin workstations/nodes and limit kubectl exec from production namespaces.- Centralize audit logs in SIEM and keep 90+ days retention for forensics.

Example actionable queries (for responders)- Find recent clusterrole creations: - grep/ELK: kubernetes.audit.requestURI:/apis/rbac.authorization.k8s.io/v1/clusterroles AND verb:create AND timestamp:[now-5m TO now]- Find execs by SA: - audit logs where verb="create" AND requestURI contains "/exec" AND user.username="system:serviceaccount:namespace:name"

This strategy balances immediate blocking of high-impact events, anomaly detection for scripted or lateral activity, and a clear containment/investigation path so SREs can respond quickly and reduce blast radius.

Unlock Full Question Bank

Get access to hundreds of Threat Modeling and Secure System Design interview questions and detailed answers.

Join thousands of developers preparing for their dream job.