Security Fundamentals and Core Concepts Questions

Situation: A large-scale DDoS targets a customer's public REST/GraphQL APIs, causing high latency, increased error rates, and service degradation for customers.

Playbook (runbook style)

1. Objectives

• Restore availability within SLA, protect origin, preserve legitimate traffic, and gather evidence for post-mortem.

2. Immediate mitigation (first 0–30 min)

• Enable global emergency rate-limits at edge (per-IP, per-subnet, per-endpoint) with aggressive drop policies for unknown clients.
• Activate CDN/WAF emergency ruleset: block anomalous geographies, tighten bot score thresholds, enforce challenge (CAPTCHA) on suspicious flows.
• Switch to provider scrubbing (DDoS mitigation service / scrubbing center) – route traffic via upstream scrubbing (BGP announce or DNS redirect) if volumetric attack exceeds CDN capacity.
• Disable non-essential, heavy-weight endpoints (bulk pulls, analytics, debug endpoints); return 503 with Retry-After for degraded endpoints.
• Increase logging/packet capture at edge and preserve timestamps/headers for forensic analysis.

3. Roles & responsibilities

• Incident Commander (Solutions Architect/Lead): coordinates technical response, external vendor liaison, business updates.
• Network/SRE Lead: implement BGP redirects, scrubbing, firewall changes.
• Security/WAF Engineer: tune WAF rules, rate limits, blocking lists.
• App Owner/Backend Engineers: toggle feature flags, scale back non-critical services, validate health checks.
• Customer Success/Product/Legal: stakeholder comms, customer notices, regulatory / escalation support.
• Vendor SPOC (CDN/Scrubber): implement scrubbing, provide throughput/mitigation status.

4. Communication plan

• Triage channel: dedicated Slack/War-room for technical teams.
• Executive updates: 15-min cadence to stakeholders with status, impact, ETA.
• Customer notifications: concise status page entries and targeted email if SLAs impacted; avoid technical speculation.
• External vendor updates: maintain 1:1 with vendor SPOC until normal.

5. Escalation path

• If mitigation not restoring traffic in 15–30 min → escalate to cloud provider/network carrier + activate BGP community-based scrubbing and legal/abuse teams.
• If SLA breach imminent → exec escalation and legal/regulatory notification.

6. Business continuity

• Failover read-only or cached API endpoints via CDN cache and API gateway; serve stale-but-consistent responses where safe.
• Use progressive degradation: prioritize critical endpoints and VIP customers; route non-critical requests to maintenance mode.
• Activate rate-limited partner routes (whitelisted IPs/API keys) to keep key customers operational.

7. Post-incident analysis (24–72 hrs)

• Collect telemetry (edge logs, WAF events, scrubbing reports, pcap), timeline actions, and metrics (RPS, error rates).
• Root cause analysis: attack vector, amplification type, attacker infrastructure.
• Lessons and remediation:
  • Tune long-term WAF rules, implement geo-blocking/managed bot mitigation.
  • Design capacity improvements: multi-CDN, autoscaling policies with surge protection, BGP routing playbooks.
  • Implement per-customer rate-limits, API quotas, and progressive backpressure.
  • Run tabletop drills and update runbook SLA/RCA templates.
• Deliverables: incident report, revised architecture diagrams, updated runbook, cost/impact analysis, and customer-facing post-mortem.

Why this works: combines immediate tactical controls with coordinated roles and communications, provider partnerships for volumetric scrubbing, measurable continuity steps, and an actionable post-incident program to reduce repeat risk.

Situation: Managed DB service vs application-level encryption for PCI/PHI columns. Analysis: Provider-managed encryption at rest (disk/table-level) offers convenience: transparent encryption, KMS integration, and provider-managed keys; lowers operational burden but still allows cloud provider admins theoretical access and may not protect against compromised DB users or backups restored outside controls. Application-level (column) encryption ensures end-to-end protection: only application holds plaintext keys, minimizing DB/host-level compromise risk and reducing compliance scope for DB admins. Trade-offs: App-level increases development complexity (key management, query limitations, inability to index/search encrypted columns unless using deterministic encryption), performance overhead, and challenges with backups and analytics. Recommendation: Use provider-managed encryption by default plus envelope encryption for high-risk columns (tokenization or client-side encryption) and manage keys in a customer-controlled KMS/HSM. For PHI/PCI, prefer application-level encryption or tokenization for the most sensitive fields combined with strict key rotation, access controls, and audit logs.

Vulnerability assessment approach for production with web apps, mobile APIs, and third-party services:
Scope: inventory all public endpoints, internal APIs, mobile backends, third-party integrations, and infrastructure components. Classify by criticality and data sensitivity.
Tools & techniques: automated SAST/DAST for apps, API scanners (e.g., OWASP ZAP), SCA for dependencies, container/image scanning, and authenticated scans of APIs. Perform credentialed host scans and runtime agents for production. Include regular pentests (annual or after major changes) focusing on high-risk assets.
Frequency: continuous automated scanning (daily/weekly), dependency checks on PRs, monthly vulnerability aggregation, and quarterly/annual manual pentests for critical assets.
Third-party risk: verify vendor security posture, request attestations, and include their CVE/patch cadence in schedule.
Remediation plan: triage by risk (CVSS + business impact), assign owners, create SLAs (e.g., critical 72 hours, high 30 days), track in ticketing system, and verify fixes with re-scan. Report trends to stakeholders and feed into release gating.
As a Solutions Architect I map findings into prioritized remediation roadmaps and communicate residual risk and mitigation cost to business owners.

Requirements clarification:

• Strong confidentiality for PII, strict tenant isolation, per-tenant encryption keys, GDPR data residency per tenant, full auditability, low latency for 10M users across 100k tenants.

High-level architecture (components):

1. Ingress/API Gateway: routes requests, enforces tenant routing, WAF, provides TLS termination.
2. Regional Tenant Router: directs traffic to the tenant’s configured region/cluster (residency).
3. Tenant Compute Clusters: isolated Kubernetes namespaces or dedicated VPCs for high-risk tenants; otherwise multi-tenant clusters with strict network policies.
4. Storage:
   • Application DB: per-tenant logical isolation (separate schemas or dedicated DB instances for sensitive tenants).
   • Object Store: per-tenant prefixes + server-side encryption.
5. Key Management:
   • Central KMS backed by HSMs (cloud KMS or on-prem HSM). Use envelope encryption: data encrypted with per-tenant DEKs; DEKs encrypted by tenant KEKs in KMS.
   • Support BYOK/Customer-managed keys and key rotation, per-tenant key policies.
6. Secrets & Config: Vault (HashiCorp) for dynamic secrets, access tokens, with tenant-scoped policies.
7. Audit & Monitoring: immutable audit logs (append-only) forwarded to SIEM; cryptographic signing of sensitive events; retention and export controls per region.
8. Identity & Access: centralized IAM with RBAC and least privilege; strong MFA for admin actions; just-in-time admin elevation.
9. Data residency control plane: tenant metadata service storing residency, key bindings, and isolation level.
10. Performance layer: caching (encrypted at rest, in-memory plaintext only in secure enclaves or protected processes) and asynchronous encryption offload service.

Data flow (example write):

1. Client -> API Gateway (mTLS/TLS) authenticates, determines tenant id.
2. Gateway -> Tenant Router sends to region/cluster per tenancy.
3. App fetches tenant DEK from local cache or requests from Vault which requests KMS to unwrap KEK to obtain DEK (use short TTL).
4. App encrypts PII using DEK (envelope) before persisting to DB or object store.
5. Audit event (who/what/when/where) signed and sent to immutable log; logs replicated to tenant’s configured region.

Key design choices & trade-offs:

• Envelope encryption reduces KMS calls (improves latency) but requires secure DEK caching; compromise TTL vs. risk.
• Per-tenant DB instances maximize isolation and compliance but increase cost/operational complexity; use for high-risk tenants and logical isolation (schemas + row-level encryption) for others.
• BYOK meets stringent compliance but increases operational burden (key escrow, rotation coordination).
• Regional clusters ensure GDPR residency but add deployment complexity and cross-region data synchronization challenges.
• Audit immutability and cryptographic signing increases storage/processing cost but provides non-repudiation.
• Using HSM-backed KMS and Vault increases security at cost of latency; mitigate with DEK caching and local encryption services.

Operational controls:

• Automated tenant onboarding/offboarding workflows to provision keys, enforce residency, and configure audit retention.
• Regular key rotation, penetration testing, and compliance audits.
• SLA-aware routing: fallbacks with strict policy (no cross-region failover for residency-restricted tenants).

This design balances confidentiality, per-tenant control, regulatory residency, and auditability while minimizing performance impact through envelope encryption, caching, and selective isolation.

An audit log is a chronological record of actions (who, what, when, where) useful for forensic analysis and compliance. Immutable or append-only logs are recommended because they prevent tampering and provide verifiable history for regulators.
One pattern: write logs to an append-only storage backed by write-once-read-many (WORM) or immutable object store (e.g., S3 Object Lock in Governance/Compliance mode) and stream them to a tamper-evident SIEM. Another option: use blockchain-backed or integrity-checked logs (e.g., append-only ledger with signed entries). These ensure forensic integrity and meet many compliance requirements.