DoorDash Staff Security Architect Interview Preparation Guide

Direct answer

In ordinary TLS (Transport Layer Security), only the server proves its identity to the client, via a certificate; the client is typically anonymous at the TLS layer, and any client authentication happens later, at the application layer, for example with a password or token. Mutual TLS, or mTLS, extends the same certificate-based handshake so both sides present a certificate: the server proves who it is to the client, and the client proves who it is to the server, before any application data flows. This makes mTLS a natural fit for service-to-service authentication in a microservice fleet, where "the client" is itself another service that needs to prove its identity, not a human typing a password.

Structured elaboration

How mTLS authenticates both sides. During the handshake, the server presents its certificate as usual. The server then additionally requests a certificate from the client, the client presents its own certificate, and the client proves possession of the matching private key by signing part of the handshake transcript. Each side then validates the other's certificate chain against its own trusted CA (certificate authority) set, the same chain-of-trust check used in any PKI (public key infrastructure): does this certificate chain up to a CA I trust, is it within its validity period, and, ideally, has it not been revoked. Only if both checks pass does the connection proceed, and the "identity" each side now has for the other is whatever is encoded in the peer's certificate, for example a service name, which downstream authorization logic can then use.

Operational workflow to bootstrap and manage mTLS certificates across a fleet.

1. Establish the CA hierarchy. Decide on, or reuse, a root CA and at least one issuing intermediate CA whose chain every service in the fleet will trust. Populate each service's trust store with this CA chain, not with every individual peer's certificate; this is what makes it scale, since a new service only needs the CA chain, not a certificate for every other service it might talk to.
2. Bootstrap trust. Every service needs some initial way to prove its own identity to the CA in order to get its first certificate issued. Options range from a manual, secure provisioning step at deploy time to an automated attestation the issuing CA can validate, for example a cloud platform's instance-identity document, or a Kubernetes ServiceAccount token.
3. Issuance. Each service, or an agent or sidecar acting on its behalf, requests a certificate for its own identity from the issuing CA and gets back a leaf certificate, typically for a key pair it generated locally so the private key never has to travel over the network.
4. Distribution. The issued certificate, and the CA chain if not already provisioned, is placed wherever the service's TLS stack reads it from: a local file, a secret store, or a sidecar proxy's configuration.
5. Rotation. Certificates are reissued before expiry; the service, or its sidecar, needs to pick up the new certificate and start presenting it, ideally without a restart or a dropped connection.
6. Ongoing trust maintenance. If the CA hierarchy itself ever needs to change, a new intermediate or a root rotation, that new trust material has to reach every service's trust store, which is the slower, fleet-wide operation.

sequenceDiagram
    participant O as Service: orders
    participant P as Service: payments
    O->>P: TLS ClientHello
    P-->>O: Server certificate (payments.internal)
    O->>O: Verify server cert against trusted CA chain
    P-->>O: CertificateRequest
    O-->>P: Client certificate (orders.internal)
    P->>P: Verify client cert against trusted CA chain
    O->>P: Encrypted application request, mTLS established

Worked example

Service orders calls service payments. orders initiates a TLS connection; payments presents its certificate, subject payments.internal, issued by Corp Issuing CA. orders checks that chain against its trust store, which holds Corp Issuing CA's certificate, and it's valid. payments then requests a client certificate; orders presents its own certificate, subject orders.internal, from the same issuing CA, and payments checks it the same way, also valid. Both sides also confirm the peer's certificate hasn't expired and isn't revoked. The handshake completes, and payments's application logic now knows, from the verified certificate rather than from a token it merely trusts, that this connection genuinely came from orders, so it can apply an authorization rule keyed on that verified identity, for example, "only orders may call the refund endpoint."

Trade-offs and pitfalls

Populating trust stores with individual peer certificates instead of the shared CA chain doesn't scale, since N services would each need N-1 entries, and it makes rotating any single service's certificate require updating every other service's trust store, defeating the point of having a CA hierarchy at all.

Skipping revocation or expiry checking on the client-certificate side is a common gap: teams often validate the server certificate carefully, since browsers do it by default, but roll their own client-side validation less carefully. An expired or revoked client certificate that's still accepted defeats mTLS's authentication guarantee.

Treating mTLS as sufficient authorization by itself is a category error: mTLS proves who is calling, authentication, but it doesn't decide what that caller is allowed to do, authorization, which is a separate policy layer that consumes the verified identity as input.

Skipping automated rotation and bootstrap doesn't scale past a handful of services; manually managing certificates across a real fleet becomes the actual bottleneck to adopting mTLS broadly.

Approach (brief)
I assess transitive-dependency risk by combining automated scanning, policy controls, build-time hardening, and runtime defenses — enforced via CI/CD gates and telemetry.

Assessment & Tooling

• Use SBOM generation (Syft) and dependency scanners: Snyk, Dependabot, OSS Index, WhiteSource/Bom-Tools for license/vuln mapping.
• Napkin risk score = severity + age of vulnerability + usage exposure (public API vs test-only). I prioritize based on exploitability and reach.

Policy Controls

• Whitelist approved packages and versions for core layers; blacklist known-broken packages and risky licenses.
• Enforce via policy-as-code (Open Policy Agent) integrated with artifact registries (Artifactory, Nexus).

Build-time Mitigations

• CI job: generate SBOM, run static SCA, fail builds on high/critical CVEs or disallowed transitive packages.
• Use reproducible builds, dependency pinning, and minimal dependency profiles (exclude dev/test transitive deps).

Runtime Mitigations

• Immutable minimal runtime images, runtime instrumentation (eBPF, Falco) to detect exploitation patterns.
• Use library isolation (shading, vendoring) and allowlist syscalls with seccomp, AppArmor/SELinux, and WAF for web apps.

CI/CD Enforcement

• Gate merges with OPA policies and SCA checks; auto-create PRs for fixable issues, block otherwise.
• Block promotion to staging/prod unless SBOM and vulnerability thresholds met; pipeline audit logging to SIEM.

Outcome & Metrics

• Track mean time to remediate, vulnerable-dep exposure, SBOM coverage. I balance security with developer velocity via automation and clear exception workflows.

Overview (positioned as Security Architect)
I would define a risk-driven vulnerability management process that integrates into CI/CD, enforcement at the registry/runtime, prioritized remediation, safe rollouts, and emergency playbooks.

CI: image scanning & gating

• Enforce pipeline scanning (Trivy/Clair/Anchore) at build + PR.
• Fail builds on policy violations (e.g., high/critical CVEs, secret leaks).
• Sign images (Cosign/Notary) and attach SBOMs to artifacts.

Registry admission policies

• Registry-level admission with Gatekeeper/OPA: reject unsigned images, block images > policy score, block base-images on allowlist/denylist.
• Attach metadata: SBOM, scan timestamp, build ID.

CVE prioritization (scoring)

• Prioritize by: exploitability (E, public exploit or EoP), runtime exposure (network-facing, service mesh ingress), service criticality (business impact), privilege level, and compensating controls.
• Produce a numeric score: Priority = f(Exploitability, Exposure, Criticality) to drive SLAs (e.g., patch within 7 days for score > 80).

Patch rollout with canarying

• Automated canary cohorts: deploy patched image to small percentage, run smoke tests and runtime security rules (Falco, eBPF-based checks), monitor metrics (errors, latency, security alerts) for a predefined window, then progressive rollout with automated rollback on anomalies.
• Use feature flags and traffic shifting (Istio/Linkerd) to limit blast radius.

Emergency mitigation plan

• Fast paths: image rollback to last signed build; network-level mitigations (K8s NetworkPolicy, service mesh deny rules); runtime controls (kill/ quarantine pods via orchestration or eBPF); WAF rules or IPS signatures if applicable.
• Incident runbook with roles, decision criteria, and communication templates.

KPIs (measure effectiveness)

• Mean Time to Remediate critical CVEs (MTTR) — target < 7 days.
• % of images scanned and SBOM-attached before registry push — target 100%.
• % of deployments passing admission policy — target 99% enforced.
• % of successful canary rollouts without rollback — trend upward.
• Number of production exploit detections vs. pre-production detections (ratio should decrease).

This approach ties technical controls to risk and business impact while enabling measurable, fast, and safe remediation.

Direct answer
Acceptance criteria for a corrective action should be a specific, measurable, time-boxed statement of what 'fixed' looks like, defined before the work starts, not after. A verification plan then confirms that criterion is actually met using real data, not just confidence that the fix was implemented correctly.

Structured elaboration

• Define the metric and target explicitly. Not 'reduce timeouts' but 'reduce the service's timeout rate from its current baseline to a specific target percentage, measured over a specific window.' A vague criterion can't be verified; a specific one can.
• Set a monitoring window long enough to be meaningful. Too short a window risks declaring success on noise; too long delays knowing whether the fix worked. The right window depends on the incident's natural frequency, for example enough days to capture a representative mix of peak and off-peak traffic.
• Separate short, medium, and long-term verification. Immediately after deploying the fix: a targeted test or synthetic check confirms the mechanism works as intended. Over the following weeks: real production monitoring against the target metric confirms it holds under real conditions, not just in a controlled test. Longer term: a periodic audit or scheduled re-check confirms the improvement is durable and hasn't quietly regressed.
• Define what "success" and "failure" mean numerically in advance, including what would trigger reopening the item if the target isn't met, so there's no ambiguity or motivated reasoning once the data comes in.
• Name who signs off, so verification isn't just a self-assessment by whoever implemented the fix.

Worked example
A corrective action targets reducing a service's timeout rate from 0.5% to 0.05% within 30 days. Acceptance criteria: timeout rate, measured as a 7-day rolling average, must be at or below 0.05% for two consecutive weeks within the 30-day window, using the same monitoring dashboard and definition of 'timeout' used to measure the original 0.5% baseline. Verification plan: short-term, a synthetic load test immediately after deploy confirms the fix reduces timeout rate under simulated peak load; medium-term, the real 7-day rolling average is checked weekly against the target for the full 30 days; long-term, the metric is re-checked at 90 days to confirm it hasn't quietly crept back up as traffic patterns shift. If the 30-day window ends with the metric at 0.15%, that's a defined failure, not an ambiguous 'mostly worked,' and it triggers a re-investigation of whether the fix addressed the actual root cause or only a symptom.

Trade-offs and pitfalls
The most common mistake is defining acceptance criteria loosely enough that almost any outcome can be called success, which defeats the purpose of having criteria at all. A second is skipping the longer-term recheck: many fixes look successful in the first two weeks and then quietly regress as conditions change, and without a scheduled longer-term verification, that regression goes unnoticed until the incident recurs.

Definition (plain business language)
Residual risk is the risk that remains after you’ve applied all reasonable security controls. It’s what could still go wrong even after you invest in prevention, detection and response.

How an executive should decide to accept it

• Assess business impact: quantify financial, legal, reputational consequences.
• Compare cost and feasibility of further mitigation vs expected loss (cost-benefit).
• Consider likelihood after controls, regulatory obligations, and risk appetite.
• Prefer acceptance when additional controls are disproportionately expensive, would block core business, or introduce unacceptable complexity.
• Require time-bound compensating controls and monitoring if accepted.

Illustrative example
A cloud app stores low-sensitivity customer preferences. Encrypting every field and re-architecting would cost $500k and delay product launch 9 months. Residual risk of a limited data exposure is low-impact and within company risk appetite, so leadership accepts residual risk to preserve revenue.

Documentation & approvals

• Complete a Risk Acceptance Form: risk description, controls applied, likelihood/impact, quantitative estimate, compensating controls, review date.
• Sign-offs: Risk Owner, CISO/Security Architect, CRO and Business Unit Executive; if high severity, CFO/CEO or Board/Risk Committee approval.
• Attach to risk register and schedule periodic reassessment and monitoring metrics.

Summary approach
Identify and prioritize business-critical assets, map likely threat actors to each, list common attack vectors, and explain why each risk undermines safety, revenue, or trust.

Top 5 assets (priority order)

1. User PII & payment data — financial loss, regulatory fines, reputational damage
2. Courier identities & background-check data — safety, liability, fraud prevention
3. Order fulfillment & transaction system (matching/payments) — revenue integrity and service availability
4. Delivery tracking & location telemetry — user safety, stalking risks, fraud disputes
5. Internal admin systems / credentials — ability to escalate impact and persist

Threat actors & mappings

• External attackers: data exfiltration (PII, payments), DDoS on ordering systems
• Fraud rings: fake accounts, chargebacks, GPS spoofing to steal orders
• Malicious couriers: location spoofing, order diversion, privacy violations
• Insiders: misuse of admin access to view PII, manipulate orders/payments

Common attack vectors

• Phishing / credential stuffing → stolen accounts/admin access
• API abuse & insufficient auth → order/payment manipulation
• Payment fraud / synthetic identities → chargebacks, revenue loss
• GPS spoofing / app tampering → misdeliveries and safety incidents
• SQLi/Exfiltration and misconfigured S3 → PII leaks

Why critical
Each risk directly impacts user/courier safety, regulatory exposure, revenue, and marketplace trust. Prioritize protecting PII/payments and hardening auth, then telemetry integrity, fraud-detection, and least-privilege for internal systems.

Clarify requirements & constraints

• Multi‑tenant: per‑customer retention windows configurable
• Support immediate erasure requests (GDPR) that can override retention
• Legal‑hold can freeze deletion for specific records or entire tenant
• Scale: millions of accounts, high throughput, low-latency reads

High‑level data lifecycle

• Ingest -> Active -> Soft‑deleted (tombstone + hidden) -> Eligible for purge -> Physical purge / crypto‑erase
• Legal‑hold flag interrupts transition to purge; immediate erase request sets urgent deletion flow

Metadata model

• Per record: tenant_id, created_at, retention_expiry, soft_deleted_at, legal_hold_id(s), erasure_request_id, deletion_state (active, tombstoned, queued, purged), audit_log_ref
• Per tenant config: default_retention_days, min/max caps, retention_policy_version

Background jobs & orchestration

• Scheduler service (distributed, leader‑election) scans expiry index partitioned by tenant shards and enqueues purge tasks into a durable queue (Kafka/SQS)
• Worker pool processes tasks: check legal holds, pending erasure, backoff and retry, mark as queued/processing
• Immediate erase API enqueues high‑priority tasks; synchronous verification returns receipt and audit id
• Legal‑hold service manages holds, notifies scheduler to cancel queued purges; keeps immutable hold history

Safe deletion approaches

• Two‑phase: soft‑delete (tombstone + hide) then delayed physical purge after verification
• For strong guarantees: crypto‑erase — encrypt per‑tenant or per‑record keys so deleting keys renders data unrecoverable (fast at scale)
• Wipe pointers in indexes, redact metadata, remove from backups per retention
• Maintain append‑only audit log (WORM) with minimal necessary retention for compliance; redact sensitive fields where regulations allow

Consistency, compliance & audit

• Immutable audit trail with proofs: erasure_request_id, timestamps, operator/service ids, hashes of deleted objects
• Provide verifiable receipts/certificates for erasure
• Role‑based access for deletion operations; approval workflows for exception cases

Performance & scalability

• Partition expiry index by tenant ranges and time buckets; use TTL indexes where supported
• Rate limit high‑priority erasures to control I/O; autoscale worker pools; use bulk deletes for cold data
• Use eventual consistency for background purges; synchronous checks for immediate erasure requests
• Offload cold data to cheaper object store with lifecycle policies; purge there via serverless bulk jobs or crypto‑erase keys

Failure modes & testing

• Idempotent workers, at‑least‑once semantics with dedupe by erasure_request_id
• Chaos testing for legal‑hold conflicts, network partitions, and backup restores to ensure deleted data not resurrected
• Regular compliance audits, retention drift detection, and alerting on backlog growth

Tradeoffs

• Crypto‑erase is fast but requires secure key management (HSM) and careful key rotation policies
• Immediate synchronous physical deletion increases latency; use async with strong receipts and SLA tradeoffs

This architecture balances legal compliance, provable erasure, operational safety, and scalability for millions of tenants.

Overview
Design a controlled, auditable exception and compensating control (ECC) framework that treats exceptions as temporary, risk-accepted deviations with full evidence, approval, monitoring, and renewal.

Lifecycle (steps)

1. Request — submit RFC in ticket system with business justification, affected assets, risk statement, duration request.
2. Triage — InfoSec evaluates risk, identifies required compensating controls, assigns owner and classification (Low/Med/High).
3. Approval — mapped to authority matrix (see below).
4. Implement — implement compensating controls, record configuration, instrument monitoring.
5. Validate — independent validation (security ops/third-party pen test or config review).
6. Monitor — continuous telemetry and alerts tied to exception.
7. Renew/Close — automatic expiry; renewal requires re-evaluation and fresh evidence.
8. Audit — periodic audit trail review and executive reporting.

Required Evidence

• Business impact and mitigation plan
• Asset inventory and config snapshots (screenshots, configs, hashes)
• Risk assessment and residual risk calculation
• Compensating control design, implementation evidence, and test results
• Monitoring/alert rules and recent telemetry
• Approval artifacts and timestamps

Approval Authorities

• Low risk: System owner + InfoSec reviewer
• Medium: InfoSec manager + Risk owner + App/Product owner
• High: CISO + Business Line Executive + Legal/Compliance

Compensating Controls (examples)

• Network micro-segmentation and ACLs in lieu of host patching
• Application-layer WAF and RASP when library cannot be updated immediately
• Enhanced logging/endpoint EDR and threat hunting when privileged access exceptions exist
• Time-bound Just-In-Time access with MFA and session recording instead of standing admin accounts

Renewal Cadence

• Low: 90 days; Medium: 30–60 days; High: 7–14 days
• Auto-expire; renew requires fresh evidence and escalated approvals

Reporting & Auditability

• Immutable audit trail in ticketing/CMDB with cryptographic integrity (WORM or append-only logs)
• Weekly exception dashboard for SOC and monthly executive risk report (counts by severity, age, compensating control effectiveness, outstanding high-risk)
• Quarterly audit package with sampled evidence, validation results, and RCA
• KPIs: time-to-remediate, percent renewed, compensating-control effectiveness, exceptions-by-owner

This framework balances business needs with risk governance and provides clear, auditable evidence for auditors and executives.

Situation & goal
Design an authz solution that handles 1M decisions/sec with low latency while bounding risk from stale cache entries.

High-level analysis

• Baseline: a single synchronous PDP per request at 1M/s is infeasible (CPU, network, and TLS overhead). Aim: p95 decision latency <10ms and availability >99.99%.

Strategies

• Caching (gateway-side & tokenized)

  • Short-lived cache entries keyed by (subject, resource, action, context hash). TTLs: 100–1000ms for high-change items, 60s for stable attributes. Use LRU + size caps. Evict on policy-change events.
  • Use cryptographically-signed decision tokens (opaque JWT-like) for offline validation up to TTL to avoid round-trips.

• PDP topology (shards & replicas)

  • Shard by tenant/resource namespace to distribute 1M/s across N shards; scale replicas per shard for read capacity. Use consistent hashing to route.
  • Keep each shard’s policy set local; propagate policy updates via ordered log (e.g., Kafka) to maintain deterministic versioning.

• Batching & async paths

  • Batch authorization queries where possible (bulk resource lists). Use coarse-grained allow-lists for frequently-accessed resources.
  • Provide fast-path optimistic allow: evaluate cached allow, proceed, background recheck; on mismatch, provide compensating action (revoke, audit).

• Consistency vs security trade-offs

  • Modes: strict synchronous for high-risk ops (financial, admin) — no caching; eventual for low-risk with short TTLs.
  • Policy-change propagation: include version stamps in tokens; PDP rejects tokens with stale-min-version for critical ops.

Metrics & monitoring

• Track cache hit ratio, decision latency p50/p95/p99, policy-propagation lag, stale-decision incidents. Set SLOs and automated revocation thresholds.

Risk mitigation

• Short TTLs for sensitive attributes, policy change invalidation hooks, anomaly detection to catch stale-allow spikes, and audit trails for forensic rollback.

Overview / assumptions
I’d treat SOC 2 Type II readiness as a 9-month program with risk-driven priorities: map current state, implement core security controls (policies, identity, logging, change control), automate evidence collection, and run continuous readiness checks. Goal: produce 6+ months of collected evidence for an auditor by month 9.

90‑day roadmap (initial sprint cadence)

• Week 0–2: Gap assessment & scoping
  • Map systems, data flows, and in-scope services; map gaps to Trust Services Criteria.
  • Identify owners; baseline risk register.
• Week 3–8: Policy & control foundation
  • Publish core policies: Access Control, Change Management, Logging/Retention, Incident Response, Vendor Management.
  • Define control objectives and control owners.
• Week 6–12: Logging, identity, and evidence pipelines
  • Centralize logs (SIEM/ELK) for in-scope systems, enable syslog/cloud audit trails; set 90‑day retention minimum.
  • Enforce SSO + MFA; implement least privilege RBAC.
  • Deploy automated evidence collection (GRC/tooling or scripts) for user access, change tickets, and config snapshots.
• Deliverables by day 90 (explicit)
  • Completed gap analysis and prioritized remediation backlog.
  • Published core policies and assigned owners.
  • Centralized logging for primary workloads and pilot retention/alerts.
  • IGA/SSO + MFA enforced for critical apps.
  • Automated evidence exports for access reviews and change control.
  • Runbook for auditor requests and a 30/60/90 remediation plan.

Integrating controls without blocking delivery

• Shift-left security: embed automated checks into CI/CD (linting, SAST, secret scanners, infra-as-code policy-as-code e.g., Open Policy Agent).
• Make controls as pipelines: PR gating for policy-as-code failures, not for minor infra warnings—use fail-fast for high risk, advisory for low risk with SLA to remediate.
• Use developer-friendly guardrails: IaC modules with secure defaults, pre-approved templates, and self-service role provisioning with time-bound elevation.
• Evidence automation: capture artifacts at runtime (build artifacts, signed deployment manifests, audit trail links) and surface them in dashboards so engineers don’t need manual exports.
• Governance cadence: weekly sync with product squads, biweekly risk review, and a "fast path" exception process with compensating controls to avoid release delays.

Measurement & next steps

• KPIs: % of in-scope systems logging, % of accounts on MFA/SSO, time-to-produce auditor artifact, percent of automated evidence.
• Plan months 4–9: remediate backlog, run internal controls testing, collect 6 months of continuous evidence, engage external auditor for readiness gap review, then Type II audit.