Provide a clear, evidence based overview of your technical foundation and demonstrated credibility as a technical candidate. Describe programming and scripting languages, frameworks and libraries, databases and data stores, version control systems, operating systems such as Linux and Windows, server and hardware experience, and cloud platforms including Amazon Web Services, Microsoft Azure, and Google Cloud Platform. Explain experience with infrastructure as code tools, containerization and orchestration platforms, monitoring and observability tooling, and deployment and continuous integration and continuous delivery practices. Discuss development workflows, testing strategies, build and release processes, and tooling you use to maintain quality and velocity. For each area, explain the scale and complexity of the systems you worked on, the architectural patterns and design choices you applied, and the performance and reliability trade offs you considered. Give concrete examples of technical challenges you solved with hands on verification details when appropriate such as game engine or platform specifics, and quantify measurable business impact using metrics such as latency reduction, cost savings, increased throughput, improved uptime, or faster time to market. At senior levels emphasize mastery in three to four core technology areas, the complexity and ownership of systems you managed, the scalability and reliability problems you solved, and examples where you led architecture or major technical decisions. Align your examples to the role and product domain to establish relevance, and be honest about gaps and areas you are actively developing.
HardSystem Design
75 practiced
Design an observability architecture for an enterprise producing billions of metrics and traces per day. Discuss strategies for adaptive sampling, metric cardinality control, storage tiering (hot/warm/cold), cost-performance trade-offs, alerting at scale, and ensuring query responsiveness for debugging and dashboards.
Sample Answer
Requirements:- Ingest billions of metrics & traces/day with <5s dashboard latency for hot queries, cost constraints, multi-tenant isolation, retention SLAs (30d hot, 90d warm, 365d cold), and SLO for alerting (<1 min for P99 alerts).High-level architecture:- Ingest layer: API gateways + buffering (Kafka/KSQ) with per-tenant rate limiting.- Processing layer: stream processors (Flink/Beam) for enrichment, dedup, adaptive sampling decisions, and cardinality controls.- Storage layer: hot (time-series DB / OLTP TSDB like VictoriaMetrics or M3 + index for traces), warm (chunked object store with metadata in a search index), cold (S3/Glacier with compacted aggregates).- Query layer: query router that fans requests to hot/warm/cold appropriately, with cached rollups (Redis/Velox) for dashboards.- Control plane: policy engine (tenant, env, resource) + observability UI.Adaptive sampling:- Two-tier sampling: always-sample error/latency anomalies, head-based sampling for traces based on dynamic score (error, latency, traffic, user-impact), tail-based reservoir sampling per trace-id to keep distribution. Configure probabilistic sampling with feedback loop using anomaly detectors in stream processors to raise sampling rates when unusual conditions appear.Metric cardinality control:- Tag/label hygiene enforced at ingestion: denylist/high-cardinality detectors, apply cardinality-limiting transformations (hashing, bucketing, rollups), dynamic cardinality quotas per tenant with soft rejection and fallback aggregation (e.g., top-k tags + "other"). Offer SDK best-practices and pre-ingest validators.Storage tiering & compaction:- Hot: write-optimized TSDB with short-term retention, full-resolution, indexes for fast dashboards and alerting.- Warm: coalesced chunks with time-based downsampling (1m→5m) and index for mid-latency queries.- Cold: compressed aggregates and traces stored in object store with on-demand rehydration; keep minimal searchable metadata for selective retrieval.- Background compaction jobs reduce cardinality and produce rollups; use lifecycle policies to move data.Cost-performance trade-offs:- Higher resolution + long retention = cost; mitigate with tiering, downsampling, and adaptive sampling. Expose SLA tiers to customers. Use burst credit model and pre-ingest transformations to avoid storing useless high-cardinality metrics.Alerting at scale:- Push-based alert evaluator colocated near hot storage for low latency; use streaming evaluation for continuous alerts and batching for low-priority alerts. Deduplicate alerts across dimensions, group by incident-runbook, and apply rate limits and escalation policies. Keep separate read path for alert queries to avoid dashboard interference.Ensuring query responsiveness:- Materialized views & pre-aggregations for common dashboard queries; LRU query cache for recent ranges; query routing to hot tier for recent data; adaptive query planning to avoid expensive full scans (rewrite to rollups when possible). Enforce query cost limits and provide async query APIs for heavy ad-hoc debugging (background job + notify when ready).Operational and governance:- Multi-tenant quotas, audit logs, cost allocation. Autoscaling ingestion & processing, SLAs for data movement tasks. Chaos testing and canary pipelines. Provide SDK linters and onboarding playbook to reduce cardinality at source.Trade-offs and justification:- Prioritize real-time alerting and recent-dashboard latency by keeping hot tier small and aggressive sampling; accept higher cold retrieval latency. This balances cost while preserving fidelity for debugging via on-demand trace retrieval and targeted retention policies.
MediumTechnical
103 practiced
You inherit an unexpectedly high cloud bill concentrated in compute and data transfer. Outline a prioritized, structured approach to identify the largest cost drivers, cut spend quickly by 20% without destabilizing production, and implement long-term cost governance and monitoring.
Sample Answer
High-level approach: triage → quick wins (safe, reversible) → medium-term commitments → long-term governance and monitoring.1) Rapid identification (0–24 hrs)- Pull billing + usage data (aws cost explorer / GCP billing export / Azure Cost Management) and group by service, account/project, region, tag. - Query top 10 cost contributors (compute, data transfer, storage) and show trends (7/30/90d). - Cross-check running inventory: live VMs, autoscaling groups, load balancers, NAT gateways, DB instances, egress-heavy services.2) Safe rapid cuts to hit ~20% (24–72 hrs)Prioritize actions with high $ impact and low risk:- Stop unused/idle resources: stop dev/test instances, unattached volumes, idle load balancers (low risk, immediate savings).- Reduce overprovisioning: temporarily scale down non-prod and batch instance sizes and scale-in ASGs by safe margin (e.g., -30%) with health checks.- Turn off or limit costly data egress: apply routing/ACLs to block nonessential outbound flows, use internal endpoints for backups.- Replace NAT gateways / inter-region transfers: switch to NAT instances or consolidate transit where safe.- Move large ephemeral workloads to spot/ preemptible instances where acceptable (use for fault-tolerant tasks).Each change: run quick impact analysis, apply to staging first, implement change window and roll-back plan.3) Medium-term optimizations (1–8 weeks)- Rightsize via CPU/memory and utilization analysis; implement automated rightsizing recommendations.- Reserved/Committed use: buy RIs/Savings Plans for steady-state instances and databases after validating baseline.- Architect for data locality: co-locate services, use CDN and edge caching to cut egress.- Migrate monolithic workloads to serverless / managed FaaS or autoscaling containers where appropriate.4) Long-term governance & monitoring (ongoing)- Tagging policy + enforcement; cost-center and owner tags mandatory on creation.- Centralized FinOps dashboard (daily alerts on anomalies, forecast vs actual, burn rate by owner).- Chargeback/showback + monthly review with engineers and product owners.- CI/CD gates: cost-impact checks for infra changes, approval workflows for high-cost resources.- Quarterly RI/commitment reviews and continuous rightsizing automation.5) Risks & trade-offs- Balance between savings and reliability—prioritize reversible, low-risk changes first; commit to RIs only after stable usage.- Communicate changes and runbook updates to stakeholders; maintain rollback and monitoring to detect regressions.Outcome: immediate ~20% reduction by eliminating idle resources, rightsizing non-prod, and blocking unnecessary egress; sustained reductions via commitments, architecture changes, and enforced governance.
HardSystem Design
93 practiced
Plan a migration of a monolithic Java application with a stateful relational database to microservices with cloud-managed databases while minimizing downtime. Outline the decomposition approach (strangler pattern), data migration strategies (dual write, CDC), cutover plan, staging and testing, and risk mitigation steps for rollbacks.
Sample Answer
Requirements & constraints- Minimize downtime, preserve data integrity, support transactional semantics where necessary, use cloud-managed DBs (e.g., Aurora/Cloud SQL), meet SLOs for cutover.High-level approach- Use the Strangler Fig pattern to incrementally carve services from the monolith, migrate data to per-service cloud-managed databases, and switch traffic domain-by-domain.Decomposition (Strangler)- Identify bounded contexts (using domain-driven design): order, customer, inventory, billing.- Prioritize low-risk/high-value slices (read-heavy, low-coupling) first.- Create new microservice façade for each context with API contract matching callers.- Route traffic via API gateway / service mesh: route calls for decomposed contexts to new services; otherwise to monolith.Data migration strategies- Database per service on cloud-managed DBs (e.g., Aurora, Cloud SQL, Cloud Spanner) with proper sizing and IAM.- Dual write during transition: - Update both monolith DB and target service DB for writes (use SDKs or a library). - Implement transactional outbox pattern in each writer to avoid partial writes.- Change Data Capture (CDC) for eventual sync: - Use Debezium / AWS DMS / Cloud Dataflow to stream changes from monolith DB to message bus (Kafka, Kinesis). - Consumers apply changes to new service DBs with idempotent handlers and schema translation.- Schema evolution & mapping: - Maintain canonical events or mapping layer; use versioned schemas, protobuf/Avro.- Validation: - Continuous checksum comparisons (e.g., row counts, checksums per key) between source and target.Cutover plan (minimize downtime)- Phased cutover per domain: 1. Deploy new service in shadow mode (reads mirrored, no production writes). 2. Dual-write enabled for controlled subset of users (canary). 3. Monitor consistency metrics; run reconciliation jobs until within tolerance. 4. Flip reads to new service via API gateway/feature flag. 5. Disable writes to monolith for that domain; reads/writes fully handled by microservice.- Use feature flags and gradual traffic shifting (10% → 50% → 100%).- Blue/green DB switchover for strict downtime-sensitive domains: provision new DB, stop writes briefly (maintenance window < few minutes), apply final CDC diff, switch connection string in gateway.Staging & testing- Environment parity: staging mirrors production scale for DB size and traffic patterns.- Contract tests: consumer-driven contract testing to ensure API compatibility.- Integration tests: simulate CDC, dual-writes, reconciliation.- Load and chaos testing: validate performance, failover, network partitions.- Data validation suite: automated checksum, end-to-end business flows, reconciliation reports.Risk mitigation & rollback- Transactional outbox + idempotent consumers to prevent duplicates; maintain retry queues.- Rollback strategies: - If new service fails: disable feature flag and route to monolith (traffic-level rollback). - If data divergence unacceptable: re-enable monolith writes, pause dual-write consumer, run fast reconciliation, or restore from pre-cutover backups. - Keep monolith schema and access layer available until service stable; avoid destructive schema changes until final cutover.- Backout runbooks: pre-defined steps, automation scripts to flip routing, rehydrate DBs, emergency contact list.- Monitoring & alerts: data lag, reconciliation errors, error rate, latency, business KPIs.- DR planning: regular backups, point-in-time recovery, tested restore procedures.Operational considerations- Idempotency keys, tracing (distributed tracing), centralized logging, metrics.- Security: IAM, encryption at rest/in transit, least privilege for CDC connectors.- Governance: migration timelines, roll-forward preferred over rollbacks where possible (compensating transactions), stakeholder communication plan.Why this works- Incremental strangling reduces blast radius, dual-write + CDC ensures data continuity with minimal downtime, feature flags enable safe cutover and immediate rollback, reconciliation and tests ensure correctness before final switch.
EasyTechnical
98 practiced
In Python, implement a function percentile95(latencies) that returns the 95th percentile of a list of latency measurements in milliseconds. Explain time/space complexity and describe approaches to compute this for streaming data or very large datasets without storing all values in memory.
Sample Answer
A straightforward way is to sort the list and pick the 95th-percentile value (interpolating if needed). For large or streaming data I'll describe alternatives (min-heap of top k, Quickselect, and streaming sketches like TDigest / P²).Approach & implementation (simple, exact for in-memory):
python
def percentile95(latencies):
"""
Returns the 95th percentile of a list of numeric latencies (ms).
If list empty -> raise ValueError.
Uses the "nearest-rank" or linear interpolation approach.
"""
if not latencies:
raise ValueError("latencies must be non-empty")
n = len(latencies)
arr = sorted(latencies)
# target index for 0.95 quantile using linear interpolation
# position in 1-based indexing: p*(n-1) + 1 converted to 0-based
pos = 0.95 * (n - 1)
lo = int(pos)
hi = lo + 1
if hi >= n:
return float(arr[lo])
frac = pos - lo
return arr[lo] * (1 - frac) + arr[hi] * frac
Key points:- Time: O(n log n) due to sorting. Space: O(n) to hold sorted array (or O(1) extra if sort in-place).- For exact value faster: use Quickselect to find the k-th order statistic in average O(n) time and O(1) extra space.- For very large/streaming data without storing all values: - Min-heap of size k where k = ceil(0.05 * n) to keep top 5% values; final root is 95th percentile (needs knowing n or tracking ratio). - Streaming algorithms: t-digest or P² algorithm give accurate quantile estimates with bounded memory (good for latency distributions). t-digest handles heavy tails well and is recommended for latency percentiles. - Reduce/merge approach: compute summaries per shard (sketches) and merge.Edge cases: empty list, non-numeric values, very small n (interpolation behavior). For production, prefer t-digest library for high-throughput telemetry.
EasyTechnical
80 practiced
Differentiate logging, metrics, and distributed tracing and give examples of tools (e.g., Prometheus, Grafana, ELK/EFK, Jaeger/Zipkin). For a stateless web service, identify which telemetry you would prioritize initially to define an SLO and why.
Sample Answer
Logging, metrics, and distributed tracing are complementary observability signals:- Logs: Discrete, high-cardinality, textual records of events (errors, user actions). Good for forensic debugging and root-cause detail. Tools: ELK/EFK (Elasticsearch, Logstash/Fluentd, Kibana), Splunk.- Metrics: Aggregated numeric time series (counts, latencies, error rates). Low-cardinality, optimized for alerting and SLOs. Tools: Prometheus (collection + alerting), Grafana (visualization), Graphite.- Distributed tracing: End-to-end, per-request causal chains that show latency breakdown across services and dependencies. Essential for performance debugging in microservices. Tools: Jaeger, Zipkin, OpenTelemetry (collector + SDKs).For a stateless web service I would prioritize metrics first to define an SLO—specifically request success rate and latency percentiles (e.g., p95/p99) collected by Prometheus and visualized in Grafana. Rationale: SLOs need quantitative, low-cost, high-signal metrics that are easy to aggregate and alert on. Start with: 99.9% successful responses over a rolling 30-day window and p95 latency < 200ms. Logs are secondary for investigating SLO breaches; tracing becomes important as the service integrates with other services or shows transient latency spikes. This staged approach balances quick SLO definition, low overhead, and practical operational value.
Unlock Full Question Bank
Get access to hundreds of Technical Background and Skills interview questions and detailed answers.
