Technology and Platform Selection Questions

Evaluation and justification of technologies services and platforms used to implement systems across the stack. Candidates should be able to select compute options including virtual machines containers and serverless platforms as well as orchestration and workflow engines messaging systems batch and streaming processing engines object and block storage data warehouses and other data platforms. The topic encompasses comparing managed services and self managed deployments cloud versus on premise hosting and choosing frameworks runtimes and overall stacks based on workload characteristics. Assessment focuses on weighing trade offs across cost operational overhead reliability latency and throughput scaling characteristics vendor lock in development velocity team familiarity and learning curve maturity and community support security and compliance and monitoring and debugging complexity. Candidates should demonstrate how system requirements map to service capabilities justify build versus buy decisions and managed service choices design proof of concept experiments and outline migration and rollout planning while making pragmatic choices that balance performance cost and operational risk.

MediumTechnical

59 practiced

Design an instrumentation plan for a new service to make it portable across observability backends. Which libraries, formats, and design patterns (for example OpenTelemetry, structured logs, semantic conventions) will you choose so the service remains vendor-neutral and easy to migrate?

Sample Answer

Requirements:- Vendor-neutral, portable across backends (Datadog, New Relic, Splunk, Prometheus, Jaeger)- Minimal app code changes to swap backends- Consistent context propagation and cross-signal correlation (logs, traces, metrics)- Low overhead and configurable sampling/aggregation

Design choices (libraries & formats):- OpenTelemetry (OTel) SDKs for tracing, metrics, and context propagation — language-specific SDKs (python/java/go)- OTLP (gRPC/HTTP) as primary export protocol to collector- OpenTelemetry Collector as a configurable vendor-agnostic shim to translate to vendor formats/exporters- Structured JSON logs with semantic fields; include trace_id/span_id and service attributes- Prometheus metrics exposition for scraping where applicable; use OTel Metrics with Prometheus exporter when needed- Semantic conventions (OTel semantic attributes) to ensure consistent names across services

Patterns & implementation:- Instrument using OTel APIs and auto-instrumentation where available; prefer manual spans around business-critical operations.- Use context propagation (W3C Trace Context) for incoming/outgoing requests and message queues.- Correlate logs and traces: inject trace_id/span_id into structured logs automatically via a log exporter or middleware.- Centralize configuration: env vars or config file to swap exporters, sampling, and collector endpoints without code changes.- Use resource attributes (service.name, service.version, deployment.environment) to tag telemetry.

Operational practices:- Route all telemetry through the OTel Collector; add processors (batch, tail-sampling) for cost control.- Define semantic-level SLO/alerting metrics (error rate, latency p95/p99, saturation) and create dashboards in chosen backend.- Testing: smoke instrumentation in dev using local Jaeger/Prometheus, run end-to-end test validating trace+log correlation.- Migration: change collector pipeline/exporter to new vendor — no app code change if using OTLP.

Example (pseudo-config):- App uses OTel SDK -> exports OTLP to Collector.- Collector pipeline: OTLP receiver -> processors (batch, attributes) -> exporter (Prometheus/jaeger/vendor).

Why this works:- OpenTelemetry + Collector decouples app from vendor, standardized semantic conventions ensure consistency, structured logs + trace IDs guarantee cross-signal correlation, and centralized configuration enables easy migration with minimal code work.

HardTechnical

52 practiced

Propose a quantified lock-in mitigation strategy for deep use of proprietary cloud services. Include modular architecture patterns, abstraction layers, data export guarantees, periodic migration drills, and cost/effort estimates to migrate components later. Suggest metrics to track lock-in risk over time.

Sample Answer

Situation/Goal: Reduce vendor-lock‑in risk while using proprietary cloud services heavily, by designing modular systems, measurable controls, and rehearsed escape paths so migrations remain feasible within quantified SLAs.

Proposed strategy (high-level):- Modular architecture: split system into “portable core” (stateless app, container runtime, message formats) and “cloud-optimized extensions” (managed DB, ML service, serverless). Communicate via well-defined APIs/events and versioned interface contracts.- Abstraction layers: - Resource abstraction: use Kubernetes + CNI/CSI to decouple compute/network/storage from cloud provider APIs. - Service adapter layer: implement internal facade libraries (thin wrappers) around cloud SDKs exposing generic interfaces (e.g., KeyValueStore, ObjectStore, Queue). Keep adapters isolated per team. - Data format guarantees: enforce open, documented schemas (JSON/Avro/Parquet), and use standardized auth (OIDC/JWT).- Data export guarantees: - Define SLAs: every managed data store must support full consistent snapshots/export in open formats within X hours. Contractually negotiate egress APIs and export rate guarantees with vendor. - Implement automated export jobs that produce monthly and pre-cutover snapshots; validate integrity via checksums.- Periodic migration drills: - Run quarterly “mini-drills”: restore exported snapshots into equivalent OSS targets (Postgres, MinIO, Kafka) in isolated accounts; full dry-run migration annually for critical services. - Drill checklist: export → ingest → smoke tests → performance benchmark → runbook validation.- Cost/effort estimates: - Maintain an escape backlog with per-component estimates (dev hours, infra hours, data transfer $): e.g., managed DB (Postgres-compatible) — 400 engineer-hours + 10 TB egress = $1.5k egress + infra; serverless functions — 200 hours to containerize + CI changes. - Update estimates after each drill; keep contingency factor (1.5x).- Metrics to track lock-in risk: - Lock-In Exposure Index (LEI) = Σ(component_weight * dependency_score). Component_weight = business criticality (0–1). Dependency_score = 0(no lock)–1(proprietary-only). - Data Egress Readiness: % of TB with validated open-format exports and successful restore tests. - Adapter Coverage: % of cloud SDK calls wrapped by internal adapters. - Drill Maturity: % of drill steps automated; mean time to restore (MTTRestore) from export to running system. - Estimated Migration Cost (EMC): rolling sum of current escape backlog $.Implementation notes:- Contractual: push export and egress SLAs into procurement.- Automation: CI pipelines to run export+restore smoke tests; alert when LEI rises or EMC exceeds threshold.This approach balances operational reliability with a quantified, test-driven escape posture that SREs can measure, improve, and present to leadership.

MediumSystem Design

51 practiced

When choosing between a vendor-hosted observability platform and assembling open-source components (Prometheus, Grafana, Jaeger, Loki), how do you evaluate trade-offs in scalability, federation, query latency, feature completeness, cost, and debugging complexity?

Sample Answer

Requirements (clarify): collect metrics, logs, traces; scale to N clusters and M tenants; support federation/long-term storage; alerting SLAs (<30s for critical); budget and team ops capacity.

High-level evaluation framework:- Scalability & federation: - Vendor: typically offers managed ingestion scaling, multi-tenant isolation, and cross-region federation out of the box. Good when you need rapid scale without hiring dedicated ops. - OSS stack: Prometheus is pull-based and single-node by design; use Thanos/Cortex for global view and long-term storage — more control but operationally heavy (HA, compaction, object-store tuning).- Query latency: - Vendor: optimized and tuned backends reduce tail latency; SLAs for query response. - OSS: low for local Prometheus; higher for federated/long-range queries (Thanos store/gateway adds latency). Requires caching and query sharding to meet SLAs.- Feature completeness: - Vendor: richer UI, integrated alerting, anomaly detection, RBAC, billing-grade retention, SaaS-only features. - OSS: highly composable—Grafana, Loki, Jaeger provide parity for most needs but integrations and advanced features (ML alerts, synthetic monitoring) may require extra tools.- Cost: - Vendor: predictable OPEX; can be expensive at high ingest/query rates. - OSS: cheaper DIY for large volume if you amortize infra and engineering time; hidden costs: SRE headcount, backups, upgrades, storage egress.- Debugging complexity & operability: - Vendor: simpler incident response (console, support), but less visibility into internals; vendor outages are out of your control. - OSS: deep visibility into pipeline, easier custom instrumentation, but debugging distributed components (compactors, queriers, ingesters) adds complexity.

Decision principles (SRE-focused):- If team lacks capacity to run distributed storage and needs fast time-to-value, choose vendor.- If cost at scale or strict data residency/compliance is primary and you can staff SREs, choose OSS with Thanos/Cortex + Grafana + Jaeger + Loki.- Hybrid: local Prometheus per cluster for low-latency alerting + vendor for long-term analytics and cross-cluster queries.

Concrete example: run Prometheus per-cluster with Alertmanager for SLO alerts, use Thanos for object-store-backed long-term metrics if you want full control; otherwise forward metrics to vendor for analytics and save operational cost.

Trade-offs summary:- Vendor = lower ops, faster features, higher OPEX.- OSS = more control, lower variable cost at scale, higher engineering overhead and debugging complexity.

MediumTechnical

58 practiced

Design a proof-of-concept to compare a managed streaming service against self-managed Kafka. Define the experiment setup, which metrics to collect (throughput, p99 latency, operational time), failure-injection scenarios, and success criteria to inform a build-versus-buy decision.

Sample Answer

Requirements & scope:- Compare managed streaming (e.g., Confluent Cloud / MSK Serverless) vs self-managed Kafka cluster for our primary workloads over 4 weeks.- Goals: measurable performance, operational effort, reliability under failure, cost-to-run. Validate fit for production SLOs (throughput, p99 latency, availability, ops time).

Experiment setup:- Two isolated environments with identical client workload generators: - Managed: provision equivalent cluster size/config per vendor guidance. - Self-managed: Kubernetes + statefulsets or VMs, same Kafka version, Zookeeper/KRaft as used in prod.- Shared benchmarking harness: producers/consumers written in Go or Java, configurable message sizes (256B, 1KB, 10KB), partition counts, replication factor, key distribution.- Run baseline steady-state tests, peak tests (traffic bursts), and long-run soak (48–72h).

Metrics to collect:- Throughput: msgs/sec and MB/sec (per-topic, per-partition).- Latency: p50/p95/p99 producer ack latency and end-to-end consumer latency.- Durability: commit lag, unacked messages.- Availability: successful publish rate, consumer errors, interruptions.- Resource utilization: CPU, memory, disk I/O, network, GC pauses (self-managed).- Operational time: hands-on minutes for setup, scaling, patching, recovery; number of runbook steps executed.- Cost: cloud costs, instance hours, managed service charges, and eng ops labor estimate.- Observability quality: richness of metrics/logs/traces and alerting ease.

Failure-injection scenarios:- Broker failure: kill leader broker(s) during steady and peak load.- Network partition: isolate subset of brokers or throttle bandwidth.- Disk full / high I/O latency on broker.- Zookeeper/KRaft metadata node failure (self-managed) or control-plane outage (managed).- Latency spikes in cloud network; simulate delayed acknowledgements.- Rolling upgrade with schema changes and client reconnects.

Measurement methodology:- Run each scenario 3x, collect metrics + timelines of recovery, mean time to detect & recover, partial/complete data loss events.- Record operator actions and time-to-restore; test runbook accuracy.

Success criteria (inform build vs buy):- Performance: managed meets or exceeds required throughput and p99 latency under baseline/peak.- Reliability: managed recovers from failures within target RTO (e.g., <2 min) without data loss and with lower MTTD/MTTR than self-managed.- Operational burden: managed reduces average weekly ops time by ≥50% or reduces critical incident count by meaningful margin.- Cost: total cost (service + engineering) <= self-managed TCO over 1 year, or extra cost justified by reduction in ops/time-to-market.- Observability & SLA: managed provides required telemetry and contractual SLAs that align with SLOs.

Deliverable:- Report with graphs (throughput/latency over time), failure timelines, ops time log, TCO spreadsheet, recommendation with clear trade-offs and next steps (pilot/gradual migration or invest in self-managed tooling).

MediumTechnical

67 practiced

Given a team building compute tasks in Python and Java, how would you choose runtimes and frameworks (e.g., container images, serverless runtimes, JVM tuning vs GraalVM) to balance startup time, steady-state performance, and maintainability for both batch and low-latency services?

Sample Answer

Clarify requirements: quantify latency SLOs, throughput, concurrency patterns, deployment cadence, team skills, and operational SLAs. Then pick runtimes by workload class.

Batch jobs- Prioritize throughput, resource efficiency and cost. Use standard OpenJDK containers (slim images) or CPython base images with optimized libraries. Prefer JVM with tuned GC (G1/ZGC) if long-running heavy CPU work benefits from JIT; tune heap limits, ergonomics, and use tiered compilation. For short-lived batch tasks, use warm worker pools (K8s Job controllers with pre-warmed pods) rather than cold serverless to avoid startup penalties.- Consider GraalVM native-image for CPU-bound CLI-like batch binaries where startup and memory matter; trade-off: longer build complexity and limited dynamic features.

Low-latency services- Prioritize startup and tail-latency. For Java microservices, evaluate GraalVM native images to reduce startup and memory footprint, improving cold-starts and container density; keep a fallback to JVM with aggressive tuning (small heap, pauseless GC, -XX:+UseStringDeduplication) if native-image incompatibilities exist. For Python, use lightweight ASGI servers (uvicorn, hypercorn) with async frameworks (FastAPI) and run with pypy where compatible for better steady-state throughput; for extreme latencies, consider converting hot paths to Java/Go or compiled extensions.- For serverless: use provisioned concurrency to meet strict cold-start SLOs or prefer containers on K8s with HPA + Pod Disruption Budgets for predictable behavior.

Maintainability & operational practices- Standardize base images, CI buildpacks, and language versions; codify JVM flags and Graal build steps in CI. Provide observability: export runtime metrics (startup time, GC pauses, RSS, heap), request latencies, and error budgets. Automate performance regression tests in CI (benchmarks, memory profiles) and document trade-offs per service.- Decision matrix: map each service by latency sensitivity, runtime cost, team expertise, and library needs; choose GraalVM/native only when measurable benefit outweighs build/maintenance cost.

Unlock Full Question Bank

Get access to hundreds of Technology and Platform Selection interview questions and detailed answers.

Join thousands of developers preparing for their dream job.