Understanding the Company's Infrastructure Context Questions
Research the company's public infrastructure information (engineering blog, tech talks, published case studies, job description). Understand what systems they operate at scale, what problems they likely face, and what your role would contribute to.
MediumBehavioral
20 practiced
Outline a 90-day plan you'd execute as a new software engineer when public infra documentation indicates frequent latency spikes in a subsystem. List weekly milestones, stakeholders to engage (SRE, product, data), experiments or dashboards to create, and deliverables that demonstrably reduce latency or MTTR. Be concrete about metrics you'll track to show impact.
Sample Answer
Situation: Joining as a new software engineer, public infra docs show a subsystem has frequent latency spikes that hurt user experience and SLOs. My 90-day plan focuses on rapid diagnosis, stakeholder alignment, measurable experiments, and delivering changes that reduce latency and MTTR.Weeks 0–2 (Onboard & Discover)- Milestones: Read infra docs, run local repro, get access to dashboards/logs/tracing, join on-call rotation for context.- Stakeholders: SRE (alerts/SLOs), Product (user impact/priorities), Data/BI (event/metric schema).- Deliverables: Runbook draft, baseline metrics snapshot (p50/p95/p99 latency, error rate, request rate, MTTR, SLO breach minutes).Weeks 3–4 (Hypothesis & Observability)- Milestones: Map request flow, add/confirm distributed tracing spans, identify top hot paths.- Experiments/Dashboards: Create a latency breakdown dashboard (by endpoint, host, region, code path) and tracing heatmap; add synthetic canaries at 1-min cadence.- Deliverables: Root-cause hypothesis document; dashboard URL; alert tuning recommendations.- Metrics to track: p95/p99 latency, anomaly frequency, SLO violation minutes.Weeks 5–8 (Targeted Experiments)- Milestones: Run controlled experiments (feature flags/canaries) to test optimizations.- Experiments: - Query indexing / DB connection pool tuning - Add caching layer (edge or in-process) for hot keys - Apply backpressure/rate limiting and circuit breakers on slow downstreams - Optimize serialization or batch requests- Deliverables: PRs with instrumentation, A/B canary runs, experiment reports.- Metrics: Relative drop in p95/p99, CPU/memory per host, request success rate, change in MTTR.Weeks 9–12 (Rollout & Institutionalize)- Milestones: Roll out validated fixes, finalize runbooks, handoff to SRE, retro.- Deliverables: Production rollout plan, automated performance tests added to CI, updated runbook and run-deck for on-call.- Stakeholder engagement: Postmortem with Product/SRE/Data, prioritize remaining work with product.- Final metrics to show impact: % reduction in p95 and p99 latency (target: 30–50% p99 reduction or meet SLO), reduction in incident count and MTTR (target: 40% MTTR improvement), improved SLO compliance (minutes of breach reduced to 0).Why this works: early measurement creates a baseline; short hypothesis-driven experiments limit blast radius; heavy SRE/Product alignment ensures fixes target user pain; final deliverables ensure improvements are automated, observable, and owned.
HardTechnical
43 practiced
Propose an approach to reconstruct the company's service dependency graph when you join with limited internal access. Use publicly available telemetry, passive tracing samples, DNS/service registry observations, and lightweight runtime probes. Describe heuristics for inferring caller-callee relationships, handling fan-in/fan-out, and validating the generated graph with minimal disruption.
Sample Answer
Requirements & constraints:- Limited internal access, non-disruptive methods only.- Use only public telemetry, passive traces, DNS/registry observations, and lightweight probes.High-level approach:1. Data collection (passive-first) - Ingest sampled tracing headers seen at ingress/egress (W3C TraceContext, B3) from public endpoints, CDN logs, load balancers, and API gateways. - Capture DNS queries and service registry announcements (SRV records, Consul, Kubernetes headless svc events) visible from your network perspective. - Observe network-level metadata: TLS SNI, IP:port, reverse DNS, HTTP Host, User-Agent, Referrer, and request paths from access logs. - Run lightweight runtime probes: non-invasive HTTP HEAD to candidate endpoints, /health or version endpoints, and short-lived client-side instrumentation that propagates a synthetic trace id (no write effect).2. Infer caller→callee heuristics - Header correlation: if requests contain propagated trace IDs or unique request IDs passing through multiple public facets, link endpoints that share the same ID sequence. - Temporal co-occurrence: correlate timestamps — upstream request A followed within expected service latency by request B from A’s IP or same client trace header implies A→B. - Path/URL mapping: consistent downstream path patterns (e.g., /v1/payments → /v1/ledger) across many traces indicate dependency. - TLS SNI + hostname: SNI resolves to service; map clients that open TLS sessions to those SNI hostnames. - DNS resolution chaining: if client resolves service X then connects to IPs associated with service Y repeatedly for certain flows, infer X→Y. - Registry lookups: service entries listing upstreams or tags provide direct edges.3. Handling fan-in / fan-out - Fan-in detection: many distinct callers whose traces converge to the same downstream endpoint → mark downstream as high fan-in; attach count and normalized traffic share. - Fan-out detection: single caller making simultaneous distinct downstream calls within the same trace context → mark as fan-out; quantify parallelism and percent of traces exhibiting fan-out. - Use probabilistic weighting: compute edge weight = fraction of observed traces that include the edge × average throughput. Keep confidence score based on number of samples and variety of sources.4. Validation & iterative hardening (minimal disruption) - Start with read-only validation: lightweight HEAD/GET probes and health/version endpoints to confirm endpoint existence and canonical name. - Shadow-trace injection: inject synthetic trace IDs into probe requests and validate they appear at supposed downstream logs (without changing behavior). - Canary traffic: with ops consent, send tiny percentage of real traffic as shadow to validate path end-to-end; must be rate-limited and non-mutating. - Cross-validate with infra: compare inferred graph against service registry, deployment manifests, configmaps, and owner lists where accessible. - Feedback loop: present low-confidence edges to service owners for verification (email/slack with example traces); incorporate corrections.5. Representation & risk management - Produce directed graph with edges annotated: weight, confidence, evidence types (trace header, DNS, probe), last-seen timestamp. - Expose uncertainty: do not auto-enforce routing changes; use the graph for discovery, onboarding, and security review. - Privacy and safety: avoid probes that exercise stateful or write paths, throttle probes, and respect robots/usage policies.Why this works:- Passive signals minimize disruption and capture real traffic patterns.- Heuristics combine orthogonal evidence (headers, DNS, timing) to reduce false positives.- Iterative validation with owners and small probes raises confidence without risk.Example quick heuristic: if 100 distinct sampled traces show client A calls host-a.example.com and within 30–200ms calls host-b.example.com with the same traceparent header in 82% of those traces, create edge A→B with weight 0.82 and high confidence.
HardSystem Design
26 practiced
Design a real-time analytics pipeline that ingests 100k events/second, supports <1s tail-to-tail latency for critical dashboards, and allows replay for late-arriving events. Outline ingestion, buffering, stream processing framework (eg: Kafka + Flink), storage for OLAP, schema management, backpressure handling, and cost trade-offs.
Sample Answer
Requirements clarification:- Ingest 100k events/sec, critical dashboards require <1s end-to-end latency (tail), and support replay of late/duplicate events. Durability, correctness (exactly-once or at-least-once with dedup), and cost-aware scaling matter.High-level architecture:Producers → Kafka (ingest/buffer) → Flink (stream processing, state + windows) → OLAP store (hot: ClickHouse/ClickHouse Cloud, Druid, or BigQuery streaming/Materialized Views) → Dashboards/BIKafka also retains raw events (cold storage) for replay.Ingestion & buffering (Kafka):- Put a Kafka cluster (or managed MSK/Confluent) with enough partitions: throughput target ~100k/s; pick partition count to support parallelism (e.g., 200–500 partitions depending on throughput per partition).- Producers: batch small (linger.ms=5-10ms), high throughput sockets, asynchronous acks=all, idempotent producers enabled to avoid duplicates.- Topic layout: raw-events (compact=false, retention configurable), raw-events-compact (compacted for keys), DLQ for malformed messages.- Retention: keep raw-events for replay window (days-weeks) + longer-term cold archive in S3 via Kafka Connect.Schema management:- Use Avro/Protobuf/JSON Schema + Schema Registry (Confluent or open-source) to enforce compatibility (BACKWARD/FORWARD) and prevent consumer breaks.- Producers validate at build-time and runtime; schema evolution policy documented.Stream processing (Flink):- Use Apache Flink in production, configured for low-latency, exactly-once processing with checkpointing and RocksDB state backend.- Parallelism matches Kafka partitions. Use event-time processing, watermarks, and allowed-lateness to handle late arrivals.- Implement: - Deduplication (event-id + TTL state) for at-least-once sources. - Windowing for dashboard aggregates (sliding/tumbling), producing both incremental updates (for <1s tail) and final windows when lateness bound passed. - Side outputs for late events (write back to raw topic or replay topic).- Tuning for <1s tail latency: - Minimize batching in producers and sinks; Flink: set low checkpointing interval (e.g., 1–2s) but balance overhead, adjust RocksDB flush thresholds. - Use asynchronous sinks and upsert semantics to OLAP to avoid blocking. - Monitor end-to-end P99 latency and tune watermarks to avoid waiting too long for late data.State & fault-tolerance:- RocksDB keyed state with incremental checkpoints to durable storage (S3/GCS) and high-frequency small checkpoints to bound recovery time.- Exactly-once sinks where supported (Kafka transactions, JDBC upserts where possible). For sinks that don't support exactly-once, use idempotent upserts (primary key + versioning) or write-ahead logs.Storage for OLAP:- Hot store for dashboards: ClickHouse/Druid for sub-second queries, or BigQuery with streaming and BI-engine for managed service (consider query latency).- Store both pre-aggregated metrics (materialized views) and raw partitions for ad-hoc analysis.- Partitioning and TTL policy: keep hot recent partitions (hours/days); offload older partitions to cheaper object store or cold OLAP.Replay for late-arriving events:- Use Kafka retention + unique event IDs. To replay, re-publish events to the raw-events topic or a replay topic with a replay header; Flink job can detect replay flag and reprocess (idempotent writes).- Alternative: use compacted topic for latest state per key and recompute aggregates from compacted snapshot.Backpressure & flow control:- At producer side: client-side rate limiting, circuit breaker; if Kafka is overloaded return an error and degrade gracefully (sampled ingestion) for non-critical events.- At Kafka: monitor partition lag, scale brokers/IO, increase partitions.- At Flink: autoscale TaskManagers, set backpressure detection, tune buffer timeout (taskmanager.network.memory.fraction, network buffers, bufferTimeout) to lower latency.- Sinking: use async, batch upserts, and backpressure-aware sinks. If sink becomes the bottleneck, write intermediate aggregates to Kafka (fast) and have separate workers flush to OLAP.Operational concerns:- Monitoring: end-to-end latency (producer→dashboard), Kafka lag, Flink checkpoint duration, task failure rates, JVM metrics.- SLOs: set P50/P95/P99 targets; use canaries and synthetic tests.- Security: TLS, IAM, encryption-at-rest.Cost trade-offs:- Partitioning & broker count vs throughput: more partitions increase parallelism but increase cluster metadata overhead.- Managed services (Confluent/MSK, Flink as a service, BigQuery) reduce ops but higher $/QPS vs self-managed.- Checkpoint frequency: more frequent → lower recovery time / higher cost (storage & CPU).- Hot OLAP retention: keeping more data hot increases storage & compute costs; use tiering (hot recent data in ClickHouse, older in BigQuery or S3).- Exactly-once semantics incur overhead (transactions, checkpoints) — choose at-least-once + idempotent sinks if cost-sensitive.Example operational flow:1. Producer sends Avro event to Kafka partition keyed by entity-id (ensures locality).2. Flink reads, uses event-time, updates RocksDB state, emits incremental metric to Kafka "metrics" topic and writes to OLAP asynchronously.3. Dashboard subscribes to metrics store; critical aggregates are available sub-second; final aggregates reconciled after lateness window.This design balances <1s latency by minimizing batching and doing incremental updates, ensures replay via retained raw events and idempotent processing, and keeps cost/time trade-offs explicit (managed vs self-hosted, checkpoint cadence, hot vs cold storage).
HardSystem Design
25 practiced
The blog references an internal service mesh adoption (sidecars and mTLS). Critique this choice: list expected performance impacts, operational costs (control plane HA, upgrades), security benefits, and the telemetry benefits. Outline a migration plan from a no-mesh environment to a mesh for around half the services, including canary steps and rollback criteria.
Sample Answer
Clarifying assumption: target mesh = sidecar-based (e.g., Istio) with mTLS enabled by default.Tradeoff critique- Performance impacts - CPU/memory: per-pod sidecar adds ~50–150MB RAM and consumes CPU for proxying, mTLS crypto, and telemetry batching. - Latency: additional hop + TLS handshake adds 1–5ms p99 tail latency; higher under bursty small RPCs. - Network: increased egress/inbound overhead (TLS record framing); bandwidth slightly higher due to headers/metadata. - Mitigation: enable connection reuse, TLS session resumption, tune proxy worker threads, use BPF/DPUs where possible.- Operational costs - Control plane HA: need multi-replica, geographically distributed control plane (pilot/istiod, CA, telemetry) → HA, backups, monitoring. - Upgrades: coordinated rolling upgrades of control plane and sidecar injector; can cause transient policy mismatches—requires canary control-plane upgrades. - Increased CI/CD complexity: sidecar injection, image updates, automated tests for e2e proxy behavior. - Skills and runbook: staff training, incident-runbooks, capacity planning.- Security benefits - Mutual TLS: automatic service identity and encrypted traffic by default, reduces credential sprawl. - Identity & policy: fine-grained L7 authorization, mTLS + JWT + RBAC reduce lateral movement. - Caveat: mesh increases attack surface (control plane components), so harden and rotate mesh certs and secure kube-api access.- Telemetry benefits - Automatic distributed traces, metrics, and enriched logs (request/response metadata, latencies, retries). - Faster root-cause for cross-service issues, consistent labeling for per-service dashboards. - Cost: telemetry volume can balloon—need sampling, aggregation, and ingestion controls.Migration plan (no-mesh → mesh for ~50% services)1. Prep (weeks) - Pilot cluster/environment; install control plane with HA staging replicas. - Baseline telemetry & SLIs for services chosen for mesh. - Create a subset (non-critical) of services ~5–10 to onboard first.2. Canary sidecar injection (automated) - Step A: Manual sidecar injection for one service pair (client+server) with mTLS disabled, observe traffic/latency for 48–72h. - Metrics: CPU/RAM, p50/p99 latency, error rate, request throughput. - Rollback if >10% CPU increase unplanned, p99 latency regresses >30%, or error rate increases >1% absolute. - Step B: Enable mTLS in permissive mode (both mtls and plain), run 72–168h. Validate connection stability, CA cert issuance and rotation. - Step C: Switch to strict mTLS for the pair; validate end-to-end.3. Expand incrementally (weeks) - Onboard 2–3 more services per day, repeating A→C and monitoring SLIs. - Run load tests simulating production traffic patterns before moving critical paths.4. Control-plane canary - Deploy control-plane minor version into staging mesh, then to 1% of namespaces in prod; monitor control-plane telemetry and sidecar compatibility. - Rollback control plane if control-plane error rates or sidecar disconnects exceed thresholds (e.g., >0.5% service mesh RPC errors or >5% increase in 5xxs).5. Observatory & cost controls - Implement sampling (traces), metrics aggregation, and retention policies; measure cost impact.6. Full rollout & hardening - Once 50% validated, automate injection via CI, enforce policy-as-code, finalize runbooks.Rollback criteria (per canary)- Immediate rollback if: - End-user errors (5xx) increase >2x baseline for 5m window. - p99 latency increase >50% sustained for 10m. - Sidecar OOMs/CrashLoopBackOff >1% of instances. - Control-plane becomes unreachable or >30% sidecars lose config sync.- Graceful rollback: disable sidecar injection, scale down proxies, redeploy pods without sidecars; revoke any partial strict mTLS policies.Final notes- Start with non-critical, chatty services to observe worst-case telemetry cost.- Automate health gates in CI/CD and add chaos tests (proxy failures, certificate rotation) before wider rollout.
MediumTechnical
28 practiced
The company reports a monthly cloud bill of ~$1.2M with highly spiky traffic and large egress charges for media. As an infra engineer, propose a prioritized roadmap of quick wins and long-term projects to reduce spend while maintaining availability. For each item include estimated effort, expected savings range, and risk level (low/medium/high).
Sample Answer
High-level approach: prioritize low-effort/high-impact items first (quick wins) to cut egress and waste, then mid/long-term platform changes to reshape traffic patterns and costs while preserving availability.Quick wins (0–3 months)1) Configure CDN for media (cache rules, long TTLs, origin shielding)- Effort: 2–4 weeks- Savings: 15–35% of egress- Risk: Low- Why: Offloads origin, reduces egress; implement route-based TTLs and cache-control headers.2) Enforce transfer acceleration & multipart upload for large objects; enable regional replication instead of cross-region reads- Effort: 2–3 weeks- Savings: 5–15%- Risk: Low–Medium- Why: Optimizes network paths and reduces cross-region charges.3) Rightsize and shutdown idle compute (autoscaling, spot instances, pod disruption budgets)- Effort: 2–4 weeks- Savings: 5–20%- Risk: Low–Medium- Why: Eliminate waste during spikes and troughs.Middle-term (3–6 months)4) Implement request-routing to edge (Lambda@Edge / Cloud Functions) for lightweight transforms- Effort: 6–10 weeks- Savings: 5–12% egress + compute shift- Risk: Medium- Why: Reduces origin hits and allows per-request caching.5) Bulk-transfer negotiation and committed use discounts with provider- Effort: 2–6 weeks (procurement)- Savings: 10–25% on compute/storage/network- Risk: Low- Why: Leverage predictable baseline.Long-term (6–18 months)6) Re-architect media pipeline: client-side adaptive bitrate + peer-assisted delivery (P2P/CDN hybrid)- Effort: 3–9 months- Savings: 20–50% egress (depends on adoption)- Risk: Medium–High- Why: Reduces centralized egress by serving from peers/edge.7) Multi-region data placement and cache-aware storage tiering- Effort: 3–6 months- Savings: 10–30%- Risk: Medium- Why: Place hot media near users; cold objects in cheaper tiers.8) Move stateful workloads to more cost-efficient runtimes (FaaS / managed services) and optimize Kubernetes cluster autoscaler- Effort: 4–9 months- Savings: 10–30% compute- Risk: MediumExecution notes:- Measure baseline, run experiments with canary domains, track egress by origin/CDN; combine items incrementally.- Prioritize CDN + rightsizing + discounts first for fastest ROI while planning architecture changes for sustained reduction.
Unlock Full Question Bank
Get access to hundreds of Understanding the Company's Infrastructure Context interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.