Infrastructure Implementation and Operations Questions

Hands on design, deployment, and operational management of infrastructure components and services. This includes setting up and configuring load balancers, database replication and high availability, caching layers, networking and network security, service discovery and routing, container deployment and orchestration, monitoring and observability, logging and alerting, backup and disaster recovery strategies, and secrets management in runtime. Candidates should be able to walk through concrete implementations, explain trade offs, demonstrate troubleshooting and performance tuning, and show how infrastructure components integrate to meet availability, scalability, and security requirements.

EasyTechnical

72 practiced

Describe when to introduce a caching layer for a web application and explain common cache invalidation strategies: TTL, cache-aside, write-through, and write-back. For each approach, give a short example and describe trade-offs between freshness and performance.

EasyTechnical

47 practiced

Explain the differences between Layer 4 and Layer 7 load balancers, hardware vs software implementations, and cloud-managed vs self-hosted options. For each type describe typical use cases, where TLS termination should occur, session affinity options, and trade-offs for latency, flexibility, and observability.

Sample Answer

**Overview — L4 vs L7**- Layer 4 (Transport): routes by IP/port (TCP/UDP). Fast, low latency, protocol-agnostic.- Layer 7 (Application): inspects HTTP(S)/gRPC headers/body, does routing by URL, cookies, host, rewrites, WAF, A/B testing.

**Typical use cases**- L4: TCP proxying, database proxies, non-HTTP protocols, high-throughput internal services.- L7: Web apps, API gateways, feature flags, path-based routing, auth enforcement.

**TLS termination**- Prefer L7 termination when you need header inspection (HTTP routing, host-based rules) or WAF features.- Prefer L4 passthrough when end-to-end encryption is required (mutual TLS) or when you cannot inspect ciphertext.- Hybrid: TLS terminated at LB for routing, then re-encrypt to backend for security-in-depth.

**Session affinity**- L4: affinity by source IP or 5-tuple — simple but brittle with NAT/load balancer clients.- L7: cookie-based affinity (HTTP cookie, insert/route-cookie) or header tokens — more reliable for web sessions.

**Hardware vs Software**- Hardware (appliances): dedicated ASICs/FPGAs, predictable performance, lower latency at high throughput, higher CAPEX, less flexible for rapid feature changes.- Software: runs on commodity servers/KVM/containers, highly flexible, easier to automate, scalable horizontally, slightly higher CPU latency per connection.

**Cloud-managed vs Self-hosted**- Cloud-managed: AWS ELB/ALB, GCP LB, Azure LB — easy operationally, integrated with autoscaling and monitoring, limited customization and possible vendor lock-in.- Self-hosted: HAProxy, NGINX, Envoy, MetalLB — full control, advanced features, customizable TLS/private certs, requires ops effort for HA/scale/observability.

**Trade-offs**- Latency: L4 + hardware lowest; software L7 adds CPU and parsing overhead.- Flexibility: L7 software/self-hosted > cloud-managed L7 > hardware.- Observability: L7 gives richer metrics/tracing (headers, paths); cloud-managed often provides built-in metrics but less deep access to raw traffic; self-hosted + sidecars can provide full traces/logs.

As a systems engineer I choose based on protocol, security posture (end-to-end crypto), operational capacity, and need for advanced routing/observability.

MediumTechnical

34 practiced

Design RBAC policies, network policies, and admission controls for a shared Kubernetes cluster used by multiple teams. Explain namespace design, service accounts, role bindings, least privilege, pod security contexts, and how to onboard a new team safely without compromising other workloads.

Sample Answer

**Approach summary**Design per-team isolation using namespaces, scoped service accounts, least-privilege RBAC, default-deny NetworkPolicies, PodSecurityContext constraints, and admission controllers (OPA/Gatekeeper + PodSecurity) to enforce policies. Onboard new teams via a gated process.

**Namespace & onboarding**- Create one namespace per team (team-a-prod, team-a-dev) plus shared infra namespaces.- Onboard flow: request ticket → create namespace with standardized labels/annotations → apply namespace ResourceQuota, LimitRange, NetworkPolicy, and baseline RBAC templates → grant access after review.

**RBAC & service accounts**- Use ClusterRole for cross-namespace infra ops; otherwise Role per namespace.- Create granular Roles and bind to team-specific ServiceAccounts or AD groups.- Example Role allowing only secrets read/write for CI:

yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: namespace: team-a-dev name: ci-secrets
rules:
- apiGroups: [""] resources: ["secrets"] verbs: ["get","create","update"]

- Prefer RoleBindings to groups/serviceaccounts, avoid ClusterAdmin.

**Network policies**- Default-deny ingress/egress per namespace.- Allow only required traffic: e.g., frontend -> backend namespace on specific ports.- Shared services exposed via authorised namespaces or an API gateway.

**Pod security & admission**- Enforce PodSecurity (restricted/baseline) and OPA Gatekeeper policies for: - disallow privileged, hostPath, hostNetwork - enforce seccomp, runAsNonRoot, fsGroup- Use validating/admission webhooks to block images from unapproved registries and enforce resource requests/limits.

**Least privilege & audits**- Grant minimal verbs, rotate service account tokens, use short-lived KSA tokens via Workload Identity where possible.- Enable audit logging and periodic IAM/RBAC reviews; use kube-bench/kube-hunter and OPA reports.

**Operational notes**- Automate namespace bootstrapping via GitOps (ArgoCD) and templates.- Provide a sandbox namespace for teams to test before production access.

MediumSystem Design

34 practiced

Design an observability strategy for a microservices architecture that supports root cause analysis for latency and error spikes. Specify what metrics, traces, and logs you would collect, sampling rates, correlation keys, SLI/SLO definitions, dashboarding, and how to instrument services minimally but effectively.

Sample Answer

**Situation & goals**Design an observability strategy for microservices to enable fast root-cause analysis (RCA) of latency and error spikes with minimal instrumentation overhead.

**Metrics to collect**- Service-level: request rate (rps), p50/p95/p99 latency, error rate (% 4xx/5xx), CPU, memory, GC pause.- Endpoint-level: latency histogram, success/error counts.- Infrastructure: host/container CPU, mem, disk, network, queue lengths, DB connection pool usage.- External dependencies: downstream latency & error rates.

**Traces**- Distributed traces (OpenTelemetry): capture full request span tree for sampled requests, include span attributes for HTTP route, method, status, DB/queue calls, payload size.- Sampling: 1% baseline + dynamic tail-based sampling (retain all traces where latency > p95 or error observed, and 100% for synthetic transactions).- Trace retention: high-res for 7–14d, low-res meta for 90d.

**Logs**- Structured logs (JSON) at INFO; error/exception stack traces at ERROR.- Include correlation keys: trace_id, span_id, request_id, user_id, service_version.- Log levels: avoid noisy DEBUG in prod; use dynamic log-level toggles per service.

**Correlation keys**- Single canonical trace_id propagated across services via headers (e.g., W3C traceparent) and request_id for idempotency.- Instrument middleware to inject and log these keys.

**SLI/SLO**- Latency SLI: p95 request latency per service and per critical endpoint.- Error SLI: % of failed requests (5xx) per service.- SLOs: latency p95 < 300ms (90-day) and error rate < 0.5% per week (adjust per SLA).- Alerting: burn-rate alerts (short-term and long-term windows), SLO error budget approaches.

**Dashboarding & RCA workflows**- High-level SLO dashboard: global uptime, error budget, latency heatmap.- Service dashboards: top endpoints, p50/p95/p99 latency, error rate, RPS, dependency latency.- Tracing view: waterfall per request, service map showing error/latency hotspots.- Pager runbook links from alerts with query templates (logs + traces + metrics) for RCA.

**Minimal effective instrumentation**- Use standardized middleware/SDKs to auto-instrument HTTP, gRPC, DB, and messaging clients.- Instrument only critical endpoints and key dependency calls with custom spans/metrics.- Add semantic attributes (business IDs) sparingly for privacy.- Feature flags to enable deeper tracing/logging only when investigating.

**Operational practices**- Tail-based sampling to focus on problematic traces.- Regular SLO reviews, simulated synthetic transactions, and runbook drills.- Maintain observability as code (dashboards, alerts in repo) for reproducibility.

This balances actionable telemetry for rapid RCA while minimizing overhead and noise.

HardTechnical

34 practiced

A relational database under a write-heavy workload shows rising write latency and increased lock contention. Walk through diagnostic steps using monitoring and DB internals, identify possible root causes (index contention, long transactions, WAL saturation, IO bottlenecks), propose immediate mitigations, and long-term architecture changes to scale writes.

Sample Answer

**Situation & first checks (diagnostic steps)**

- Check metrics: write latency, transactions/sec, lock wait time, queue depth, disk IOPS/latency, CPU, memory, network. Use Prometheus/Datadog, iostat, perf, vmstat.- DB internals: inspect active queries, pg_stat_activity / SHOW engine status, lock table (pg_locks / INFORMATION_SCHEMA), long-running transactions, replication lag, checkpoint/WAL metrics (pg_stat_bgwriter, pg_stat_wal).- Examine index stats and bloat (pg_stat_user_indexes, pgstattuple), page split rates, hot rows (high contention on same PK).- Storage: monitor WAL/redo throughput, fsync times, disk latency, queue depth (iostat, blktrace), filesystem mount options.- Reproduce under controlled load and capture traces (EXPLAIN ANALYZE, perf, strace if needed).

**Possible root causes (and signs)**

- Index contention / hot spots: many updates to same index pages → high lock waits, frequent page splits, increased WAL.- Long transactions: prevent vacuum/cleanup → bloat, locking, higher I/O.- WAL saturation/checkpoint pressure: high fsync/checkpoint time causing stalls.- IO bottlenecks: disk latency or saturated RAID causing high write latency.- Lock escalation / application-level conflicts: many small conflicting transactions.

**Immediate mitigations**

- Identify and kill or fix runaway long transactions; reduce tx scope.- Batch writes and use bulk inserts/UPSERT to reduce transaction count.- Increase checkpoint_segments / tune checkpoint_timeout and wal_writer settings to smooth WAL throughput (careful with recovery time).- Move WAL to faster media (NVMe), enable group commit, adjust fsync/commit settings only with understanding of durability trade-offs.- Add application-side backoff, use optimistic concurrency where possible.- Create targeted indexes or remove unused indexes to reduce write amplification.- Tune autovacuum thresholds to keep bloat under control.

**Long-term architecture changes to scale writes**

- Shard/write-partition data by key to reduce contention (application or DB-level sharding).- Use write-optimized stores for high-ingest tables (e.g., Kafka + stream processing into OLTP).- CQRS: separate write path (append-only, fast) from read models; materialize views asynchronously.- Horizontal scaling: Citus / Vitess / sharded clusters or distributed SQL for parallel writes.- Use partitioning (range/hash) to limit index size and contention; periodic reindexing and compaction.- Invest in faster storage (NVMe, write-optimized caching), or dedicated WAL device.- Improve schema: avoid hot-update columns, use surrogate PKs that spread inserts, consider monotonic key drawbacks.

**How I’d act as Systems Engineer**

- Run the diagnostic checklist, implement immediate mitigations during a maintenance window or controlled rollout, measure impact, then plan shard/partitioning or CQRS migration with staging, schema changes, and runbooks to minimize risk.

Unlock Full Question Bank

Get access to hundreds of Infrastructure Implementation and Operations interview questions and detailed answers.

Join thousands of developers preparing for their dream job.