System Architecture Communication and Documentation Questions

Assess the candidate ability to describe, document, and communicate system architecture both visually and verbally. Candidates should present what a system does and who uses it, identify major components and how they interact, show data flow and integration points, and explain critical architectural decisions and trade offs. Interviewers expect clear diagrams using standard conventions that show high level views, component interactions, and deployment topology, accompanied by concise narrative documentation. Strong answers include multiple views tailored to the audience, labeled diagrams, and justification of design choices while avoiding unnecessary implementation detail. Candidates should be able to discuss scaling strategies, reliability and operational considerations including failure modes, migration paths, observability, and deployment considerations. The scope includes common architectural building blocks such as microservices, application programming interfaces, databases, caching layers, and message buses, as well as consistency and availability implications and service to service communication patterns, and the connection between technical choices and business context.

HardSystem Design

56 practiced

Design a diagram and documentation approach for a prediction cache that must be invalidated whenever model weights are updated or when feature transforms change. Describe the invalidation workflow, the contracts between model deployment and cache purge, and how you'd show this on an architecture diagram and in deployment runbooks.

Sample Answer

Requirements and constraints:- Functional: prediction cache must be invalidated on (a) model weights change (new model version) and (b) any change in feature transforms (code, parameters, featurization pipeline).- Non-functional: low-latency lookups, strong consistency for stale predictions avoidance (no serving outdated predictions after rollout), support rolling deployments, resilient to partial failures, auditable.

High-level architecture (ASCII):Client -> API Gateway -> Prediction Service -> Cache (Redis/Memory + version key) ↕ Feature Store ↕ Model Registry ↕ CI/CD Deployment Pipeline

Core components & responsibilities:- Model Registry: stores model version metadata, checksum, artifact URI, transform hash, and semantic version. Exposes model_version id and transform_hash.- Prediction Service: computes features, applies transforms, queries cache, and serves model. Uses a cache key schema that includes model_version and transform_hash.- Cache (Redis/Cluster): keyed by: cache_key = H(feature_vector)+":" + model_version + ":" + transform_hash- Cache Purge Service (or hooks): listens for registry events (webhook/message bus) and issues purge operations (DEL or keyspace notifications) or sets a new active version.- Deployment Pipeline: when deploying a new model artifact or transform change, it publishes an event to Model Registry and to Message Bus (e.g., Kafka). Includes pre-deploy checks and post-deploy cache invalidation steps.

Invalidation workflow (strong-consistency, orchestrated):1. Prepare new model & transforms in staging. Compute artifact checksum and transform_hash.2. CI/CD creates new model_version in Model Registry (state = staging).3. Run integration and canary tests. If pass, pipeline sets model_version -> activating.4. Before switching traffic, pipeline publishes "invalidate: model_version=X transform_hash=Y" to message bus.5. Cache Purge Service receives event and: - Option A (immediate purge): atomically delete keys matching old_version or set active_version to X and allow cache TTLs for old keys to expire. For Redis clusters, use SCAN + DEL in batches with rate limiting; or maintain versioned namespace and flip active pointer (fast). - Option B (pointer flip, recommended for zero-downtime): write new active pointer in Model Registry; Prediction Service checks pointer—if request uses stale combo, it treats as cache miss. Purge service can then lazily Evict or expire old namespace.6. Deployment Pipeline flips live traffic to new model_version. Prediction Service begins populating cache under new namespace.7. Post-deploy: Purge Service performs background sweep to remove old keys and emits audit event.

Contracts between deployment and cache purge:- Model Registry API: - POST /model_versions {artifact_uri, transform_hash, semantic_version} - PATCH /model_versions/{id} {state: activating|active|rollback} - GET /active -> returns {model_version, transform_hash, active_namespace}- Invalidation event schema (message bus): - {event_type: "invalidate", model_version, transform_hash, timestamp, deploy_id, purge_strategy}- Service-level contract: Prediction Service must include registry model_version and transform_hash in cache key and must treat any mismatch between request-time pointer and cached key as miss.- SLA: purge API must acknowledge within X seconds/minutes; deployment must wait for acknowledgment or use pointer-flip fallback to guarantee no stale serving.

Diagram annotations to show on architecture diagram:- Show namespaces for cache keys and active pointer in Model Registry.- Show synchronous (pointer flip) vs asynchronous (batch delete) paths.- Mark critical steps: publish invalidate event, ack purge, flip traffic.

Deployment runbook entries (playbook style):- Pre-deploy: - Validate model artifact checksum; compute transform_hash. - Create model_version in registry and run canary tests.- Deploy (automated steps, with manual guard if required): - Trigger invalidate event with purge_strategy=pointer-flip. - Wait for PURGE_ACK (timeout 2 min). If no ack, abort or continue with pointer-flip fallback. - Flip active model pointer in registry. - Shift X% traffic (canary) -> monitor metrics (latency, error-rate, prediction-drift). - Gradually increase to 100% if healthy.- Post-deploy: - Trigger background cache sweep for old namespaces; monitor purge progress. - Verify no stale predictions (sample audit queries comparing new model outputs). - Log deploy_id and purge events for audit.- Rollback: - Flip pointer back to previous model_version. - Trigger re-populate cache by running bulk warm-up jobs if necessary.- Emergency invalidation (feature bug discovered): - Send immediate invalidate event with purge_strategy=immediate-delete and set registry active pointer to safe model_version; notify on-call and run remediation steps.

Trade-offs:- Immediate delete ensures no stale keys but can be slow for large caches; pointer-flip is fast and safe but leaves old keys until GC (higher memory).- SCAN+DEL vs Redis keyspace separation: use namespaces (version prefix) to enable constant-time pointer flip.- Auditing and idempotency are critical: use deploy_id in events.

Key metrics to monitor:- purge latency, cache hit rate pre/post-deploy, memory usage, failed purge ops, model-serving mismatch counts (requests served with mismatched model_version).

This approach provides deterministic invalidation, clear contracts between deployment and cache components, and runbook procedures to operate safely during model rollouts or emergency fixes.

MediumTechnical

51 practiced

You need to communicate a capacity plan for an ML inference service. Which diagrams and tables would you include in documentation to convey current utilization, expected growth, headroom, and autoscaling policies? Provide a concise list of metrics to include and where to show them on a diagram or appendix.

Sample Answer

Start with a short executive summary (1 paragraph) that states current capacity, risk level (OK/warning/critical), and recommended actions.

Diagrams to include- High-level architecture diagram: show components (clients, LB, inference pods/VMs, GPU/CPU pools, cache, queue). Overlay current average and peak RPS and P50/P95 latency next to the inference service.- Capacity dashboard (time-series): multi-line graph of RPS, concurrent requests, CPU%, GPU util%, memory%, and latency (P50/P95/P99) over last 30/90 days.- Heatmap/calendar: hourly/daily request heatmap to show peak windows and seasonality.- Autoscaling policy flowchart: triggers (metrics + thresholds), scale steps, cooldowns, max/min instances, and priority between horizontal vs vertical scaling.- Headroom & risk diagram: stacked bar per instance type showing used capacity, reserved headroom, and available headroom.

Tables/appendices to include- Current utilization summary table: instance type, count, vCPU/GPU, memory, avg CPU%, avg GPU%, avg memory%, avg latency, P95 latency, error rate, current cost/hr.- Expected growth projection: baseline RPS, monthly growth %, projected RPS at 1/3/6/12 months, required instances and cost projection.- Headroom analysis table: peak RPS supported, safety buffer (%), time-to-scale (cold start), capacity shortfall scenarios.- Autoscaling parameter table: metric, threshold, scale-up step, scale-down step, cooldown, max/min.- Runbook appendix: manual override steps, rollback plan, contact list.

Concise metrics to collect and where to show them- Request metrics: RPS, concurrent requests — show on architecture diagram (current) and time-series dashboard.- Latency: P50/P95/P99 — show on time-series dashboard and per-instance table.- Resource utilization: CPU%, GPU util%, memory% — show on capacity dashboard and instance table.- Queue/backlog depth & time-to-first-byte — show on time-series and heatmap for peaks.- Error rate and 5xx/4xx breakdown — show on dashboard and current utilization table.- Autoscaler signals: metric used (e.g., GPU util), threshold, cooldowns — show on autoscaling flowchart and autoscaling table.- Cost metrics: cost/hr and cost per million inferences — include in utilization and projection tables.- Scaling latency: cold start time and time-to-add-instance — include in headroom table.

Keep diagrams annotated with units and time windows (e.g., last 7/30/90 days). Include assumptions (SLA, safety buffer %) and key risk scenarios (sustained +50% burst, single AZ failure).

HardTechnical

47 practiced

An exec asks: 'Why can't we just deploy models as Docker containers behind a load balancer and call it a day?' Provide a structured one-page response that covers architectural, operational, cost, and scalability considerations, and list the diagrams you would attach to support your argument.

Sample Answer

Short answer: because serving ML models reliably and cost-effectively requires more than container + load balancer — you need lifecycle management, specialized inference infrastructure, observability, scaling strategies, and governance. Below I summarize the key architectural, operational, cost, and scalability reasons and the mitigations I’d propose.

Architectural concerns- Model lifecycle: training → validation → versioning → rollout/rollback needs CI/CD and metadata (model registry). Plain containers don’t manage versions or lineage.- Hardware & runtime: many models need GPUs/TPUs, mixed-precision, or runtime optimizations (TensorRT, ONNX). Scheduling GPUs on generic container hosts and ensuring deterministic inference demands orchestration/service mesh + model servers (Triton, TorchServe).- State and features: feature stores, warm caches, pre/post-processing, and session affinity for stateful ops aren’t solved by a load balancer alone.

Operational risks- Observability: need request tracing, latency/p99, throughput, per-model accuracy/label-collection, data-drift alerts and automated retraining triggers.- Reliability & safety: canary/circuit-breakers, rate limiting, throttling, backpressure. Containers behind an LB lack traffic-splitting, health-promoted rollouts, or model explainability hooks.- Security & compliance: model access controls, encrypted model artifacts, auditing — operational controls beyond container networking.

Cost considerations- Idle resource waste: keeping GPU-backed containers always-on is expensive. Need autoscaling, batching, and queueing to amortize cost.- Right-sizing and optimization: model quantization, batching strategy, and specialized inference runtimes reduce cost; naive containerization misses these.- Hidden costs: labeling for monitoring, storage for features and logs, costs of A/B experiments and rollback windows.

Scalability and performance- Latency vs throughput trade-offs: synchronous inference needs low-p99 latency; batching for throughput increases latency — requires adaptive routing and dynamic batching.- Autoscaling granularity: scale-by-model or per-version; horizontal pod autoscaler reacts to CPU not model-specific metrics, so custom controllers are required.- Multi-tenancy and fairness: resource isolation, admission control, and quota policies needed as number of models/users grows.

Recommended platform elements (mitigations)- Model registry + CI/CD for models (automated validation, canary/tiered rollouts)- Inference layer: model servers (Triton/Seldon) with GPU-aware schedulers- Orchestration: Kubernetes + custom autoscalers (KEDA), service mesh for routing/splits- Observability: end-to-end tracing, drift/error monitoring, labeled-data capture- Cost controls: dynamic batching, cold-start strategies (scale-to-zero + warm pools), spot instances where possible

Diagrams to attach1. High-level serving architecture: clients → LB → API gateway → model inference cluster (model server pods) → feature store/cache → monitoring & logging → model registry/CI.2. Model CI/CD pipeline: training → validation → registry → automated tests → canary rollout → promote/rollback.3. Scaling & latency trade-off: adaptive batching, autoscaler, warm pool vs scale-to-zero flow.4. Cost comparison: always-on GPU containers vs autoscaled/batched inference (cost per 1M requests).5. Observability/Feedback loop: inference telemetry → drift detection → retraining pipeline.

Conclusion: Containers + LB is a starting point, but production ML needs platform patterns (registry, optimized runtimes, autoscaling, observability, governance) to be reliable, performant, and cost-effective.

EasyTechnical

50 practiced

Explain the criteria you would use to decide between synchronous (HTTP/gRPC) vs asynchronous (message-bus, Kafka) communication between an API gateway and model-serving backend for an ML inference request. In your answer, include how you would reflect this decision on an architecture diagram and the documentation you'd write for SREs.

Sample Answer

High-level rule: choose synchronous (HTTP/gRPC) when the client needs a low-latency, request-response result within the user/API call SLA; choose asynchronous (message bus/Kafka) when you need high throughput, decoupling, buffering, or long-running/queued inference where immediate response isn't required.

Key decision criteria:- Latency & SLA: <100–200ms inference -> synchronous (gRPC for binary/low-latency; HTTP/REST when simpler). If SLA allows seconds/minutes or callback/notification, consider async.- Throughput & burst tolerance: high sustained or spiky traffic -> async with Kafka to buffer and smooth load.- Reliability & durability: require at-least-once/guaranteed processing and replayability -> Kafka.- Ordering & consistency: need strict ordering -> Kafka partitions; if not important, sync is fine.- Backpressure & resource control: async enables natural backpressure and consumer scaling; sync needs autoscaling and circuit breakers.- Complexity & operational cost: sync simpler to implement and debug; async requires schema registry, consumer groups, offsets, retention policies.- Idempotency & retries: async systems require idempotent consumers and dedupe; sync can use immediate retries but must avoid blocking resources.- Observability & tracing: distributed tracing is simpler with sync (single request trace), but async needs correlation IDs propagated through messages.

Architecture diagram notes:- Synchronous: API Gateway -> load-balanced gRPC/HTTP(s) -> model-serving fleet (autoscaled). Show TLS, retries, circuit breaker, request timeout, tracing header propagated.- Asynchronous: API Gateway -> produce to Kafka topic(s) (schema registry) -> consumer group of inference workers -> results stored in DB or pushed to response topic/notification service. Show partitions, retention, consumer offsets, dead-letter topic, schema registry, monitoring.

SRE documentation to include:- Decision table: when to use sync vs async and supported SLA tiers.- Endpoints & contract: gRPC proto / HTTP swagger, field-level schemas, expected payload sizes, max request rate, timeouts.- Kafka contract: topic names, partition keys, schema versions, retention, compaction, acks, required producer configs.- Operational runbook: common alerts (consumer lag, topic under-replicated partitions, high tail latency, error-rate), mitigation steps (scale consumers, increase partitions, restart brokers, apply backpressure), DLT handling.- Metrics & SLIs: p95/p99 latency, QPS, consumer lag, error rates, successful inferences, resource utilization.- Testing & rollout: canary strategy, load-test thresholds, replay procedure for reprocessing messages.- Security & compliance: auth (mTLS/OAuth), encryption at rest/in transit, PII handling.This gives SREs runnable playbooks and a clear architecture to operate both modes reliably.

MediumTechnical

55 practiced

Hardening documentation request: produce a short checklist and diagram annotations to show how TLS, mTLS, and network segmentation are applied end-to-end in an ML serving stack (ingress, API gateway, internal services, feature store). Explain how you'd represent secrets management and rotation policies in the docs.

Sample Answer

Checklist — high level (apply to ingress, API gateway, model servers, feature store, internal services):- TLS at edge: enforce HTTPS (TLS1.3), HSTS, modern cipher suites, OCSP stapling.- mTLS for internal service-to-service auth: mutual cert validation and policy enforced by service mesh (e.g., Istio) or sidecar.- Network segmentation: separate VPCs/subnets and Kubernetes namespaces for ingress, API layer, model serving, data/feature store; least-privilege NSG/SG rules.- API gateway responsibilities: terminate external TLS, validate JWT/OAuth, RBAC, rate-limit, forward requests over mTLS to internal services.- Secrets: store in Vault/KMS; no plaintext in config or images; use short-lived credentials via dynamic secrets.- Logging & monitoring: TLS/mTLS handshake telemetry, cert expiry alerts, audit logs for secrets access.- Automated rotation: certs and keys rotated automatically; emergency revocation plan.- Testing: periodic pen-tests, TLS scanners (SSL Labs), and mTLS verification scripts.

Diagram annotations (left→right flow):1) Client → Ingress Load Balancer: HTTPS (TLS1.3). Edge certs in cloud KMS; LB enforces ciphers, HSTS.2) LB → API Gateway: TLS termination optional at LB; API Gateway enforces external TLS and validates client identity (JWT/OIDC). Annotation: "Ingress TLS boundary".3) API Gateway → Internal Mesh: mTLS enforced (client cert from gateway) via sidecar/proxy. Annotation: "mTLS: mutual auth + encryption between gateway and services".4) Internal Services (model-server, feature-store, preprocessing): each runs in separate namespace + subnet. Annotation: "Network segmentation: restrict egress/ingress with NSGs and NetworkPolicies".5) Feature Store ↔ Model Server: mTLS and RBAC; data plane encrypted, metadata access controlled by IAM roles.6) Secrets store (Vault/KMS) annotated centrally: services request short-lived creds via authenticated identity (Kubernetes service account/IAM). Annotation: "Dynamic secrets; signed by CA; audit logged".

Secrets management & rotation representation in docs:- Diagram shows Vault/KMS + PKI CA component. Document policies: - Storage: all secrets in Vault/KMS, sealed at rest with provider-managed HSM. - Access: strong auth (OIDC/Kubernetes SA or IAM), least privilege policies per role, no static tokens in repos. - Rotation: short-lived TLS client certs (e.g., 24–72 hours) issued by internal CA; automated renewal by sidecar or cert-manager; root/intermediate CA rotated annually with staged rollover; application credentials rotated daily or on-use (dynamic DB creds). - Revocation: CRL/OCSP endpoints and automated revocation upon compromise; emergency revocation runbook. - Audit & alerting: all issuance/renewal/revocation events logged to immutable audit store; alerts for expirations <7 days.- Include snippet examples (in docs) of Vault policy, cert-manager config, and sample networkpolicy rules for reproducibility.

Unlock Full Question Bank

Get access to hundreds of System Architecture Communication and Documentation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.