Performance Monitoring & Observability Questions

Instrumenting systems so performance is continuously measured and regressions are visible. Covers performance metrics and SLIs, dashboards and time-series signals, tracing, alerting on latency and saturation, and using telemetry to guide tuning. Focuses on the ongoing measurement loop rather than one-off profiling.

MediumTechnical

47 practiced

Outline how you would instrument a backend API to provide distributed tracing across services implemented in Python, Java, and Go. Explain context propagation mechanisms, span tagging for model metadata (model id, version, input size), sampling strategies, and how traces would help diagnose high-latency model calls and resource contention.

Sample Answer

Situation: I’d instrument services using OpenTelemetry (OTel) and OTLP export to a tracing backend (Jaeger/Tempo/Lightstep). Use language SDKs for Python, Java, Go so instrumentation is consistent.

Context propagation:- Use W3C Trace-Context (traceparent/tracestate) as primary propagation for cross-language compatibility; fall back to B3 if any older components require it.- Initialize automatic instrumentation (HTTP frameworks, gRPC, message queues) so the incoming trace context is extracted and attached to the current span; ensure outgoing HTTP/gRPC clients inject trace headers.

Span and attribute design (model metadata):- Create spans for API request → orchestration → model inference. Attach attributes on the inference span: - model.id: "resnet50-v2" - model.version: "2025-03-12-finetune" - model.framework: "torch" - input.size: 1024 (bytes) or input.shape: [1,3,224,224] - batch.size: 8 - device: "gpu:2" - model.stage: "preproc/infer/postproc"- Record events for GPU queueing, model load, cache hits, and resource waits.

Sampling strategies:- Default: probabilistic (e.g., 1-5%) to control volume.- Adaptive tail-based sampling: keep full traces with high latency/error rates; important for catching expensive model calls. Configure rules: sample 100% when status >= 500 or latency > SLO threshold (e.g., >300ms).- Head-based sampling for cost control; enrich sampled set using dynamic decisions (e.g., A/B groups, model versions under experiment).

How traces diagnose issues:- High-latency model calls: follow trace tree to find whether latency is in network, preprocessing, GPU queue, model execution, or post-processing. Inference span split into child spans (queue_wait, model_load, forward_pass, memcpy) to localize.- Resource contention: traces with device attribute show multiple long queue_wait durations and correlated high GPU utilization; correlation with host metrics (prometheus) and span tags (batch.size) reveals whether large batches cause queuing.- Version regressions: filter traces by model.version to compare p50/p95 latency and error rates; identify which version introduced longer forward_pass times.- Root cause: combine traces with flamegraphs and logs (link trace_id in logs) to reconstruct sequence and correlate with infra metrics.

Operational notes:- Ensure low-overhead instrumentation: sampling, offload heavy attributes (full inputs) to object storage and include reference URL in span.- Secure PII: avoid sending raw inputs; instead send hashed/summarized features where necessary.- Test propagation end-to-end (client → API → inference workers → async storage) and validate trace continuity across language boundaries.

Unlock Full Question Bank

Get access to hundreds of Performance Monitoring & Observability interview questions and detailed answers.

Join thousands of developers preparing for their dream job.