Technical Debt and Sustainability Questions

Covers strategies and practices for managing technical debt while ensuring long term operational sustainability of systems and infrastructure. Topics include identifying and classifying technical debt, prioritization frameworks, balancing refactoring and feature delivery, and aligning remediation with business timelines. Also covers operational concerns such as monitoring, observability, alerting, incident response, on call burden, runbook and lifecycle management, infrastructure investments, and architectural changes to reduce long term cost and risk. Includes engineering practices like test coverage, continuous integration and deployment hygiene, code reviews, automated testing, and incremental refactoring techniques, as well as organizational approaches for coaching teams, defining metrics and dashboards for system health, tracking debt backlogs, and making trade off decisions with product and leadership stakeholders.

EasyTechnical

67 practiced

What are the most important monitoring metrics for assessing the health and sustainability of a deployed ML model from a testing and reliability perspective? Describe at least five metrics, who cares about each, and what thresholds or alerting logic you would consider.

MediumTechnical

80 practiced

Compare and evaluate automated data validation frameworks such as Great Expectations, Deequ, and custom validation code for an ML platform. For each approach list strengths, weaknesses, integration points with CI, and how they affect long-term technical debt and testability.

Sample Answer

Summary: For an ML platform, choose between out-of-the-box frameworks (Great Expectations, Deequ) and custom validation code based on scale, team skills, and need for standardization. Below is a concise comparison focused on strengths, weaknesses, CI integration, technical debt, and testability.

Great Expectations- Strengths: Rich, human-readable expectations, data docs, numerous built-in checks (distributions, types, uniqueness). Good for data contracts and stakeholder visibility.- Weaknesses: Python-centric, heavier to run in distributed environments, can require templates for large numbers of datasets.- CI Integration: Integrates via pytest/CLI; run expectations in pipelines (GitHub Actions, Jenkins); can block PRs on failing suites.- Tech debt & Testability: Lowers debt by standardizing checks and documentation; tests are declarative and versionable—easy to mock and unit-test expectation suites.

Deequ- Strengths: Scala/Java library optimized for Spark; fast, scalable metrics and anomaly detection; good for large-scale batch pipelines.- Weaknesses: JVM ecosystem only; less user-friendly docs/UI compared to GE; expectation expressiveness is code-based.- CI Integration: Embed checks in Spark jobs; run as part of CI by spinning test clusters or using lightweight Spark runners; integrate results into CI gates.- Tech debt & Testability: Lowers runtime cost for big data but can introduce JVM/Spark infra debt; tests require Spark context—use local Spark for unit tests.

Custom Validation Code- Strengths: Maximum flexibility; can encode business-specific rules and complex transformations; language/framework agnostic.- Weaknesses: Reinventing features (reporting, metrics, lineage); inconsistent implementations across teams; higher maintenance.- CI Integration: Straightforward—unit tests, integration tests, and pipeline checks; but requires building test harness and reporting.- Tech debt & Testability: Higher long-term debt unless well-architected; testability depends on modularity—good coverage and reusable utilities mitigate risks.

Recommendation (ML Engineer view)- For medium+ scale and cross-team needs, use Great Expectations for Python-centric pipelines and Deequ for Spark-heavy jobs; enforce common patterns and CI gating.- Reserve custom code for narrow, complex validations; wrap them in standardized test harnesses and emit metrics compatible with the chosen framework to avoid fragmentation.

MediumTechnical

81 practiced

Design a testing plan to detect and prevent data leakage between training and evaluation datasets in a supervised learning pipeline. Include automated checks, schema or timestamp validations, and post-training statistical tests you would run in CI.

Sample Answer

Overview: Build a layered testing plan that (1) prevents leakage at ingestion, (2) detects accidental leakage before training, and (3) verifies no leakage post-training in CI using deterministic, automated checks and statistical tests.

Pre-ingest / automated pipeline checks:- Schema constraints: enforce feature types, nullable flags, cardinality limits, and prohibited columns (e.g., future labels or derived features flagged as "leak-prone").- Column allowlist/blocklist in config; fail job if blocklist present.- Referential integrity: validate joins (no one-to-many explosion) and check join-key uniqueness ratios.- Timestamp validations: require event timestamps and enforce monotonicity ranges; reject data with timestamps in model evaluation window.

Pre-training unit tests (run in CI on sampled data):- Data split determinism: reproducible random seed; verify splits by hash of key feature(s).- Temporal split checks: ensure train_max_timestamp < eval_min_timestamp (or configurable gap).- Feature leakage rules: automatically compute correlation/MI of each feature with label on combined dataset—flag features with extremely high correlation in small windows.- Duplicate/key overlap check: compute fraction of keys in train ∩ test; fail if > threshold (e.g., 0%).

Post-training statistical tests (CI smoke tests):- Label distribution drift: compare label distribution (KS test / chi-square) between train and eval; flag large discrepancies that indicate leakage or sampling bugs.- Feature distribution sanity: for top-k features, run two-sample KS or Wasserstein tests; unexpected perfect matches for many features suggest leakage.- Target leakage detector: train a simple surrogate model on combined data to predict split membership; AUC >> 0.6 indicates dataset differences/leakage.- Model performance sanity: ensure eval metrics are within expected ranges vs historical baseline; sudden jumps suggest leakage.

Operationalization / workflows:- Integrate checks into CI as separate stages: unit checks → pre-training validations → train → post-training tests → gated deployment.- Fail fast with clear error messages and a remediation checklist pointing to likely root causes (bad join, pipeline rerun, mislabeled timestamp).- Maintain provenance metadata (dataset hashes, schema versions, split seeds, job run id) stored in ML metadata store for reproducibility and audit.

Monitoring in production:- Continuously monitor prediction-feature and feature-label correlations; alert if new high correlations appear.- Periodic replay tests comparing live predictions on shadow data vs expected behavior.

This layered approach combines deterministic schema/timestamp enforcement, automated overlap/correlation checks pre-training, and statistical evidence post-training to detect and prevent data leakage.

MediumTechnical

60 practiced

Your team logs show a sharp increase in page counts related to model-serving alerts caused by transient upstream data issues. As the ML engineer on-call, outline a triage and remediation plan to both fix the immediate problem and reduce future on-call burden. Include metrics to track progress.

Sample Answer

Situation: Alerts spiked for model-serving pages after transient upstream data issues (bad schema / nulls / rate spikes), causing noisy on-call pages and degraded predictions.

Immediate triage (first 30–60 minutes):- Triage & scope: Confirm alert validity, correlate timestamps with upstream pipeline logs, and identify affected models/endpoints and traffic %.- Mitigate blast radius: - Activate traffic diversion: switch affected endpoints to a safe fallback (cached predictions or simple rule-based model) via feature flag or load‑balancer routing. - If rollback possible and safe, disable the latest model deploy or pause the data feed consumer.- Short-term fix to stop pages: increase alert thresholds temporarily or mute non-actionable paging alerts while preserving incident logging.- Communicate: post incident status to stakeholders and on-call rotations (estimated ETA, actions taken).

Root-cause remediation (hours to days):- Fix upstream issue with data team (schema, null handling, retry/backoff). Add idempotent consumers where needed.- Harden model-serving: - Input validation layer with schema checks and reject/transform bad records. - Defensive model wrappers: graceful degradation (fallback predictions), default outputs when inputs invalid. - Circuit breaker pattern: if upstream error rate > X% or latency > Y ms, route to fallback.- Deploy quick tests: canary on small traffic with enriched logging.

Long-term engineering/process changes (weeks):- Instrumentation & observability: - SLIs/SLOs for prediction success rate, input validation pass rate, model latency, and upstream data error rate. - Dashboards showing upstream producer health correlated with model errors.- Alerting improvements: - Convert noisy alerts into paged vs. non-paged categories; require multiple symptom thresholds before paging (e.g., sustained 5-minute error rate > threshold). - Add anomaly detection on input schema/feature distributions.- Resilience: unit and integration tests for input schema changes; contract tests with upstream producers.- Runbooks & on-call playbooks updated with step-by-step mitigations and owners.- Postmortem and blameless review with action items and deadlines.

Metrics to track progress:- Page count per incident and pages/week (target: reduce noisy pages by 80%).- MTTD (mean time to detect) and MTTR (mean time to remediate) — aim to reduce MTTR by 50% in 30 days.- Percentage of predictions using fallback vs. primary model (keep fallback <5% normally).- Input validation pass rate (target >99.9%).- Upstream data error rate and time to resolution for upstream incidents.- Number of incidents caused by upstream transient issues per quarter (trend down).

This approach stops immediate impact, fixes root causes, and reduces future on-call burden by improving resilience, observability, and alert quality.

MediumTechnical

62 practiced

You're receiving frequent low-priority alerts from a model serving cluster that cause high on-call burden. Propose an alerting strategy to reduce noise while maintaining safety. Include alert categorization, deduplication, severity tuning, and SRE best practices you would apply.

Sample Answer

Start by clarifying goals: reduce noisy low-priority alerts so on-call can focus on actionable incidents while preserving safety for user-facing or model-safety failures.

1) Alert categorization- Tier A (P1/P0): user-impacting model failures — model returns NaNs, 100% prediction latency degradation, 5xx serving errors, data schema mismatch causing mispredictions.- Tier B (P2): degradations that need investigation but not immediate rollback — sustained accuracy drift > X% over 1 hour, throughput drop 30% with retries succeeding.- Tier C (P3/info): transient, low-impact events — occasional increased latency spikes, minor feature distribution shifts, single-host resource alerts.

2) Deduplication & grouping- Use upstream aggregation: group alerts by root cause (model id, deployment, request trace id) and suppress duplicates within a short window (e.g., 5–15 min) using dedupe keys: model_version + endpoint + error_type.- Implement correlation rules: if many host-level CPU alerts map to a single autoscaler event, open one incident.

3) Severity tuning & alert windows- Convert short-lived spikes into metrics with evaluation windows (e.g., 5m->15m) and require sustained violation before paging.- Use multi-threshold alerts: warn (email/slack) for transient deviations, page on sustained breach or escalation (warn -> page).- For model drift: require both statistical drift AND drop in a business metric (proxy label or sentinel) before paging.

4) SRE best practices- Runbooks: attach concise remediation steps and playbooks (rollback model, scale replicas, flush cache).- Ownership & SLAs: assign clear alert owners per model; define SLOs (latency, availability, prediction quality) and alert on SLO burn rate.- Alert fatigue metrics: track MTTA/MTTR and paging frequency per alert; retire alerts with low-actionability.- Automated remediation: safe auto-remediation for well-tested cases (traffic shift to previous model after 3 consecutive 5m errors).- Postmortems & tuning: blameless reviews for noisy alerts; tune thresholds and detection logic after incidents.

Example rules:- Page if 5xx rate > 5% sustained for 15m OR median latency > 2s for 10m.- Slack notify if feature drift score > threshold for 1 hour; page only if paired with label-based metric drop > 3%.

This balances noise reduction with safety by ensuring pages only for sustained, high-impact issues while surfacing early signals as lower-severity notifications for investigation.

Unlock Full Question Bank

Get access to hundreds of Technical Debt and Sustainability interview questions and detailed answers.

Join thousands of developers preparing for their dream job.