Model Training and Selection Questions

Encompasses choosing appropriate model families and designing robust training strategies. Covers model selection trade offs between linear models, tree based methods, and neural networks depending on data size, interpretability, and latency constraints. Includes practical aspects of training at scale such as distributed training approaches, data and model parallelism, gradient accumulation, checkpointing, experiment tracking and versioning, design of training pipelines, hyperparameter search and resource aware tuning, handling class imbalance and noisy labels, cost management and compute efficiency, and evaluation strategies including controlled experiments and A and B testing. Also discusses retraining policies such as periodic retraining, trigger based retraining for data drift, and continuous or online learning patterns.

MediumSystem Design

89 practiced

Design a training platform to support up to 100 concurrent experiments per day where each experiment can use up to 8 GPUs and train on datasets up to 1TB. Describe cluster orchestration, job scheduling, autoscaling, spot/spot-instance usage, checkpointing strategy, experiment tracking, and guardrails for fair resource quotas between teams.

Sample Answer

Requirements & constraints:- Up to 100 concurrent experiments/day, each up to 8 GPUs and 1 TB datasets, tolerant of spot preemption, resume from checkpoints, fair team quotas, low latency scheduling.

High-level architecture:- Kubernetes cluster with multiple node pools: on-demand GPU nodes (cold), spot GPU nodes (cheap, preemptible), CPU-only nodes for orchestration. Use a managed K8s (EKS/GKE/AKS).- Use a distributed-training operator (Kubeflow MPIJob or PyTorch Operator) to launch multi-GPU jobs in pods with NCCL/RDMA support.

Job scheduling & fair sharing:- Front-end service / API for experiment submission (yaml/json). Jobs submit as Kubernetes custom resources.- Scheduler layer: use K8s scheduler + Volcano or custom scheduler for gang scheduling and fairshare. Enforce per-team quotas via ResourceQuota and PriorityClasses; implement fair-share by weight and job age to avoid starvation.- Gang scheduling ensures all 8 GPUs allocated simultaneously or job waits.

Autoscaling:- Cluster Autoscaler for node pools, scale based on pending pods. Separate autoscaling rules for spot and on-demand pools.- Use Horizontal Pod Autoscaler for auxiliary services, not for training pods.- Maintain a small buffer of warm on-demand nodes to reduce cold-starts for priority experiments.

Spot instances & preemption:- Prefer spot node-pool for best-effort / low-cost jobs. Tag experiments with priority.- Use node affinity/taints to prefer spot nodes; fallback to on-demand when spot capacity unavailable.- Use pod disruption budgets and preemption hooks: on spot interruption webhook, receive SIGTERM, trigger immediate checkpoint save (see below) and let K8s drain.

Checkpointing & data:- Periodic and event-driven checkpointing to durable object storage (S3/GCS) with upload retries and atomic renames. Use incremental checkpointing (e.g., saving only optimizer + model diffs) to reduce I/O.- Store metadata in a database (Postgres) with checkpoint versions, locations, and experiment state.- For large datasets (1TB): mount via POSIX-compatible cached filesystem (e.g., S3 Fuse with local caching, or a distributed file system like Lustre/FSx) to avoid full local copies. Stage hot shards to local NVMe for high-throughput training.- Use resumable training harness (PyTorch Lightning or custom) that reads checkpoint manifest and resumes deterministic RNG states.

Experiment tracking & reproducibility:- Central MLFlow or Weights & Biases: log hyperparams, metrics, artifacts, and checkpoint references. Link to experiment CRD id.- Store environment/container image, git commit, command-line, and dataset checksum to ensure reproducibility.

Monitoring & observability:- Prometheus + Grafana for GPU/CPU/memory/IO; logs to ELK/Cloud Logging. Alert on high preemption, long queue times, or quota breaches.

Guardrails & quotas:- Per-team resource quotas (max concurrent GPUs, max daily GPU hours) enforced via admission controller and quota controller.- Cost-aware policies: soft limits with approvals for burst beyond quota; hard caps enforced by admission.- Priority classes: critical jobs use on-demand nodes and cannot be scheduled on spot; best-effort use spot only.- Automated policy: if team exceeds budget, throttle new submissions and notify owners.- Security: RBAC, image scanning, network policies, least-privilege S3 access with IAM roles per experiment.

Trade-offs & reasoning:- Kubernetes + operator gives flexible orchestration and gang scheduling; Volcano helps fairness. Spot instances reduce cost but require robust checkpoint/resume and quick interruption handling. Centralized experiment metadata + artifact store ensures reproducibility and traceability. Quotas and priority classes balance fairness vs. urgent workloads.

Edge cases:- Massive simultaneous spot revocations: fall back to on-demand or queue until capacity available.- Very large checkpoints: use multipart upload and chunked restore, prefer incremental checkpoints to save bandwidth.- Network bottlenecks: use colocated storage or cached shards.

This design balances cost, reliability, and fairness while enabling up to 100 concurrent experiments using GPUs efficiently.

HardSystem Design

54 practiced

Architect a multi-tenant ML training platform for an enterprise of 500 ML engineers. The platform must provide per-team isolation, quota management, secure access controls, experiment tracking, model registry, multi-cloud support, cost controls, autoscaling GPU clusters, and compliance auditing. Outline key components, APIs, tenant isolation model, pricing/cost attribution, and failure recovery strategies.

Sample Answer

Requirements & constraints:- 500 ML engineers across multiple teams, per-team isolation, quotas, RBAC, experiment tracking, model registry, multi-cloud (AWS/GCP/Azure), GPU autoscaling, cost attribution, compliance/audit logs, low-latency provisioning.

High-level architecture:- Frontend UI + CLI + REST/gRPC API Gateway (Authn/z via SSO/OIDC, MFA)- Control Plane: Orchestrator (Kubernetes control, cluster managers per cloud), Quota/Policy Service, Billing & Cost Attribution, Audit Logging, Metadata + Experiment Tracking (MLflow-compatible), Model Registry, Secrets Manager, Artifact Store (S3-compatible), Scheduler for training jobs, Autoscaler for GPU pools.- Data Plane: Per-tenant Namespaces / VPCs with isolated Kubernetes namespaces + network policies + IAM roles; optional dedicated clusters per team for high isolation.- Multi-cloud layer: Cluster Agents in each cloud reporting to Control Plane via secure mTLS.

Key components & responsibilities:- AuthZ Service: RBAC/ABAC policies, team membership, workspace scoping.- Quota Manager: enforces CPU/GPU/Storage/ConcurrentJobs limits, soft/hard quotas, alerting.- Autoscaler: scales GPU node pools by prioritized queue; integrates spot/preemptible handling.- Experiment Tracking API: log metrics/artifacts, lineage tracking, tags, search.- Model Registry API: versioning, approvals, signatures, vulnerability scans, deployable bundles.- Cost Attribution: resource-metering at pod level + cloud-cost ingestion -> map to team, project, experiment; supports showback/chargeback.- Compliance & Audit: immutable audit logs, retention policies, CASB integration, drift detection.

APIs (REST/gRPC) examples:- POST /v1/train -> {team, project, image, resources, datasetURI, params}- GET /v1/experiments/{id}/metrics- POST /v1/models/{id}/promote -> {stage, approver}- GET /v1/quotas/{team}

Tenant isolation model:- Default: single control plane, strongly isolated namespaces with network policies, K8s PodSecurityPolicies, IAM role per team, and encrypted per-tenant secrets. For stricter needs: dedicated clusters/VPCs per team or compliance-based ephemeral clusters.- Storage isolation: per-tenant bucket prefixes + encryption keys (KMS per-tenant or envelope encryption).- Execution isolation: pod-level seccomp, uid remapping; optional GPU node taints and node pools per tenant.

Pricing / cost attribution:- Meter CPU/GPU hours, storage, egress, managed services. Use sidecar or kubelet telemetry to emit resource usage to Billing service. Map usage to tags: team/project/experiment. Provide reports, budgets, alerts, and quota enforcement. Support showback (reports) and chargeback (invoicing) with configurable rates per resource and spot discounts.

Failure recovery & reliability:- Control plane: active/active multi-AZ, stateless services behind LB, state stores (Postgres, Elasticsearch, object store) with backups and cross-region replication.- Job resilience: checkpointing support, retries, preemption-aware training (save checkpoints to artifact store), job resubmission policies.- Autoscaler: graceful scale-down with drain and checkpoint window; preemptible instance fallbacks.- Disaster recovery: periodic snapshots of metadata + registry + artifact store; runbook for failover to secondary region/cluster. DR tests quarterly.

Security & compliance:- SSO + least-privilege RBAC, signed images (notary), vulnerability scanning, secret rotation, KMS key separation per-tenant, encrypted audit logs, SIEM integration, SOC2/GDPR controls.

Trade-offs:- Single control plane saves ops cost but requires strong software isolation; dedicated clusters increase cost but simplify compliance. Favor hybrid: most teams share control plane, high-risk teams get dedicated clusters.

Operational notes:- Offer templates and SDKs (Python SDK, CLI) to submit jobs; provide quotas + cost dashboards, automated policy enforcement, and onboarding flow.

MediumTechnical

50 practiced

A training job on a GPU cluster intermittently throws OOM errors during the final epoch only. Describe a systematic debugging approach to reproduce the issue and propose short-term mitigations and long-term fixes. Mention profiling tools, memory leak causes, and checkpoints to avoid wasted compute.

Sample Answer

Approach — reproduce first1) Make the run deterministic and reproducible: fix random seeds, single-GPU run, same data/shuffle order, same software stack (CUDA/cuDNN/PyTorch/TensorFlow). Try reproducing the failure locally with a copy of the final-epoch workload (e.g., train for N-1 epochs then run final epoch repeatedly) or by reducing dataset size so the final epoch occurs faster.2) Add lightweight logging across epochs: GPU mem via nvidia-smi, torch.cuda.memory_reserved()/allocated()/reset_peak_stats(), or TensorFlow’s tf.config.experimental.get_memory_info() to spot monotonic growth.

Profiling tools- System: nvidia-smi -l 1, Nsight Systems (nsys), nvprof deprecated.- PyTorch: torch.cuda.memory_summary(), torch.cuda.max_memory_allocated(), torch.autograd.profiler.profile, torch.profiler (with record_shapes, profile_memory).- TensorFlow: tf.profiler, tf.config.experimental.get_memory_info(), TensorBoard profiling.- Python-level: tracemalloc, objgraph, memory_profiler to find retained Python objects.

Common leak causes to check- Non-detached tensors kept across iterations (e.g., appending loss.item() vs loss tensor).- Accidental .retain_graph() or storing computation graph in lists, caches, or logging.- Hooks that capture inputs/outputs.- DataLoader worker pin_memory / leaking iterators or persistent caches.- Gradients accumulated unintentionally (optimizer.step missing or zero_grad misplaced).- Third-party libs caching augmentation outputs.

Short-term mitigations- Reduce batch size or switch to gradient accumulation smaller steps.- Enable mixed precision (AMP) to reduce memory footprint.- Call torch.cuda.empty_cache() between epochs (temporary relief).- Force torch.cuda.reset_peak_stats() and gc.collect() to free Python refs.- Resume from a checkpoint before final epoch and run final epoch with smaller batch.

Long-term fixes- Find root cause via profiler/tracemalloc and remove retained references; ensure tensors saved for logging are .detach().cpu().numpy().- Use gradient checkpointing to trade compute for memory.- Add unit tests that run many short epochs to catch leaks in CI.- Implement robust checkpointing: periodic/full checkpoints (model + optimizer + RNG states) and “safe” last-good checkpoint before final epoch so compute isn’t lost.- Add monitoring/alerts for memory trends and automated retry logic.

Checkpoints to avoid wasted compute- Save periodic checkpoints (every N epochs) and an incremental “best” checkpoint.- When a run fails in final epoch, resume from last checkpoint with adjusted config (smaller batch / AMP) rather than restarting from scratch.- Save optimizer and scheduler state and RNG seeds to reproduce exact training state when resuming.

Example quick PyTorch snippet to detect growing refs:

python

for epoch in range(epochs):
    train()
    print(torch.cuda.memory_summary())
    import gc; gc.collect()

This systematic approach helps reproduce, identify whether the problem is code-level retention vs infrastructure, and apply temporary mitigations while implementing permanent fixes.

HardTechnical

59 practiced

Write Python pseudocode for a training job orchestrator that supports fault-tolerant checkpointing, automatic retries with exponential backoff, migration of jobs upon preemption to new nodes, and consistent selection of the 'best model' across retries. Assume a simple job API and object store for checkpoints; focus on orchestration logic and correctness.

Sample Answer

Approach: orchestrator runs training jobs with strong checkpointing to an object store, retries with exponential backoff, migrates on preemption by rescheduling to new node and restoring latest checkpoint, and deterministically tracks "best model" across attempts by evaluating checkpoint metadata (e.g., validation metric) and keeping global best.

python

import time, random
from typing import Optional

# Pseudocode interfaces
class JobAPI:
    def submit(node, checkpoint_uri=None) -> job_id: ...
    def status(job_id) -> ("RUNNING"|"FAILED"|"SUCCEEDED"|"PREEMPTED"): ...
    def request_cancel(job_id): ...
    def get_logs(job_id): ...

class ObjectStore:
    def upload(path, data): ...
    def list_prefix(prefix) -> [uris]: ...
    def download(uri) -> data: ...

# Orchestrator
MAX_RETRIES = 5
BASE_BACKOFF = 5  # seconds
MAX_BACKOFF = 300

def choose_node():
    # placeholder for scheduling policy
    return "node-" + str(random.randint(1,10))

def latest_checkpoint_uri(job_name):
    entries = ObjectStore.list_prefix(f"{job_name}/checkpoints/")
    return max(entries, key=lambda u: metadata(u).timestamp) if entries else None

def metadata(uri):
    # returns object with {timestamp, val_metric, step}
    return ObjectStore.download(uri + ".meta")

def update_global_best(job_name, ckpt_uri):
    meta = metadata(ckpt_uri)
    best = ObjectStore.download(f"{job_name}/best.meta") if exists(f"{job_name}/best.meta") else None
    if not best or meta.val_metric > best.val_metric:
        ObjectStore.upload(f"{job_name}/best.ckpt", ckpt_uri)            # pointer
        ObjectStore.upload(f"{job_name}/best.meta", meta)
        return True
    return False

def orchestrate(job_name, entry_point):
    attempt = 0
    backoff = BASE_BACKOFF
    global_best = None

    while attempt <= MAX_RETRIES:
        node = choose_node()
        ckpt = latest_checkpoint_uri(job_name)  # resume if possible
        job_id = JobAPI.submit(node, checkpoint_uri=ckpt)
        start = time.time()

        while True:
            s = JobAPI.status(job_id)
            if s == "RUNNING":
                time.sleep(2)
                continue
            if s == "SUCCEEDED":
                # trainer should have uploaded final checkpoint and meta
                final_ckpt = latest_checkpoint_uri(job_name)
                update_global_best(job_name, final_ckpt)
                return {"status":"SUCCEEDED", "best": ObjectStore.download(f"{job_name}/best.meta")}
            if s == "PREEMPTED":
                # immediate migration: preserve latest checkpoint, reschedule
                latest = latest_checkpoint_uri(job_name)
                if latest:
                    update_global_best(job_name, latest)
                JobAPI.request_cancel(job_id)
                attempt += 1
                break  # resubmit
            if s == "FAILED":
                # transient or deterministic failure: retry with backoff
                JobAPI.request_cancel(job_id)
                attempt += 1
                time.sleep(min(backoff, MAX_BACKOFF))
                backoff *= 2
                break

        # loop continues to resubmit until attempts exhausted

    return {"status":"FAILED", "best": ObjectStore.download(f"{job_name}/best.meta") if exists(f"{job_name}/best.meta") else None}

Key points:- Checkpoints and metadata are authoritative; orchestrator picks latest by timestamp/step.- On preemption, we immediately reschedule to a new node using latest checkpoint.- Exponential backoff mitigates flapping; retries limited.- Global best stored atomically as pointer + meta to ensure consistent selection across retries.

Edge cases:- Partial checkpoint uploads: use atomic rename or write-then-tag pattern.- Conflicting concurrent uploads: use compare-and-swap on best.meta.- Long resume time: add TTLs, spot/preemptible awareness.

Alternatives:- Use lease-based node allocation, consensus service (etcd) for global best, or controller that coordinates multi-node synchronous checkpoints.

EasyTechnical

56 practiced

You receive a new tabular regression task. Describe a sequence of simple baseline models and evaluations you would run before training complex models. Include data sanity checks, simple feature engineering, linear and tree baselines, and how to interpret signal-to-noise to decide next steps.

Sample Answer

1) Data sanity checks (first 1–2 hours)- Row/column counts, dtypes, missingness, cardinality, outliers (IQR/z-score), target distribution and range, duplicates, time/order leakage.- Quick plots: histogram of target, scatter vs key numeric features, correlation matrix; check units and impossible values.

2) Simple baselines (fast, interpretable)- Dummy predictors: predict mean and median; compute MAE/RMSE/R² — these set the floor.- Feature checks: target vs single-feature linear fit; create simple features (log transform, bins for skewed numerics, count/freq encoding for categoricals, datetime parts). Evaluate with cross‑val (k-fold or time-series split if temporal).

3) Linear models- OLS / Ridge / Lasso with standardized features. Use pipelines and one-hot or target encoding for categoricals.- Examine coefficients, residual plots, and train/val gap. If Ridge outperforms OLS, indicates multicollinearity/overfitting.

4) Tree baselines- Shallow Decision Tree (max_depth=3) to capture simple nonlinearity; then RandomForest/LightGBM with default params.- Use cross‑validation and early stopping (for boosting). Compute permutation feature importance and partial dependence for top features.

5) Evaluate signal-to-noise- Compare model RMSE to baseline (mean) and to estimated measurement noise (if available). Compute explained variance (R²); low R² with small gap between simple and complex models suggests limited signal.- Learning curves: plot train vs val error vs data size. If both errors high and close → high bias (need better features/model complexity). If train error low but val high → high variance (regularize or get more data). If errors plateau → irreducible noise/feature ceiling.

6) Decide next steps- Good improvement over mean + clear feature importances → invest in tuning/ensembling and feature engineering.- Small improvement or low R² and plateauing learning curves → focus on better features, label quality, or gather more/cleaner data rather than heavier models.

Practical checks: always use robust CV, log transforms for skew, and track metrics in experiments.

Unlock Full Question Bank

Get access to hundreds of Model Training and Selection interview questions and detailed answers.

Join thousands of developers preparing for their dream job.