Debugging and Performance Optimization Questions

Requirements & constraints:

• Reduce repeated expensive feature computation for online and batch use; maintain correctness (stale-free within SLA), low latency for online inference, support high write/update rate for upstream data, and durable storage for batch jobs.

High-level architecture:

• Feature compute service writes computed feature vectors to a cache layer + backing feature DB. Online/serving reads first try cache, fallback to compute or DB.

Cache key scheme:

• Key = feature_group : entity_id : feature_set_version : context_hash
  • feature_group (logical family), entity_id (user/item PK), feature_set_version (schema + logic hash or commit id), context_hash (optional: time-bucket or request attributes like locale).
  • Example: purchases:user123:v1.4:2025-11-22T10 -> deterministic, supports versioning and selective invalidation.

Invalidation policy:

• Versioned invalidation: bump feature_set_version when transformation code or logic changes => automatic global invalidation.
• Time-to-live (TTL): per-feature TTL based on freshness requirement (e.g., last_purchase: 24h).
• Event-driven: listen to upstream data change events (CDC) to evict / update affected entity keys.
• Soft vs hard invalidation: soft mark stale to trigger async refresh while serving reads may return stale within bounded staleness SLA if allowed.

Storage layer:

• Hybrid:
  • In-memory (Redis/Memcached cluster) for hot keys and low-latency online reads with LRU and clustering, replication for HA.
  • Disk-backed feature store (Parquet on object store or OLAP DB) as authoritative store for batch recompute and cold reads.
  • Optional local process-level LRU cache for micro-batching to avoid repeated Redis calls.

Consistency & correctness:

• Serve-from-cache only if feature_set_version matches request; otherwise recompute or fetch authoritative store.
• Use write-through or write-back strategies: prefer write-through on compute completion to keep cache and backing store consistent; use compare-and-swap or atomic writes in Redis to avoid races.
• For event-driven updates, prefer push-based update to cache to keep bounded staleness.

Measuring impact:

• Metrics:
  • Cache hit rate (per-feature, per-entity), hit latency histogram, miss latency, end-to-end inference latency, freshness/staleness metric (time since last update), downstream model accuracy drift correlated with staleness.
• KPIs & experiments:
  • A/B test: route subset traffic to cache-disabled path to measure baseline latency and cost; compare inference latency P50/P95, throughput, and model output differences.
  • Calculate cost savings: compute cycles avoided = miss_count * compute_cost; translate to CPU/GPU and cloud cost.
• Monitoring & alerts:
  • Alert on drop in hit rate, increased miss latency, or rising model error correlated with stale features.
• Logging:
  • Log per-request whether served from cache, key version, and staleness to enable root-cause and audit.

Trade-offs:

• Short TTLs reduce staleness but increase recompute; event-driven invalidation lowers staleness but requires reliable upstream events.
• In-memory size vs eviction: prioritize hot entities and critical features; use tiered caching to optimize cost/latency.

This design balances low-latency serving with correctness via versioned keys, event-driven invalidation, and robust monitoring to measure hit/miss impact on latency and model correctness.

Briefly: num_workers controls how many subprocesses load/transform batches in parallel; pin_memory speeds host->GPU transfer by allocating page-locked memory; prefetch_factor controls how many batches each worker preloads; worker_init_fn lets you initialize per-worker state (seeds, open file handles) to avoid contention. Mis-tuning any can cause CPU/GPU stalls, contention, or OOMs.

How each affects performance:

• num_workers: too low → CPU bound; too high → context-switching, memory pressure, file-descriptor contention. Optimal often equals CPU cores minus system/other processes, but depends on per-sample cost.
• pin_memory: generally helps GPU throughput by faster cudaMemcpy; adds CPU memory pressure and slight allocation overhead. Use when training on GPU.
• prefetch_factor: larger value smooths bursts by keeping more ready batches per worker; increase if GPU is idle waiting for data, reduce if memory pressure or worker CPU load is high.
• worker_init_fn: ensure deterministic RNG, avoid re-opening shared resources per batch, and set per-worker file handles to reduce locks/contention.

Methodical experiment:

1. Instrument pipeline: measure data loading time, transform time, batch-to-GPU transfer, and GPU utilization. Use torch.utils.bottleneck, nvprof/nvtx or nsys, and simple timers around DataLoader iteration.
2. Baseline: run one epoch with current settings and record metrics: samples/sec, avg data_load_ms (time from iteration start to batch ready), GPU utilization, CPU usage, memory.
3. Sweep strategy (grid/controlled):
   • Fix seed/environment. Ensure no other heavy processes.
   • Vary num_workers: test {0,1,2,4,8,16} (stop increasing when CPU% >85 or samples/sec decreases).
   • For the best num_workers, test pin_memory True/False and measure transfer latency & samples/sec.
   • For chosen num_workers+pin, sweep prefetch_factor {1,2,4,8} (default 2) to reduce jitter.
4. Use worker_init_fn to set torch.manual_seed(seed + worker_id) and open per-worker resources if needed; verify no contention by inspecting file descriptors and per-worker CPU.
5. Evaluate: choose settings giving highest sustained samples/sec with stable GPU utilization and acceptable CPU/memory. Verify across several epochs and batch sizes.
6. Validate in production workload (same dataset, hardware, container limits).

Practical tips:

• Watch for OS-level limits (ulimit -n), maybe increase file descriptors.
• In containers, ensure CPU quotas/core visibility match expectations.
• If transforms are heavy (e.g., decoding, augmentation), consider moving to faster libraries, parallelizing transforms, or caching preprocessed data.
• Automate profiling and log metrics to compare runs.

This structured sweep with instrumentation finds the sweet spot for your workload and hardware.

Most likely causes

• Missing optimizer.zero_grad() so gradients accumulate and explode.
• Too-large learning rate or bad optimizer hyperparams (e.g., momentum/eps).
• Numerical issues in loss (log(0), division by zero, sqrt of negative), bad labels (NaN/Inf/one-hot mistakes).
• Exploding gradients from model architecture or poor weight init.
• Mixed-precision (AMP) without GradScaler or improper scaling.
• BatchNorm/Dropout mode mistakes (eval vs train) or dataset with extreme outliers.

How to reproduce and trace the exact cause

• Reproduce deterministically: set torch.manual_seed, deterministic cudnn, small dataset, single-batch runs.
• Run a single batch and print/inspect:
  • loss.item(), torch.isfinite(loss)
  • parameter and gradient stats: min/max/mean, torch.isnan checks
• Use torch.autograd.set_detect_anomaly(True) to get stack traces on NaNs/Infs during backward.
• Insert hooks or small wrappers to catch when values become non-finite:
  • register_full_backward_hook or check after backward for any non-finite gradients.
• Isolate components: replace loss with MSE of outputs and zeros to confirm model vs loss function issue.
• Try FP32 only if using AMP to see if mixed precision causes it.

Concrete fixes and production-robust loop

• Always zero grads: optimizer.zero_grad() (or model.zero_grad()) at start of step.
• Validate loss finite before backward; skip batch and log if not.
• Use gradient clipping: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm).
• Lower LR / tune optimizer; add weight decay.
• Add eps safeguards in loss (e.g., log(x + 1e-8)), use stable numerics (log-sum-exp).
• For AMP, use torch.cuda.amp.GradScaler() and scale/unscale properly.
• Add regular monitoring and alerting for NaN/Inf, record batch index and inputs that cause failure.
• Initialize weights appropriately and consider spectral norm / gradient penalties for unstable networks.

Robust training loop example

scaler = torch.cuda.amp.GradScaler()  # if using AMP
for batch_idx, batch in enumerate(loader):
    optimizer.zero_grad()
    x, y = batch['x'].to(device), batch['y'].to(device)
    with torch.cuda.amp.autocast():  # remove if not using AMP
        outputs = model(x)
        loss = loss_fn(outputs, y)
    if not torch.isfinite(loss):
        logger.error(f"Non-finite loss at batch {batch_idx}: {loss}")
        continue  # skip or inspect batch
    scaler.scale(loss).backward()  # use loss.backward() if no AMP
    # optional: detect anomalies (slower)
    # torch.autograd.set_detect_anomaly(True)
    # clip grads
    scaler.unscale_(optimizer)  # required before clipping with AMP
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    scaler.step(optimizer)
    scaler.update()

Extra production best-practices

• Unit-test loss on synthetic inputs, include numeric-stability tests.
• Log gradients/statistics and failing batches to storage for post-mortem.
• Automatic rollback or safe checkpoints when metric collapse detected.
• Use smaller LR and warmup schedule; consider learning-rate finder during experimentation.

These steps let you identify whether the issue is optimizer usage, numerical instability in the loss, data problems, or precision-related—and apply targeted, production-ready fixes.

Requirements:

• Hard SLO: low p95/p99 latency (e.g., p99 < 10ms) with high throughput and mixed model sizes.
• Minimize variability from OS scheduling, CPU contention, memory locality, and task placement.

How each technique reduces tail variability

• Request scheduling: use token-bucket or admission control to bound concurrency per model and avoid queue buildup that creates long tails. Priority-aware schedulers push small/latency-sensitive requests ahead of batch/throughput requests to prevent head-of-line blocking.
• Prioritized queues: separate queues by request class (hot path vs background). Prioritization prevents long-running jobs from delaying short ones, reducing p95/p99 for latency-sensitive traffic.
• CPU pinning: bind worker threads/processes to specific cores to eliminate scheduler preemption and migrations, ensuring consistent CPU cache warmth and predictable latencies.
• NUMA-awareness: allocate memory and place workers on the same NUMA node as their pinned cores; avoid remote memory access which adds high tail latency due to cross-node memory access and bus contention.
• Worker model placement: colocate related models/tasks to minimize cross-core communication; avoid oversubscription and isolate noisy neighbors by dedicating cores/NUMA nodes to critical models.

Evaluation plan on real hardware

1. Baseline: measure p50/p95/p99, throughput, CPU/memory/NIC metrics with existing setup using representative trace (including bursts).
2. Controlled experiments (one change at a time):
   • Admission control thresholds: vary concurrency limits and observe latency distribution.
   • Enable prioritized queues and measure improvement in p95/p99 under mixed workloads.
   • Apply CPU pinning: compare core migrations, context switches, and latency variance.
   • Enable NUMA-aware allocations: compare remote vs local memory page faults and tail latencies.
   • Worker placement: test different placement strategies (dedicated vs shared cores).
3. Metrics & tools: perf, sar, numastat, top, eBPF tracing, flame graphs, histograms, and SLO error budget burn-down. Use qps/latency heatmaps and CDFs to compare p95/p99.
4. Stress tests: inject noisy neighbors, network jitter, and GC pauses to validate robustness.
5. Rollout: staged A/B on production-like nodes with continuous monitoring and automatic rollback if SLOs degrade.

Key trade-offs: dedicating cores increases cost; aggressive prioritization can starve background tasks—use fairness tokens and monitoring.

Common causes (with detection, minimal repro, and safe fixes):

1. Dangling references / global caches

• Detect: rising RSS over time (prometheus, psutil), tracemalloc/objgraph to find retained objects.
• Repro: appends to a global list each request.
• Fix: remove/limit cache size (LRU), use weakref.WeakValueDictionary, explicitly del references and call gc.collect(). Deploy change behind feature flag and roll restart workers.

2. Large tensors retained (PyTorch/TensorFlow)

• Detect: torch.cuda.memory_summary(), nvidia-smi, per-process GPU/CPU metrics; tracemalloc for CPU.
• Repro: accumulate outputs in a list without .detach() or .cpu().
• Fix: detach()/cpu()/numpy() when storing, use with torch.no_grad() during inference, delete tensors and torch.cuda.empty_cache(). Patch and restart worker processes gracefully.

3. Unclosed sessions/graphs (TF1-style)

• Detect: growing graph count / handles, profilers.
• Repro: creating new tf.Session() per request without close().
• Fix: use context managers or reuse sessions; migrate to TF2 eager execution. Hotfix: ensure .close() in finally blocks and restart processes for immediate reclaim.

4. Thread/worker leaks (multiprocessing, DataLoader)

• Detect: increasing process/thread count, zombie workers in ps and metrics.
• Repro: spawn new Process per task without join/terminate.
• Fix: use worker pools, ensure proper shutdown, set max_workers, add healthchecks and rolling restarts.

5. Event listeners / logging handlers retained

• Detect: objgraph shows bound methods; high memory after adding handlers dynamically.
• Repro: add logging handler per request.
• Fix: avoid adding handlers repeatedly; remove handlers on teardown. Patch live if possible, then restart.

6. C-extension / library leaks (numpy, PIL, custom C code)

• Detect: tracemalloc shows native allocations; memory not freed after gc.collect().
• Repro: call library function in loop leaking native memory.
• Fix: upgrade library, limit lifetime by isolating calls in subprocess (process-level isolation) and restart worker periodically.

General safe remediation strategy:

• Add observability (RSS, heap profiles, GPU), set alerts.
• Reproduce locally with stress test and use tracemalloc/objgraph/torch profilers.
• Prefer fixes that avoid immediate downtime (feature flags, worker-by-worker rolling deploy).
• For urgent reclaim, restart affected processes (graceful draining) and deploy patch.