Neural Network Architectures: Recurrent & Sequence Models Questions
Comprehensive understanding of RNNs, LSTMs, GRUs, and Transformer architectures for sequential data. Understand the motivation for each (vanishing gradient problem, LSTM gates), attention mechanisms, self-attention, and multi-head attention. Know applications in NLP, time series, and other domains. Discuss Transformers in detail—they've revolutionized NLP and are crucial for generative AI.
MediumTechnical
17 practiced
Explain prompt engineering considerations for few-shot/zero-shot generation with large pretrained Transformers. Discuss prompt formatting, context window limits, demonstration selection, and risks like prompt injection or hallucination in production.
Sample Answer
Few-shot and zero-shot prompt engineering with large pretrained Transformers is about shaping model behavior within hard constraints (context window, non-determinism) while managing safety and reliability.Prompt formatting- Be explicit and structured: instructions, constraints, and output format (e.g., "Return JSON with keys: id, summary, confidence").- Use separators (newline, "###", or triple dashes) to reduce ambiguity between instruction, context, and examples.- Normalize style (temperatures, tokens) and include negative examples when helpful.Context window limits- Count tokens for instructions + demonstrations + context. Keep core instruction compact; move long context to retrieval or chunking pipelines.- Use retrieval-augmented generation (RAG) to feed only top-k relevant passages; summarize long context before prompting.- For long chains, implement sliding windows or iterative prompts with state tracking.Demonstration selection (few-shot)- Choose diverse, representative examples covering edge cases and common failure modes.- Prefer 3–8 high-quality examples; order matters—start with canonical positive examples, then variations.- Use in-context calibration: include examples with the exact output format you expect; avoid contradictory examples.Risks and mitigations- Hallucination: mitigate with grounding (cite sources via RAG), ask for provenance, and validate outputs against knowledge bases or rules.- Prompt injection: treat all external inputs as untrusted; separate user content from system instruction, canonicalize inputs, and enforce instruction precedence. Use model policies and post-hoc filters to detect injected directives.- Exposure of sensitive data: redact or token-map private fields before prompting.- Non-determinism & safety: set temperature appropriately; add guardrails (reject/flag uncertain answers), and run safety classifiers.Operational best practices- Monitor outputs, log prompts + responses for auditing, run adversarial prompt tests, and implement automated validators (schema checks, fact-checkers).- Where high assurance needed, combine model outputs with deterministic downstream logic and human-in-the-loop verification.This combination—clear formatting, mindful context budgeting, careful demo choice, and layered mitigations—yields reliable few/zero-shot behavior suitable for production.
HardSystem Design
25 practiced
Design an inference serving architecture for a Transformer-based chatbot that must handle 10k concurrent requests with average latency <300ms per request. Address model sharding, batching, GPU utilization, autoscaling, and cost trade-offs. Include how you'd support both short and long context conversations.
Sample Answer
Requirements & constraints:- 10k concurrent requests, P50 latency <300ms, support short (<=1k toks) and long (>=16k toks) contexts, cost-aware, high GPU utilization, reliable autoscaling.High-level architecture:- Ingress LB → Request router (stateless) → Frontend workers (CPU) → Inference tier (GPUs) → Context store + async long-context service → Response aggregator → Client.Key components & strategy:1. Model sharding: use tensor/model parallelism (e.g., Megatron-LM-style) across 4–8 GPUs per replica for a single large Transformer. Keep multiple replica groups for throughput. For extremely large models, pipeline parallelism + tensor parallelism reduces per-GPU memory.2. Batching & dynamic micro-batching: Frontend workers collect requests per GPU worker into adaptive micro-batches (max latency budget ~50–100ms), then fuse into larger batches on GPU using optimized kernels (FlashAttention, cuBLASLt). Use dynamic batching with priority for low-latency short requests.3. GPU utilization: Maintain target occupancy by combining synchronous large-batch throughput and low-latency micro-batches. Use mixed precision (AMP/FP16) and operator fusion to reduce memory and compute.4. Autoscaling: Two-layer autoscale: - Fast autoscale based on request queue length and tail latency (scale out GPU workers). - Slow autoscale for model replica counts based on sustained QPS and cost. Use warm pool of standby GPU replicas to avoid cold start latency.5. Short vs long context: - Short: full decode on primary GPU path with batching and streaming tokens. - Long: keep extended context in a vectorized context store (Redis/FAISS) with retrieval-augmented generation: summarize or chunk long context into key-value caches, fetch only relevant chunks per request. For truly long context decoding (e.g., full 64k tokens), shard KV cache across GPUs and use KV caching to avoid recomputing past keys.6. Cost trade-offs: - Fewer large GPUs (A100/ H100) with bigger model-parallel groups reduces inter-replica overhead but increases tail latency risk; more smaller GPUs increases flexibility but higher cross-GPU comms. - Use spot/ preemptible instances for non-critical capacity + reserved instances for baseline. - Offload embeddings / retrieval and light pre/post-processing to CPUs to save GPU cycles.Data flow & optimizations:- Router assigns to replica based on model version, current batch size, and whether request needs long-context path.- KV cache persisted per session; immutable past keys stored in SSD-backed object store and hot keys cached in GPU memory.- Monitor latency heatmaps, GPU utilization, batch size distribution; tune batching latency budget and warm pool size.Fault tolerance & SLOs:- Graceful degradation: route to smaller distilled model under heavy load; return partial results with streaming token fallback.- Circuit-breakers + backpressure to keep P50 <300ms.This design balances throughput and latency by combining sharded model execution, adaptive batching, KV caching for long context, proactive autoscaling and cost-aware instance mixes.
MediumTechnical
20 practiced
A time-series forecasting Transformer is giving biased predictions during holiday seasons that are underrepresented in training. Propose concrete model and data interventions to correct seasonality bias, including augmentation, feature engineering, loss adjustments, and evaluation strategies.
Sample Answer
Approach summary: treat holiday-season error as a class-imbalance / distribution-shift problem. Apply data-level fixes (augment/upsample, simulated holidays), feature engineering (explicit event encodings, cyclical/time embeddings), model-level changes (event-aware embeddings, attention priors, specialized output/loss), training strategies (fine-tune on holiday samples, domain adaptation), and robust evaluation (holiday holdouts and calibration).Concrete interventions1) Data augmentation / sampling- Oversample holiday windows (repeat or SMOTE-like interpolation on windows).- Synthetic augmentation: add realistic holiday demand spikes using bootstrapped residuals or generative models (GAN/TimeVAE) conditioned on holiday features.- Create paired windows: for each normal window, generate a "holidayized" version by adding typical holiday uplift pattern.2) Feature engineering- Binary holiday indicator + holiday type (categorical): encode with learned embedding.- Relative-day features: days-to-holiday, days-since-holiday (continuous).- Cyclical encodings: sin/cos for day-of-year, day-of-week, week-of-year.- External covariates: promotion flags, weather, region-specific calendars.- Event embedding: learn embeddings for holiday clusters (e.g., major vs local).3) Model interventions- Time-aware Transformer: add learned event embedding concatenated to token embeddings so attention can condition on event type.- Hierarchical approach: coarse Transformer predicts baseline seasonality; fine Transformer models holiday residuals (additive).- Probabilistic outputs: predict quantiles or parametric distribution (Gaussian/Pareto/Negative Binomial) to capture spike uncertainty.- Attention prior: bias attention weights to emphasize recent similar holidays (use cross-attention to holiday templates).4) Loss & training- Weighted loss: increase weight for holiday timestamps to prioritize holiday fit.- Example PyTorch weighted MSE:- Use quantile loss (pinball) to better capture asymmetric costs (underprediction more costly).- Focal/regret-aware loss: emphasize large underpredictions by scaling loss by error magnitude.- Two-stage training: pretrain on full data, then fine-tune on an oversampled holiday cohort with lower LR.5) Evaluation & monitoring- Create a holiday holdout set (leave-one-holiday-out and across years) to test generalization.- Stratify metrics: report RMSE/MAE/CRPS separately for holiday vs non-holiday, and by holiday type.- Calibration checks: prediction intervals coverage, reliability diagrams for quantiles.- Backtest business metrics: stockouts, lost revenue simulated by underprediction thresholds.- Alerting: monitor holiday-specific bias drift post-deploy and trigger re-training when bias > threshold.Trade-offs and notes- Oversampling helps bias but can overfit; mitigate with augmentation and validation on unseen holiday years.- Weighted loss improves holiday fit but may degrade average performance — tune weights and use multi-objective validation (holiday and global metrics).- Probabilistic outputs increase complexity but provide better risk control.This combined strategy—explicit holiday features, targeted augmentation, event-aware model design, loss emphasis, and focused evaluation—reduces holiday-season bias while tracking trade-offs between holiday accuracy and overall performance.
python
import torch
def weighted_mse(pred, target, is_holiday, w=5.0):
base = (pred - target)**2
weights = torch.where(is_holiday, w, 1.0)
return (base * weights).mean()HardTechnical
20 practiced
You're observing training instability when fine-tuning a large Transformer: loss spikes and NaNs occasionally. Outline a step-by-step debugging plan to find and fix the issue. Include checks for data, optimization, numerical stability, mixed precision, and scheduler settings.
Sample Answer
Start with a hypothesis-driven, reproducible checklist so you can isolate the cause quickly. Work top-down: reproduce, log, isolate (data vs. model vs. optimizer vs. hardware), fix, verify.1) Reproduce & logging- Fix seed, single-GPU run, deterministic flags, reduced dataset + tiny model to trigger failure fast.- Add aggressive instrumentation: loss, grad norms, weight norms, lr, batch stats per step. Check for NaNs/infs each step:2) Data checks- Validate tokenization, decoding, label alignment; ensure no extreme values, empty sequences, or very-long padding.- Check input tensors for NaN/inf and extreme magnitudes before forward pass.- Verify batch composition (class imbalance, degenerate targets) and that masking/attention masks are correct.3) Numerical stability & initialization- Inspect weight initialization and any custom layers. Ensure LayerNorm uses fp32 parameters if using fp16.- Replace any unstable ops (log, divide, sqrt) with safe variants (e.g., clamp denominators).- If using Softmax with large logits, confirm logits aren’t exploding; apply temperature/clip if needed.4) Mixed precision (AMP)- If using fp16/AMP, run one experiment in full fp32 to see if instability disappears.- Use torch.cuda.amp.GradScaler with default dynamic scaling:- If NaNs persist, try static loss scaling, or disable autocast for suspect layers (LayerNorm, softmax).5) Optimizer & LR- Check learning rate magnitude vs. pretrained checkpoint (often need much smaller LR for large Transformers, e.g., 1e-5–5e-5). Temporarily reduce LR by 10x.- Disable adaptive optimizers (AdamW) momentarily and try SGD or Adam with betas tuned. Inspect optimizer state for infs.- Verify weight_decay application only to weights, not to LayerNorm/bias.6) Gradient issues- Log gradient norms and per-layer spikes. Use gradient clipping (global norm ~1.0–5.0) as first mitigation.- Check for exploding gradients due to loss scale; unscale before clipping when using GradScaler.7) Scheduler & warmup- Ensure scheduler warmup exists and is long enough; sudden jump in LR can cause spikes. For fine-tuning, prefer linear warmup over 100–1000 steps depending on dataset.- Plot lr schedule vs steps to confirm monotonicity.8) Checkpoint & state compatibility- If loading pretrained weights, ensure exact architecture match and consistent parameter dtype. Mismatch can silently corrupt states (e.g., different LayerNorm epsilon).- Try reinitializing optimizer state when loading weights.9) Hardware/drivers- Run on different GPU or CPU-only run to rule out hardware issues. Update CUDA/cuDNN/drivers.10) Isolation strategy- Binary search: disable mixed precision, set tiny lr, disable scheduler, check. Re-enable one feature at a time.- When fixed, run longer tests, verify no regression, and add monitored assertions.Common fixes: reduce LR, enable/adjust warmup, use GradScaler and unscale before clipping, clip grads, fix data preprocessing bugs, clamp numerical ops, exclude LayerNorm from fp16, reinit optimizer on checkpoint load.
python
if torch.isnan(loss) or torch.isinf(loss):
torch.save(model.state_dict(), "nan_model.pt"); raise RuntimeError("NaN loss")python
scaler = torch.cuda.amp.GradScaler()
with torch.cuda.amp.autocast():
out = model(x); loss = loss_fn(out, y)
scaler.scale(loss).backward()
scaler.unscale_(optimizer)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm)
scaler.step(optimizer); scaler.update()MediumSystem Design
18 practiced
Design a sequence-to-sequence encoder-decoder architecture for machine translation using RNNs with attention. Specify encoder type, decoder type, attention variant (global/local/additive/multiplicative), training loss, and how you would handle variable-length sequences and batching for production training.
Sample Answer
Requirements & constraints:- Bidirectional encoder, target-language autoregressive decoder, low-latency for production, support variable-length sequences and efficient GPU batching.High-level architecture:- Encoder: 2–4 layer bidirectional LSTM (or GRU) with layer norm and dropout. Outputs per-time-step hidden states H = {h1..hT}.- Decoder: 2–4 layer unidirectional LSTM with input-feeding (previous context concatenated to input) and teacher forcing during training.- Attention: Global additive (Bahdanau) attention for good alignment and gradient behavior; optionally scaled dot-product (multiplicative) if speed/memory matters.Training details:- Loss: Cross-entropy (token-level) with label smoothing (0.1) and optional coverage penalty to reduce repetition.- Optimization: AdamW with learning-rate schedule (warmup + inverse sqrt or cosine decay). Gradient clipping (e.g., 1.0).Variable-length & batching:- Use tokenized sequences with BOS/EOS tokens.- Pad to batch max length and maintain attention/mask tensors to ignore padded positions in encoder/decoder and loss computation.- Bucketing: group examples by similar lengths to reduce padding waste.- Dynamic batching: accumulate micro-batches to reach target tokens-per-batch (e.g., 32k tokens) and then update — improves GPU utilization.- Packed sequences (e.g., PyTorch PackedSequence) in encoder to speed RNNs; or mask-aware implementations.- For inference: use batching with beam search; manage variable lengths by tracking finished hypotheses and masking.Production considerations:- Quantize model (fp16/inference engine) and export via ONNX/TorchScript.- Cache encoder keys/values for streaming or incremental decoding.- Monitor latency/throughput; tune beam width and batching window; fallback to greedy for low-latency path.Trade-offs:- Additive attention = better for alignment on small models; multiplicative scales better for large dims and is faster.- RNNs simpler than Transformers for smaller data; Transformers may be preferred at scale.
Unlock Full Question Bank
Get access to hundreds of Neural Network Architectures: Recurrent & Sequence Models interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.