Broad coverage of modern and advanced neural network architectures, design principles, and components. Candidates should understand core structural elements such as neurons, layers, weights, biases, activation functions, forward and backward passes, and how architecture choices influence learning. Know a range of architecture families including feedforward networks, convolutional neural networks, recurrent neural networks including long short term memory and gated recurrent unit variants, transformer architectures with self attention and multi head attention, vision transformer adaptations, and graph neural networks. Understand inductive biases that make certain architectures appropriate for particular data modalities, trade offs between depth and width, parameter efficiency and computational complexity, and practical considerations such as initialization, normalization, optimization, and scaling strategies. Be able to explain when to choose one architecture over another for a given problem, how to combine or adapt architectures for domain specific needs, and how modern architecture advances address limitations of prior models.
HardTechnical
94 practiced
You're given sensor time-series data with noisy labels and severe class imbalance (rare failure events). Propose a robust model architecture that may combine CNNs, TCNs, or transformers, and a training strategy to handle label noise and imbalance. Cover loss choices (focal, robust losses), augmentation strategies (time warping, jittering, mixup for time-series), semi-supervised/self-supervised pretraining, sampling or re-weighting strategies, and how to calibrate probabilistic outputs for monitoring and alerting purposes.
Sample Answer
**Approach summary**I would build a hybrid encoder: a light 1D-CNN front-end for local pattern extraction → stacked TCN blocks (dilated causal convs + residuals) for long-range temporal context → a transformer encoder layer (multi-head attention) to capture global dependencies and uncertainty-aware representations. Head: two outputs — classification (softmax) and an auxiliary contrastive embedding for self-supervised pretraining.**Model & why**- CNN front-end: efficient local feature extraction and noise smoothing.- TCN: stable gradients for long sequences, retains causal structure.- Transformer: flexible global attention to focus on rare precursors to failure.- Auxiliary embedding: enables contrastive SSL and consistency losses.**Training strategy**- Pretrain self-supervised: - Contrastive (SimCLR-style) with augmentations (time warping, jittering, scaling, magnitude warping). - Masked reconstruction (masked forecasting) using TCN+Transformer to learn temporal patterns.- Fine-tune with labelled data using semi-supervised consistency regularization (pseudo-labels above confidence threshold + MixMatch-style mixup for time-series).**Augmentation**- Time warping, random cropping, jittering, scaling, frequency-domain noise, and time-series mixup (interpolate signals and labels) to increase minority variety.**Losses & noise robustness**- Use focal loss (gamma tuned) to emphasize rare failures.- Combine with a robust loss component (Generalized Cross Entropy or symmetric cross-entropy) to mitigate noisy labels.- Label-cleaning: soft-labels via EMA teacher (Mean Teacher) to smooth noisy annotations.- Curriculum loss: start with high-confidence samples (small loss) then incorporate noisier ones.**Sampling / re-weighting**- Class-aware sampler with dynamic oversampling of rare events plus importance weighting by inverse class frequency and sample uncertainty (higher weight to low-uncertainty minority samples).- Use batch composition constraints to keep diverse contexts.**Calibration & monitoring**- Calibrate probabilistic outputs with temperature scaling and isotonic regression on a holdout set.- Estimate uncertainty: predictive entropy from Monte Carlo dropout or deep ensembles for thresholding alerts.- Monitor calibration drift in production with online reliability metrics (Brier score, expected calibration error) and trigger re-calibration when drift exceeds threshold.**Evaluation & metrics**- Use PR-AUC, recall@fixed-precision, F-beta (beta>1), confusion-cost weighted metrics.- Evaluate under label noise via synthetic label flips and holdout-clean set if available.This architecture + multi-stage training balances representation learning, noise robustness, and calibrated uncertainty for reliable failure detection in imbalanced noisy time-series.
EasyTechnical
122 practiced
Explain the differences between Batch Normalization, Layer Normalization, Instance Normalization, and Group Normalization. Describe how normalization statistics are computed for each, where they are typically applied (convs, transformers, RNNs), their dependence on mini-batch size, and inference-time behavior. Recommend strategies for small-batch or streaming production settings where per-batch statistics are unreliable.
Sample Answer
**Overview — purpose**Normalization stabilizes and accelerates training by centering/scaling activations; variants differ in the axes used to compute mean/variance and thus in batch-size sensitivity and typical use-cases.**BatchNorm**- Statistics: mean/var computed per channel across batch + spatial dims (N, H, W).- Typical use: convolutional nets.- Batch-size dependence: sensitive — needs sufficiently large N to get stable stats.- Inference: uses running averages accumulated during training (fixed).- Notes: works well for convs but problematic for small/online batches; synchronized BN across devices can help.**LayerNorm**- Statistics: per-sample mean/var across features (all channels), computed over last dimension(s).- Typical use: Transformers, RNNs.- Batch-size dependence: independent of batch — stable for small/streaming.- Inference: same computation as training (no running averages).**InstanceNorm**- Statistics: per-sample per-channel mean/var across spatial dims (H,W).- Typical use: style transfer, image generation.- Batch-size dependence: independent of batch.- Inference: same as training (or use learned affine params); often effective for tasks requiring instance-level contrast.**GroupNorm**- Statistics: per-sample mean/var computed within groups of channels.- Typical use: convs when batch size small (detection, segmentation).- Batch-size dependence: independent of batch; group size is a hyperparameter.- Inference: same as training.**Recommendations for small-batch / streaming production**- Prefer LayerNorm or GroupNorm (or InstanceNorm where appropriate) which do not rely on batch stats.- Use SyncBatchNorm if you must use BatchNorm across multi-GPU with small per-GPU batch.- For online/streaming, compute per-sample normalization (LayerNorm) or maintain exponential moving averages of statistics with careful decay; validate distribution shift and consider calibration layers or adaptive normalization (e.g., Batch Renorm).- Monitor activation distributions in production and include small fine-tuning on in-domain data if drift appears.
MediumTechnical
78 practiced
Design an architecture that fuses graph neural networks with transformer-style attention to predict molecular properties given both a molecular graph (atoms, bonds) and a short SMILES token sequence. Describe how you would encode nodes and tokens, the order of operations (message passing then cross-attention vs cross-attention then message-passing), positional/structural encodings for nodes, batching considerations, pretraining objectives (contrastive, masked), and a deployment-friendly fusion strategy.
Sample Answer
**Overview**I would build a dual-encoder fusion: a GNN that encodes molecular graph structure and a Transformer that encodes SMILES tokens, with lightweight cross-attention layers to fuse modalities. This balances structural priors and sequence signals while remaining deployment-friendly.**Node & Token Encoding**- Nodes: atom type embedding + formal charge + aromaticity + hybridization (learned embeddings) + scalar features (degree, implicit H) projected to node dim.- Edges: bond type embeddings used in message functions.- Tokens: subtoken (BPE) embeddings + learned token type + small positional embedding for SMILES order.**Positional / Structural Encodings**- Graph structural encodings: Laplacian eigenvectors (dropped-in as continuous node features) + random-walk centrality or distance-to-functional-group scalar; concatenate to node embedding.- SMILES: learned absolute positional embeddings.**Order of Operations**Preferred: GNN message-passing first to produce structure-aware node representations, then cross-attention where SMILES tokens attend to nodes and vice-versa (bi-directional cross-attention). Reason: graph MP injects topological context; cross-attention aligns sequence fragments with substructures. Optionally interleave (MP ↔ cross-attn) for deeper fusion if compute allows.**Architecture Sketch**- K layers: (GNN MP x n_mp) → (Cross-Attention block) → (optional joint Transformer on fused tokens/nodes)- Cross-attention implemented as multi-head attention with sparse attention masks for efficiency.**Pretraining Objectives**- Masked Node/Token Modeling: mask atoms and SMILES subtokens, predict types.- Contrastive: graph vs SMILES representation (global pooled embeddings) with InfoNCE to align modalities.- Edge/Distance prediction auxiliary tasks (predict adjacency or pairwise distances) to strengthen structural learning.**Batching & Efficiency**- Use graph batching (disjoint union) with block-sparse attention; pack variable-length SMILES with padding/mask.- Precompute Laplacian encodings offline.- For cross-attention, limit attended nodes per token via learned top-k or locality-based pruning.**Deployment-Friendly Fusion**- During inference allow a fall-back single-modality mode: if SMILES missing, use GNN-only; if graph missing, use Transformer-only.- Fuse using a small gating MLP over pooled representations to produce final property prediction—keeps runtime small and enables model pruning/quantization.- Export as two submodules (GNN encoder, Transformer encoder) plus lightweight cross-attention/gating to ease CI/CD and A/B rollout.**Evaluation**- Validate with downstream fine-tuning and ablations: order (MP→cross vs cross→MP), pretraining losses, structural encodings. Monitor calibration and OOD robustness (scaffold splits).
EasyTechnical
83 practiced
Explain residual (skip) connections: what they are mathematically, why they enable training of much deeper networks, and how they affect gradient flow and representational capacity. Compare pre-activation vs post-activation residual blocks and give practical guidance on where to place normalization and activation when introducing residuals into a new architecture.
Sample Answer
**What residual (skip) connections are — mathematically**- A residual block learns F(x) and outputs y = x + F(x). In code/math:
text
y = x + F(x; W)
- If a final activation σ is used after addition, it's y = σ(x + F(x)).**Why they enable much deeper networks**- Identity shortcut provides an easy path for information and gradients; the network can learn F(x)=0 to preserve x, avoiding catastrophic degradation as depth increases.- They change optimization from learning H(x) to learning the residual H(x)-x, which is often easier.**Effect on gradient flow and representational capacity**- Gradients through addition: dL/dx = dL/dy * (1 + dF/dx). The “1” from identity prevents vanishing gradients and stabilizes backprop, allowing signals to propagate across many layers.- Representationally, residuals increase expressivity by composing residual functions; identity shortcuts do not reduce capacity because F can represent zero and residual stacks can approximate complex mappings.**Pre-activation vs Post-activation residual blocks**- Post-activation (original ResNet): y = x + σ(BN(W * σ(BN(x)))) then final activation. Works but can hamper gradient because activation sits after addition.- Pre-activation (He et al., 2016): y = x + F(BN, ReLU; x) where BN+ReLU are before convolutions and addition is followed by identity. Empirically improves training, regularization, and gradient flow in very deep nets.**Practical guidance for new architectures**- Put normalization and nonlinearity inside the residual branch (pre-activation style) for very deep models — BN/LayerNorm -> activation -> weight.- Ensure the shortcut is a clean identity if dimensions match; otherwise use a 1x1 conv (with minimal params) to align dims, typically without activation on the shortcut.- If using GroupNorm or LayerNorm (small batches), apply them inside the block similarly.- Avoid heavy operations (dropout/activation) on the shortcut path; keep it as linear identity to preserve gradient “1” term.This framing balances optimization stability and representational flexibility for production-ready deep models.
MediumSystem Design
93 practiced
Sketch a practical plan to train a 2-billion-parameter transformer model from scratch on cloud GPUs/TPUs. Specify distributed parallelism choices (data, tensor, pipeline), optimizer and mixed-precision strategy, gradient accumulation, checkpointing cadence, dataset sharding/IO strategy, and memory optimization techniques (activation checkpointing, ZeRO/optimizer-sharding). Discuss cost/time trade-offs and a rollout plan to move the trained model into production with fine-tuning pipelines.
Sample Answer
**Clarify constraints**- Target: 2B-parameter Transformer, train from scratch; cloud GPUs/TPUs; budget/time trade-offs matter.- Throughput target: reasonable wall-clock (weeks), reproducibility, production-ready for fine-tuning.**Distributed parallelism**- Baseline: Data + Tensor parallelism. Use tensor-parallel (e.g., 2–4-way) to split large layers and keep latency low; data-parallel across many replicas.- If >8 devices per model, add pipeline parallelism (2 stages) to increase per-device memory headroom and improve utilization.- Example: 16 GPUs = tensor-4 × pipeline-2 × data-2.**Optimizer & precision**- Use AdamW with weight decay and LAMB/AdamW hybrid if large-batch scaling needed.- Mixed precision: bf16 on TPUs or AMP (fp16 + dynamic loss scaling) on GPUs to reduce memory and improve speed.**Gradient accumulation & batch**- Use gradient accumulation to emulate large global batch (e.g., local batch 8 × accumulation 32 => global batch 256).- Tune LR schedule (AdamW warmup + cosine decay) corresponding to effective batch.**Checkpointing cadence**- Frequent small checkpoints (every epoch or every N steps like 10k) for safety; keep rolling checkpoint every hour for long jobs and full checkpoints every 4–12 hours. Keep metadata for deterministic resume.**Dataset sharding & IO**- Pre-shard data into many files (TFRecord/Arrow/LMDB). Use streaming with local SSD prefetch and a read-ahead cache.- Shuffle by reservoir/windowed shuffle to avoid global shuffle cost.- Use parallel readers per worker and verify deterministic seed for reproducibility.**Memory optimizations**- Activation checkpointing (recompute) for transformer blocks.- Use ZeRO stage 2/3 (optimizer state / gradient / parameter sharding) to reduce memory per device; combine with tensor parallelism.- Offload optimizer states to CPU or NVMe if GPU memory constrained.**Cost/time trade-offs**- Larger tensor/pipeline splits reduce communication but increase complexity. More data-parallel replicas speed training linearly but increase cost.- Use preemptible/spot instances for non-critical epochs to save cost; maintain frequent checkpoints.- For speed: more high-bandwidth interconnect (A100-HGX / TPU v4) reduces grad-sync overhead but costs more.**Rollout & fine-tuning**- Export model weights + tokenizer + config; run validation suites and small-task probes.- Build fine-tuning pipelines: task-specific adapters or LoRA for fast, low-cost adaptation; maintain reproducible scripts and CI for evaluation.- Deploy via model-serving with quantization (INT8 / 4-bit) + batching; monitor drift and latency; keep retraining schedule or continual fine-tuning.I would prototype with a smaller model (200M) to validate pipeline, then scale with the above parallelism and ZeRO configuration.
Unlock Full Question Bank
Get access to hundreds of Neural Network Architectures interview questions and detailed answers.
