Neural Network Architectures Questions

Broad coverage of modern and advanced neural network architectures, design principles, and components. Candidates should understand core structural elements such as neurons, layers, weights, biases, activation functions, forward and backward passes, and how architecture choices influence learning. Know a range of architecture families including feedforward networks, convolutional neural networks, recurrent neural networks including long short term memory and gated recurrent unit variants, transformer architectures with self attention and multi head attention, vision transformer adaptations, and graph neural networks. Understand inductive biases that make certain architectures appropriate for particular data modalities, trade offs between depth and width, parameter efficiency and computational complexity, and practical considerations such as initialization, normalization, optimization, and scaling strategies. Be able to explain when to choose one architecture over another for a given problem, how to combine or adapt architectures for domain specific needs, and how modern architecture advances address limitations of prior models.

MediumTechnical

87 practiced

Explain the message passing framework used by graph neural networks. Describe the main components: message function, aggregation, update, and readout. Give examples of graph tasks where GNNs excel and pitfalls like over-smoothing or scalability issues.

Sample Answer

The message-passing framework formalizes how Graph Neural Networks (GNNs) propagate and combine information across a graph. At each layer t, every node v updates its hidden state h_v^t by receiving messages from neighbors and combining them.

Main components:- Message function (m): defines what is sent from neighbor u to v, often m(h_u^{t-1}, h_v^{t-1}, e_{uv}) — e.g., a linear transform of neighbor features, or attention-weighted features (GAT).- Aggregation (AGG): permutation-invariant reduction of incoming messages: sum, mean, max, or learned pooling/attention. Aggregation prevents order sensitivity and controls how neighborhood information is compressed.- Update function (U): combines aggregated message with the node’s previous state, e.g., U(h_v^{t-1}, agg) = MLP([h_v^{t-1}, agg]) or a GRU/residual step (GCN uses normalized sum + linear + nonlinearity).- Readout: global or task-specific pooling to produce graph-level outputs (sum/mean, set2set, global attention) or just use node states for node-level tasks.

Example tasks where GNNs excel:- Node classification (social networks, citation networks)- Link prediction (recommendation, knowledge graphs)- Graph classification / regression (molecular property prediction, protein interaction)- Traffic forecasting (spatio-temporal graphs)

Pitfalls and limitations:- Over-smoothing: repeated aggregation makes node embeddings converge, losing discriminative power for deep GNNs. Mitigations: residual/jump connections, normalization, limiting depth.- Oversquashing: long-range information gets compressed into fixed-size vectors causing information loss. Mitigations: increasing capacity per edge (attention), hierarchical/graph-coarsening methods.- Expressivity: many GNNs match 1-WL test — fail to distinguish some graph structures; use higher-order or subgraph-based models for more power.- Scalability: full-batch message passing is memory/time heavy on large graphs. Solutions: neighborhood sampling (GraphSAGE), mini-batching, graph partitioning (Cluster-GCN), sparse kernels, and distributed training.

Design choices (aggregation, update, depth) should match task locality, graph size, and required expressivity.

MediumTechnical

96 practiced

Implement a 2D convolution forward pass in Python using numpy and the im2col approach for a single input batch and multi-channel input. Your function conv_forward(x, w, stride, padding) should accept x shape (N, C, H, W) and w shape (F, C, HH, WW) and return the output tensor. Aim for clarity and correctness rather than optimizing for speed.

Sample Answer

Approach: Use im2col to convert each input patch into columns, reshape filters to rows, then matrix-multiply to compute all convolutions for the batch. This implementation favors clarity and correctness.

python

import numpy as np

def im2col(x, HH, WW, stride, pad):
    # x: (N, C, H, W)
    N, C, H, W = x.shape
    H_p = H + 2*pad
    W_p = W + 2*pad
    out_h = (H_p - HH) // stride + 1
    out_w = (W_p - WW) // stride + 1

    x_padded = np.pad(x, ((0,0),(0,0),(pad,pad),(pad,pad)), mode='constant')
    cols = np.zeros((N, C, HH, WW, out_h, out_w), dtype=x.dtype)

    for y in range(HH):
        y_max = y + stride*out_h
        for x_ in range(WW):
            x_max = x_ + stride*out_w
            cols[:, :, y, x_, :, :] = x_padded[:, :, y:y_max:stride, x_:x_max:stride]

    cols = cols.transpose(0,4,5,1,2,3).reshape(N*out_h*out_w, -1)  # (N*out_h*out_w, C*HH*WW)
    return cols, out_h, out_w

def conv_forward(x, w, stride=1, padding=0):
    """
    x: (N, C, H, W)
    w: (F, C, HH, WW)
    returns out: (N, F, out_h, out_w)
    """
    N, C, H, W = x.shape
    F, Cw, HH, WW = w.shape
    assert C == Cw, "Input channels must match filter channels"

    cols, out_h, out_w = im2col(x, HH, WW, stride, padding)  # (N*out_h*out_w, C*HH*WW)
    w_row = w.reshape(F, -1)  # (F, C*HH*WW)

    # compute (N*out_h*out_w, F) then reshape
    out_cols = cols.dot(w_row.T)  # (N*out_h*out_w, F)
    out = out_cols.reshape(N, out_h, out_w, F).transpose(0,3,1,2)  # (N, F, out_h, out_w)
    return out

Key points:- im2col extracts all sliding patches; matrix multiply replaces nested loops.- Time: O(N * F * out_h * out_w * C * HH * WW) — same as naive but uses BLAS-friendly ops.- Space: O(N * out_h * out_w * C * HH * WW) for the im2col matrix.Edge cases:- Check stride/padding produce integer output dims.- Handles multi-channel and batch >1.Alternatives:- Direct nested loops for simplicity; use FFT-based conv for large kernels.

HardTechnical

73 practiced

You are training a 1.5 billion parameter transformer model. Discuss strategies to manage GPU memory usage during training, including mixed precision, gradient checkpointing, ZeRO optimizer sharding, activation offloading, and data parallelism vs model parallelism. Include trade-offs in throughput and engineering complexity.

Sample Answer

Start by defining the goal: fit a 1.5B-parameter Transformer on available GPUs while keeping training throughput acceptable and engineering complexity manageable. Below are practical strategies, how they save memory, and trade-offs.

Mixed precision (AMP, FP16/BF16)- What: store weights/activations in FP16/BF16, keep master copy of weights in FP32 for optimizer updates.- Benefit: ~2× reduction in activation/parameter memory and faster tensor cores.- Trade-offs: possible numerical instability (loss scaling required), small engineering overhead but widely supported (PyTorch AMP, NVIDIA Apex).

Gradient checkpointing (activation recomputation)- What: store fewer activations; recompute during backward.- Benefit: large activation memory reduction proportional to depth; useful when activations dominate.- Trade-offs: increases backward compute overhead (reduces throughput), implementation easy with libraries but debugging can be harder.

ZeRO optimizer sharding (DeepSpeed)- What: shard optimizer states, gradients, and optionally parameters across data-parallel ranks (ZeRO stages 1–3).- Benefit: drastically reduces per-GPU memory for optimizer states (can enable training much larger models).- Trade-offs: communication overhead (all-reduce/resharding), complexity in setup; ZeRO-3 replicates less but requires more communication and careful IO.

Activation offloading- What: move infrequently used activations/optimizer states to CPU/NVMe.- Benefit: frees GPU memory when GPU memory is the bottleneck.- Trade-offs: big throughput hit due to PCIe/NVMe latency; suitable when GPU memory is too small but compute capacity is high.

Data parallelism vs model parallelism- Data parallelism: simple, replicates model; combined with ZeRO reduces memory cost. High scalability for batch-parallel workloads but per-GPU memory still required for full model replica (unless ZeRO).- Model parallelism (tensor + pipeline): splits model across GPUs (Megatron-LM tensor parallel, pipeline parallel). Enables larger single-model memory footprint without sharding optimizer states.- Trade-offs: model parallelism reduces memory per GPU but adds inter-GPU latency and complex scheduling; pipeline introduces micro-batching and bubble inefficiency.

Practical recipe for 1.5B:- Start with BF16 + AMP + ZeRO stage 1 or 2 to get immediate wins.- Add gradient checkpointing if activations dominate.- Use ZeRO-3 or combine ZeRO + tensor parallelism only if needed for further memory reduction; prefer DeepSpeed or FairScale to avoid reimplementing sharding.- Use activation offloading as last resort.

Throughput vs complexity summary:- Mixed precision: high throughput gain, low complexity.- Checkpointing: moderate complexity, lowers throughput.- ZeRO: high memory reduction, medium–high complexity, communication overhead can lower throughput if network is slow.- Model parallelism: high complexity, can hurt throughput unless carefully tuned (overlap comm/comp).- Activation offload: highest memory relief, largest throughput penalty.

Measure and iterate: profile memory/compute (nvprof/torch.cuda.memory_stats), tune batch size, use overlap (comm/comp), and choose the minimal combination that fits hardware constraints.

HardTechnical

81 practiced

Propose a set of metrics and automated interventions to detect and mitigate unstable training runs for deep neural networks, including exploding losses, gradient anomalies, and large generalization gaps. Include examples of programmatic actions to take when thresholds are exceeded.

Sample Answer

Framework:- Detect → Diagnose → Mitigate → Log & Rollback. Monitor per-step, per-epoch and validation checkpoints.

Key metrics to track (real-time):- Training loss trend and smoothed derivative (ema_loss, dloss/dt)- Gradient norm stats: L2 norm, max absolute, per-layer median and skew- Gradient variance across micro-batches and replicas- Weight update magnitude (||Δw|| / ||w||)- Validation loss & accuracy; train/val gap and gap trend- Learning rate and LR scheduler stage- Batch statistics: activation means/std, NaN counts, % clipped gradients- Hardware/throughput anomalies: step time spikes, OOM events

Automated thresholds & detectors:- Exploding loss: if ema_loss increases > 3× baseline over 5 steps or dloss/dt positive and > threshold -> trigger- Gradient explosion: gradient L2 > 10× historical median OR any NaN/Inf -> trigger- Vanishing gradients: gradient L2 < small_eps for many layers consistently -> trigger- High generalization gap: val_loss - train_loss > gap_thresh (e.g., 0.2 relative) and gap increasing over 3 epochs -> trigger

Programmatic interventions (ranked, from lowest-risk to stronger):1) Soft interventions (fast, reversible)- Reduce LR by factor (0.5), or apply cosine/linear warmdown- Apply gradient clipping (global norm 1.0) and/or per-layer clipping- Increase weight decay slightly; enable/raise dropout- Switch optimizer to more robust variant (SGD with momentum ↔ AdamW) via snapshot fine-tune- Recompute batchnorm stats (run a few eval steps)Example action (python-like):

python

if grad_norm > GRAD_EXPLODE_THRESH:
    optimizer.param_groups[0]['lr'] *= 0.5
    enable_grad_clip(max_norm=1.0)
    checkpoint(save_reason='grad_explode_autoscale')

2) Medium interventions (requires short pause)- Rollback to last clean checkpoint and resume with lower LR / smaller batch- Switch to mixed precision safe mode (disable AMP) or enable loss scaling- Increase micro-batch count (accumulate gradients) to reduce variance3) Strong interventions (manual review recommended)- Halt training, alert on-call with diagnostics (plots: loss, grad histograms, activations)- Re-run data integrity checks (label leakage, corrupted entries), random seed resample- Auto-launch hyperparameter sweep (LR, batch size, optimizer) using pre-defined safe grid and resume on best config

Diagnostics to log with every trigger:- Top-k layers by gradient magnitude change, activation histograms, recent checkpoints, random sample inputs that caused high loss, GPU memory profile, LR schedule state.

Alerting & automation policies:- Auto-mitigate for transient anomalies (< N triggers/hour); escalate if persistent.- Maintain a "safe mode" that chains soft interventions automatically and creates an incident if not resolved after 3 attempts.- Record interventions in experiment metadata and tag runs for later analysis (root cause & improvements).

Expected outcomes:- Faster automated recovery from common instabilities (exploding gradients, LR spikes), fewer wasted GPU-hours, clearer audit trail for repeated failures enabling long-term fixes like curriculum changes or model reparametrization (LayerNorm, residual scaling).

HardTechnical

82 practiced

Explain the mixture-of-experts (MoE) architecture used to scale model capacity sparsely. Describe gating mechanisms, load balancing, training instabilities, and practical deployment challenges. Under what production scenarios is MoE most beneficial?

Sample Answer

Mixture-of-Experts (MoE) is a conditional‑compute architecture that scales model capacity sparsely: only a subset of "expert" sub-networks (large feed-forward blocks or small networks) are activated per token or example, while a lightweight gating network routes inputs to experts. This yields much larger parameter counts with comparable FLOPs and inference cost.

Gating mechanisms- Soft vs. hard gating: soft (weighted sum across experts) is smooth but multiplies compute; hard/top-k selects k experts (e.g., k=1 or 2) for efficiency. - Typical design: a small linear projection produces logits over experts; apply softmax + top-k selection (Switch, GShard). Noisy gating (add learned noise) helps exploration early in training. - Routing choices: token-level vs. sequence-level, deterministic hashing (routing-by-hash) vs. learned routing.

Load balancing- Problem: expert starvation (some experts get most tokens), causing poor utilization and generalization. - Solutions: auxiliary load-balancing loss (encourages uniform expert probability), capacity factors per expert (limit tokens routed per batch), expert dropout, periodic reinitialization, and using routing softmax temperature scheduling. GShard and Switch use combined importance and load losses.

Training instabilities- Collapse modes: gating collapse (everyone routed to a few experts), exploding gradients from imbalanced updates, and high variance due to nondifferentiable hard routing. - Mitigations: auxiliary losses, gradient clipping, per-expert learning rate scaling, larger batches to stabilize statistics, and noisy/top-k annealing. For hard routes, implement straight‑through or surrogate gradients carefully; many systems use soft probabilities during backward pass and hard selection forward.

Practical deployment challenges- Communication overhead: experts often sharded across devices → heavy all-to-all collectives during each MoE layer; bandwidth and latency become bottlenecks. - Batching & throughput: to utilize experts efficiently you need large micro-batches and dynamic batching; real-time low-latency serving (small batch, unpredictable routing) hurts throughput. - Memory & checkpointing: storing many expert parameters and optimizer states increases memory; requires activation/optimizer sharding (e.g., ZeRO), or expert offloading. - Determinism & debugging: routing nondeterminism complicates reproducibility and debugging. - Autoscaling & resource fragmentation: uneven per-expert load complicates autoscaling; placement and topology-aware scheduling (co-locating heavy experts) help. - Security/robustness: learned routing can expose rare experts to spurious inputs—monitor for drift.

When MoE is most beneficial- Large-scale pretraining where parameter count matters (e.g., very large language models) and batch sizes are large enough to amortize routing/communication. - Use cases tolerant of higher tail latency or that can use batched offline inference (retrieval, ranking, recommendation, batch generation). - Multilingual or multi-domain models where experts can specialize (language-specific experts). - Scenarios prioritizing parameter efficiency vs. FLOPs: you want more representational capacity without proportional compute cost.

Trade-offs and best practices- Use top-k hard routing (k=1 or 2) for efficiency; add auxiliary load loss and capacity limits. - Design system stack for efficient all-to-all, large micro-batches, and optimizer sharding. - Evaluate end-to-end cost: MoE reduces FLOPs but increases complexity, memory, and network IO—worthwhile when you can provision the system and your workload matches batched throughput patterns.

In short: MoE offers a compelling way to massively increase model capacity with conditional compute, but it demands careful gating design, load-balancing regularization, stability techniques, and significant engineering for distributed deployment. It's best when training/serving patterns allow batching and when specialization (multilingual, multi-domain) yields outsized gains.

Unlock Full Question Bank

Get access to hundreds of Neural Network Architectures interview questions and detailed answers.

Join thousands of developers preparing for their dream job.