Common Deep Learning Architectures Questions

Familiarity with CNNs for images, RNNs/LSTMs for sequences, attention mechanisms, and Transformers for NLP. Understanding when and why to use each. Basic knowledge of pre-trained models and transfer learning.

MediumTechnical

143 practiced

Compare quantization-aware training (QAT) vs post-training quantization (PTQ) for shrinking Transformers for CPU inference. Discuss trade-offs in accuracy, calibration requirements, per-channel vs per-tensor quantization, expected latency improvements, and when to prefer QAT over PTQ in a production pipeline.

Sample Answer

High-level summary: PTQ is fast and low-effort — you take a trained Transformer, run a calibration dataset or even simple statistics, and quantize weights (and optionally activations). QAT simulates quantization during training so the model learns to be robust to reduced precision; it requires retraining/fine-tuning but usually yields higher accuracy, especially at lower bitwidths.

Accuracy trade-offs:- PTQ: Often preserves accuracy for 8-bit weight-only quantization on many Transformers, but can fail for activation-sensitive layers (LayerNorm, softmax) or aggressive quantization (4-bit). Expect small degradation (~0–3% relative) at 8-bit; larger for <8-bit.- QAT: Typically recovers most accuracy lost at lower precision (4–6 bits) by adapting weights; can match FP32 within noise for many tasks after fine-tuning.

Calibration requirements:- PTQ needs a representative calibration dataset. Quality/coverage matters — domain mismatch causes large accuracy drops. Calibration may require per-layer tuning (clipping ranges, KL-based, MSE).- QAT reduces reliance on calibration because training optimizes for quantized behavior; still benefits from representative training/validation.

Per-channel vs per-tensor:- Per-channel weight quantization (separate scale per out-channel) is preferred for linear layers in Transformers — reduces quantization error for skewed weight distributions and preserves accuracy with minimal compute overhead on CPUs that support per-channel ops (or with fused routines).- Per-tensor is simpler and faster (single scale) and may be adequate for activations; but hurts weights with high dynamic range. For CPU inference, per-channel weight + per-tensor activation is a common sweet spot.

Expected latency improvements:- 8-bit integer GEMM can yield 2×–4× CPU throughput improvements vs FP32 depending on BLAS/kernel quality and memory bandwidth; 4-bit can give larger gains but needs custom kernels (and may require packing/unpacking overhead).- Real-world latency gain depends on operator fusion, memory layout, and whether quantized kernels avoid costly conversions. Weight-only quantization helps memory footprint more than latency unless compute is bottlenecked.

When to prefer QAT in production:- Use PTQ when you want fast deployment, have good calibration data, and target standard 8-bit quantization — low engineering cost and often sufficient.- Choose QAT when PTQ causes unacceptable accuracy drops (tasks with sensitive activations, low-bit targets), when you need aggressive bitwidth reduction, or when you can afford fine-tuning compute. QAT is recommended for long-lived models where investing training cost reduces inference cost and improves robustness across domains.

Practical tips:- Start with PTQ + per-channel weights; evaluate on a holdout representative set. If accuracy loss or tail cases persist, move to QAT with a few epochs of fine-tuning and quantization-aware fake-quant ops. Monitor calibration drift and validate latency on target CPU with real kernels.

MediumTechnical

92 practiced

Implement a multi-head attention module in PyTorch as an nn.Module. Inputs: queries, keys, values shaped (batch, seq_len, d_model), number of heads; include scaled dot-product attention, dropout, and return the projected output with correct shape. Focus on correct shape handling, masking support, and numerical stability.

Sample Answer

Approach: implement MultiHeadAttention as nn.Module that linearly projects Q/K/V, reshapes for heads, computes scaled dot-product attention with optional mask and dropout, then concatenates heads and projects output. Pay attention to shapes: (B, S, D) -> (B, H, S, d_k) for attention and back. Use float32/64 stability via softmax on scaled scores and mask with large negative value.

python

import torch
import torch.nn as nn
import torch.nn.functional as F
import math

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads, dropout=0.1):
        super().__init__()
        assert d_model % num_heads == 0, "d_model must be divisible by num_heads"
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads

        self.w_q = nn.Linear(d_model, d_model)
        self.w_k = nn.Linear(d_model, d_model)
        self.w_v = nn.Linear(d_model, d_model)
        self.out_proj = nn.Linear(d_model, d_model)
        self.dropout = nn.Dropout(dropout)

    def forward(self, query, key, value, mask=None):
        # query/key/value: (B, S, D)
        B, S_q, _ = query.size()
        S_k = key.size(1)

        # Linear projections
        q = self.w_q(query)  # (B, S_q, D)
        k = self.w_k(key)
        v = self.w_v(value)

        # Reshape -> (B, num_heads, S, d_k)
        def reshape(x):
            return x.view(B, -1, self.num_heads, self.d_k).transpose(1,2)
        q = reshape(q)
        k = reshape(k)
        v = reshape(v)

        # Scaled dot-product: (B, H, S_q, S_k)
        scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.d_k)

        # Masking: mask shape can be (B, 1, S_q, S_k) or (B, S_q, S_k) or (S_q, S_k)
        if mask is not None:
            # ensure mask is bool or 0/1
            mask = mask.unsqueeze(1) if mask.dim() == 3 else mask
            scores = scores.masked_fill(mask == 0, float('-1e9'))

        attn = F.softmax(scores, dim=-1)
        attn = self.dropout(attn)

        # Attention output
        out = torch.matmul(attn, v)  # (B, H, S_q, d_k)
        out = out.transpose(1,2).contiguous().view(B, S_q, self.d_model)  # (B, S_q, D)
        out = self.out_proj(out)
        return out

Key points:- Numerical stability: scale by sqrt(d_k) and mask with large negative before softmax.- Mask shapes supported (broadcastable); mask uses 0 for masked positions.- Complexity: O(B * num_heads * S_q * S_k * d_k) time, O(B * num_heads * (S_q+S_k) * d_k) intermediate memory.- Edge cases: ensure d_model % num_heads == 0, handle empty sequences, and use contiguous() after transpose to avoid view errors.

HardTechnical

87 practiced

You're deciding how to scale model performance given limited compute and data: options include increasing depth, width, dataset size, or compute budget (longer training). Using empirical scaling-law thinking, describe how you'd estimate expected returns, plan data acquisition or synthetic data generation, compute budget forecasting, and recommend a practical scaling path for a mid-stage startup constrained by both compute and labeled data.

Sample Answer

Situation: Our startup must improve model performance but is limited in both labeled data and compute. We need a pragmatic, empirical plan that maximizes ROI.

Approach (empirical scaling-law thinking):- Use simple power-law models to estimate returns: validation loss L ≈ L0 + A⋅N_params^-α + B⋅D^-β + C⋅T^-γ where N_params = model size, D = dataset size, T = compute (training FLOPs). Fit α,β,γ using small-scale experiments or literature priors (e.g., α≈0.3–0.5, β≈0.2–0.4).- Run a 3×3 grid of cheap probes (small/medium/deep × small/medium datasets) for 1–2 epochs to estimate gradients of improvement per resource unit (∆metric per 1% compute, per 10% data).

Data plan:- Prioritize labeled data acquisition where marginal gain per dollar is highest: evaluate active learning — label highest-uncertainty samples first to improve sample efficiency.- Use synthetic data only where distribution shift is manageable: fine-tune with domain-adaptive augmentation and validate on a small held-out real set. For structured tasks, use weak supervision/snorkel to multiply labels cheaply.- Set a target data-efficiency metric (e.g., gain in F1 per 1k labels) from probes to decide buy vs synth.

Compute forecasting:- Convert desired training regime into FLOPs using experiments: FLOPs ≈ epochs × dataset_size × params × constant. Use probe scaling slopes to estimate diminishing returns; compute cost = FLOPs / (GPU_FLOPS × utilization) × $/hr.- Forecast 3 scenarios (conservative, moderate, aggressive) with expected metric gains and costs; pick one with best marginal improvement per $.

Recommendation for mid-stage startup:1. Short experiments to estimate scaling exponents α,β,γ.2. Invest first in smarter data (active labeling + weak supervision) until marginal gain per $ equals model-scaling gain.3. If dataset growth plateaus, increase model width moderately rather than depth (width often cheaper to parallelize and tune).4. Use mixed-precision and gradient accumulation to maximize existing GPU utilization; budget for burst cloud training for one larger run when projected gain justifies cost.5. Continuously validate synthetic-data utility and stop adding synthetic labels when validation improvement stalls.

Outcome metric: choose business-aligned KPI (e.g., error reduction per $) and iterate monthly. This yields a measurable, low-risk scaling path balancing compute and data constraints.

HardTechnical

92 practiced

Explain gradient checkpointing (activation checkpointing): describe how it reduces activation memory by trading compute, how to choose checkpoint placement in a Transformer stack, and how it interacts with mixed precision and optimizer sharding. Provide rough estimates of memory savings vs compute overhead for N-layer models.

Sample Answer

Gradient (activation) checkpointing saves GPU memory by not storing all intermediate activations during the forward pass; instead you save (“checkpoint”) a subset of activations, discard the rest, and recompute them during the backward pass. This trades extra forward computation (recomputations) for reduced peak activation memory.

How it reduces memory / trade-off- If you checkpoint K points in an N-layer network, you only store activations at those K points; activations inside blocks between checkpoints are recomputed during backward. Memory ~ O(K·A_block) + O(params), instead of O(N·A_layer).- Compute overhead ≈ (recomputed forward FLOPs) / (original forward FLOPs). With naive single recompute of each non-checkpointed layer you pay ~1× extra forward for those layers (total ~ ~ (1 + (N-K)/N) ≈ 2× if K=1). Practical overheads are often 10–40% for well-chosen checkpointing; worst-case can approach ~2×.

Checkpoint placement in a Transformer stack- Natural unit: transformer layer (attention+FFN). Checkpoint boundaries should align to full layers or groups (e.g., every layer or every M layers).- Trade-offs: fewer checkpoints (larger blocks) → more memory saved, more recompute. Many frameworks choose checkpoint per layer (store inputs to each layer) or group layers into chunks of size g (store every g-th layer). For N-layer model with chunk size g, memory reduces roughly by factor ≈ g, compute overhead ≈ (1 + (g-1)/2) per-block recompute pattern.- Consider residual connections: checkpoint layer inputs before residual to simplify recompute; for fused implementations, ensure deterministic recompute order.

Interaction with mixed precision- Mixed precision (FP16/AMP) lowers activation size, reducing need for checkpointing but also reduces recompute cost per FLOP. Checkpointing still beneficial for very deep models; ensure numerical stability during recompute (use same autocast/scaler context). Keep master copy of parameters in FP32 for optimizer; recompute should use identical casting to forward.

Interaction with optimizer sharding (ZeRO)- Optimizers like ZeRO partition optimizer state across devices, reducing param/optimizer memory but not activation memory. Checkpointing is complementary: use checkpointing to reduce activations, ZeRO to reduce optimizer state. Be mindful: recomputation increases communication if activation recompute requires parameter fetches or all-reduce; place checkpoints so recompute uses local parameters when possible. Also, with stage-3 sharding, ensure checkpoints don't force re-gathering of parameters repeatedly—batch recompute to amortize communication.

Rough numeric estimates (N-layer baseline)- No checkpoint: activation mem ≈ N·A (per-layer activation A).- Checkpoint every layer (store layer inputs): activation mem ≈ A + O(params) → ~1/N of baseline for very large N (practical: ≈ 1/ N reduction for activations).- Chunk size g (store every gth layer): activation mem ≈ (N/g)·A → memory ≈ 1/g of baseline. Compute overhead ≈ ~ (1 + (g-1))? real-world: overhead ≈ (g+1)/2 average extra forward passes per layer in block; so g=2 → ~1.5×, g=4 → ~2.5×.Practical numbers: with g=4 (checkpoint every 4 layers) you often save ~3–4× activation memory with ~1.5–2.5× extra compute; with g=2 you save ~2× memory with ~1.2–1.5× compute overhead.

Best practices- Start with per-layer checkpointing or g=2, measure wall-clock and memory; tune g for target hardware.- Combine checkpointing + mixed precision + ZeRO for max scale: MP reduces activation size, ZeRO reduces optimizer state, checkpointing reduces remaining activation peak.- Profile: recompute cost, communication overhead, and numerical stability; ensure deterministic recompute and correct AMP/scaler usage.

MediumTechnical

71 practiced

Given a pre-trained CNN and a small domain-specific dataset (1,000 images, 5 classes), propose transfer learning strategies: feature extraction, fine-tuning last N layers, and gradual unfreezing. Explain how to pick which layers to train based on domain similarity, how to choose learning rates per layer, and how to validate the approach in production.

Sample Answer

Approach overview:- Feature extraction: freeze backbone, replace and train classifier only — fast, low overfit risk.- Fine-tuning last N layers: unfreeze last N convolutional blocks and the head — balances representation adaptation and stability.- Gradual unfreezing: progressively unfreeze from head → deeper layers over multiple training phases, lowering LR as you unfreeze.

How to pick which layers to train (based on domain similarity):- High similarity (natural images, similar textures/object scale): fine-tune only the classifier or last 1–2 blocks.- Moderate similarity (same modality but different domain-specific cues): unfreeze last 3–4 blocks.- Low similarity (medical, infrared, or very different low-level stats): consider unfreezing more layers or whole model, possibly re-training early conv layers too.- Practical heuristic: inspect early-layer activations/filters and t-SNE of features — if class clusters are separable with frozen features, minimal fine-tuning suffices.

Learning rates per layer:- Use discriminative LRs: highest LR for newly initialized head (e.g., 1e-3), lower for unfrozen deeper layers (e.g., 1e-4 to 1e-6), decreasing by factor 2–10 per earlier block.- For gradual unfreezing, start with optimizer warm-up on head then lower LR when unfreezing to avoid catastrophic forgetting.- Use AdamW or SGD with momentum and weight decay; consider layer-wise LR scheduling (e.g., cosine annealing).

Validation and production readiness:- Use stratified k-fold (e.g., 5-fold) and hold-out test set to estimate generalization; with 1,000 images, prefer repeated stratified CV.- Monitor calibration (ECE), per-class precision/recall, and confusion matrices to catch class imbalance issues.- Run robustness checks: augmentations, domain shifts (lighting, scale), and adversarial/noise tests.- Tracking and deployment: store best checkpoints, use model cards with training data stats and known failure modes.- In production, implement A/B testing, shadow mode inference, drift monitoring (input distribution, prediction confidence), and automated rollback if metrics degrade.

Unlock Full Question Bank

Get access to hundreds of Common Deep Learning Architectures interview questions and detailed answers.

Join thousands of developers preparing for their dream job.