Neural Network Fundamentals Questions

Core concepts and building blocks of artificial neural networks including neuron models, network layers and layer types (dense fully connected, convolutional, recurrent, and transformer blocks), activation functions and their properties (for example ReLU, sigmoid, tanh, GELU), forward propagation, loss functions, and the basics of optimization such as gradient descent. Explain backpropagation conceptually and mathematically using the chain rule, describe forward and backward passes, and discuss common practical issues such as vanishing and exploding gradients, weight initialization, regularization techniques, and when specific architectures are appropriate for particular problem domains (for example convolutional networks for vision and sequence models for temporal data).

EasyTechnical

73 practiced

Describe the mathematical model of a single artificial neuron (perceptron). Include the role of weights, bias, the linear combination, and the activation function. Provide the formal equation, explain how the neuron computes its output and how it implements a decision boundary for binary classification, and state the perceptron's limitation with non-linearly-separable data.

EasyTechnical

72 practiced

Explain backpropagation conceptually using the chain rule. Describe the forward pass and backward pass, what intermediate values must be cached during forward propagation, and how gradients for weights and biases are computed layer by layer. Provide a short scalar example (for instance y = (wx + b)^2) and compute derivatives manually to illustrate.

HardSystem Design

126 practiced

Design a production inference architecture for transformer-based LLMs that supports both low-latency single-request serving (~50 ms tail latency for short contexts) and high-throughput batched inference for offline workloads. Discuss model sharding, KV-cache strategies, mixed-precision, quantization, batching heuristics, caching hot responses, autoscaling, and orchestration between CPU and GPU resources.

Sample Answer

Requirements & constraints:- 50 ms tail latency for short contexts (<=128 tokens) for single-request interactive traffic.- High-throughput batched offline workloads (maximize GPU utilization).- Support large transformer LLMs that may not fit on a single GPU.

High-level architecture:- Front-end API + routing layer -> Inference gateway that chooses low-latency path or batch path -> Fleet of inference workers (GPU-backed) + CPU workers for preprocessing/postprocessing -> Shared metadata store (Redis) for hot-response & semantic cache -> Orchestrator (Kubernetes + custom autoscaler).

Model sharding:- Use 2-level sharding: tensor/model-parallel (Megatron-style) to split large layers across GPUs and pipeline parallelism for sequence-parallel throughput. For latency-critical short contexts, prefer fewer pipeline stages and replicate smaller model-shard groups (shard replicas) to reduce inter-GPU hops.- Keep shard placement topology-aware (NVLink / same node first) to minimize inter-node PCIe.

KV-cache strategies:- Store KV-cache in GPU device memory for active sessions to get lowest latency.- For many concurrent sessions, use a tiered strategy: GPU-resident hot KV caches, host pinned-memory warm caches, and SSD-backed cold spill using memory-mapped files.- Implement LRU + TTL eviction and session prioritization; prefetch KV-cache for expected follow-ups.- For batched offline runs, discard per-request KV caching and use full sequence batching to avoid state overhead.

Mixed-precision & quantization:- Run core matmuls in FP16/BF16 with AMP; enable Tensor Cores.- Apply post-training quantization: 8-bit (INT8) or quant-aware techniques (GPTQ, LLM.int8) for weight-only quantization; keep attention/LayerNorm in FP16/BF16 to preserve stability.- Validate accuracy vs latency trade-offs per model; maintain two builds: high-accuracy (FP16/INT8 hybrid) and ultra-low-latency (aggressive quantization + distillation).

Batching heuristics:- Dynamic batching with latency budgets: gateway tags requests with SLO; low-latency requests routed to micro-batches (size 1–4) with max wait ~5–10 ms; throughput requests buffered into larger batches for GPUs.- Use grouped batching by sequence length and model-shard affinity to reduce padding waste.- CUDA Graphs + kernel fusion for repeated short-batch execution paths to remove kernel launch overheads.

Caching hot responses:- Two-layer semantic cache in Redis + local LRU on worker: exact-match cache for frequent prompts; approximate semantic cache (embedding-based ANN) for near-duplicates.- TTL + versioning on cached outputs; validate policy for sensitive/generated content freshness.- Cache key derivation includes model-version, prompt-normalization, and decoding params (temperature, top-k).

Autoscaling & cost control:- SLO-driven autoscaler: scale replica pools independently for low-latency and throughput workloads. Maintain a hot pool of pre-warmed GPU shards for low-latency traffic to avoid cold-starts.- Predictive scaling using traffic patterns and queue length forecasts; fast scale-up via warm-standby instances, graceful scale-down with session drain and KV persistence.- Right-size instances: mixture of multi-GPU nodes for sharded large models and single-GPU high-frequency nodes for small models.

Orchestration between CPU and GPU:- CPU handles tokenization, input validation, policy checks, embedding lookup, and heavy I/O. Use pinned host memory for fast GPU transfers (CUDA IPC/NVLink).- Offload non-critical parts (e.g., beam search hypothesis scoring for large beams, reranking) to CPU when GPU contention exists.- Use device-aware scheduler (Kubernetes custom device plugin or Ray/Triton) that schedules containers close to required GPUs/NVLink fabric.- Use memory-mapped SSD for KV-cache spill and asynchronous prefetch to GPU.

Implementation & tooling:- Use inference runtime with optimizations: NVIDIA Triton / FasterTransformer / FlashAttention / cuBLASLt, integrated with model-serving (TorchScript/ONNX) and CUDA Graphs.- Telemetry: latency p90/95/99, KV hit rates, GPU utilization, batch sizes, queue lengths. Automated experiments to validate quantization quality.

Trade-offs:- More aggressive quantization saves latency/cost at potential quality loss — keep rollback and A/B testing.- Larger batch sizes maximize throughput but increase tail latency; enforce separate paths and SLO-aware scheduling.- Full GPU-resident KV-cache consumes memory; tier and evict aggressively for scale.

Example flow (low-latency):- Request enters gateway -> SLO tag -> route to low-latency worker -> tokenize on CPU -> transfer input to GPU pinned memory -> single-token or micro-batch inference using FP16+INT8 weights with GPU KV-cache -> decode & return (<50 ms tail targeted with pre-warmed shards + CUDA Graphs).

Example flow (high-throughput):- Offline jobs queued -> batched grouping by length -> scheduled to sharded pipeline with large batch sizes -> optimized kernels + mixed precision -> persist outputs to storage.

This design balances latency and throughput through separate optimized paths, tiered KV-cache, hybrid precision, sharding-aware placement, intelligent batching, and SLO-driven autoscaling.

EasyTechnical

108 practiced

Explain regularization techniques commonly applied in neural networks: L1/L2 weight penalties (weight decay), dropout, early stopping, and data augmentation. For each technique describe how it affects training, generalization, and any interactions with optimizer or batchnorm layers.

Sample Answer

L1/L2 weight penalties (weight decay)- What: Add ||w||1 or ||w||2^2 to loss. L2 (weight decay) is most common.- Effect on training: Penalizes large weights, encourages smaller, smoother weights; adds bias to gradients (gradient += λw for L2).- Generalization: Reduces overfitting by shrinking parameters, effective when model capacity is high.- Interactions: With optimizers like Adam, implement as true weight decay (decoupled) rather than adding to gradients to match theoretical behavior (use AdamW). With BatchNorm, don’t apply decay to scale/shift (γ, β) since they’re not regular model weights.

Dropout- What: Randomly zero activations during training with probability p; scale during inference.- Effect on training: Slows convergence (noisy gradients), forces redundancy and co-adaptation reduction.- Generalization: Acts like model averaging of many subnetworks, reduces overfitting.- Interactions: With optimizers, no special change; with BatchNorm, dropout can be redundant — place dropout after BatchNorm or reduce dropout rate because BatchNorm’s noise and normalization interact; many modern CNNs use one or the other.

Early stopping- What: Stop training when validation performance stops improving.- Effect on training: Prevents over-training past the point of best generalization; reduces wasted compute.- Generalization: Direct control of bias-variance tradeoff via stopping time.- Interactions: Early stopping implicitly regularizes and can interact with learning-rate schedules (e.g., restart LR might delay stopping); monitor stable validation metric and use patience.

Data augmentation- What: Apply label-preserving transformations (flips, crops, color jitter, token masking, back-translation).- Effect on training: Increases effective dataset size, slows per-epoch convergence but improves robustness.- Generalization: Forces invariances, reduces overfitting, often yields largest gains.- Interactions: With BatchNorm, ensure augmentation distribution similar between train and test; heavy augmentation may change activation distributions—BatchNorm still helps but consider larger batch sizes or SyncBatchNorm for distributed training. Augmentation pairs well with weight decay/dropout but can reduce their required strength.

Summary: These methods target overfitting via different mechanisms—capacity control (L1/L2), stochastic ensemble (dropout), training-time cutoff (early stopping), and richer data distribution (augmentation). Combine thoughtfully: use decoupled weight decay, be cautious placing dropout with BatchNorm, rely on augmentation first, and use early stopping to pick the final checkpoint.

HardTechnical

85 practiced

Implement the backward pass for scaled dot-product attention (single head) in numpy. Given Q, K, V matrices of shapes (B, T, d_k) and an upstream gradient dOut of shape (B, T, d_k) with attention computed as Out = softmax(QK^T / sqrt(d_k)) V, derive and implement steps to compute dQ, dK, dV efficiently. Discuss numerical stability of the softmax gradient.

Sample Answer

Approach — compute forward intermediates (S = QK^T / sqrt(dk), A = softmax(S), Out = A V). Backprop: dV = A^T @ dOut; dA = dOut @ V^T. Then use softmax Jacobian-vector product per row: for each row a and upstream g, ds = a * (g - sum(g * a)). Scale by 1/sqrt(dk). Finally dQ = ds @ K and dK = ds^T @ Q. All operations are batched and vectorized.

python

import numpy as np

def softmax_stable(x, axis=-1):
    x_max = np.max(x, axis=axis, keepdims=True)
    e = np.exp(x - x_max)
    return e / np.sum(e, axis=axis, keepdims=True)

def scaled_dot_product_attention_backward(Q, K, V, dOut):
    # Q,K,V: (B, T, d_k), dOut: (B, T, d_k)
    B, T, d = Q.shape
    scale = 1.0 / np.sqrt(d)

    # Forward (needed intermediates)
    # scores S: (B, T, T)
    S = np.matmul(Q, K.transpose(0,2,1)) * scale
    A = softmax_stable(S, axis=-1)            # (B, T, T)
    Out = np.matmul(A, V)                     # (B, T, d)

    # Grad w.r.t V
    # dV[b] = A[b].T @ dOut[b]
    dV = np.matmul(A.transpose(0,2,1), dOut)  # (B, T, d)

    # Grad w.r.t A
    # dA[b] = dOut[b] @ V[b].T
    dA = np.matmul(dOut, V.transpose(0,2,1))  # (B, T, T)

    # Softmax row-wise Jacobian-vector product:
    # For each row: ds = a * (g - sum(g * a))
    # vectorized:
    inner = np.sum(dA * A, axis=-1, keepdims=True)  # (B, T, 1)
    dS = (dA - inner) * A                            # (B, T, T)

    # scale for QK^T / sqrt(d)
    dS *= scale

    # dQ = dS @ K ; dK = dS^T @ Q
    dQ = np.matmul(dS, K)                            # (B, T, d)
    dK = np.matmul(dS.transpose(0,2,1), Q)          # (B, T, d)

    return dQ, dK, dV

Key points:- Complexity: O(B * T^2 * d) time, O(B * T^2) extra memory for attention matrix.- Softmax numerical stability: compute softmax with max-subtraction (implemented). The gradient formula used (ds = a * (g - sum(g*a))) is numerically stable and equivalent to applying Jacobian; using stable softmax values ensures stable gradients.- Edge cases: tiny d_k (scale), very large T (memory blowup), batched shapes consistency.

Unlock Full Question Bank

Get access to hundreds of Neural Network Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.