Neural Network Fundamentals Questions

Core concepts and building blocks of artificial neural networks including neuron models, network layers and layer types (dense fully connected, convolutional, recurrent, and transformer blocks), activation functions and their properties (for example ReLU, sigmoid, tanh, GELU), forward propagation, loss functions, and the basics of optimization such as gradient descent. Explain backpropagation conceptually and mathematically using the chain rule, describe forward and backward passes, and discuss common practical issues such as vanishing and exploding gradients, weight initialization, regularization techniques, and when specific architectures are appropriate for particular problem domains (for example convolutional networks for vision and sequence models for temporal data).

HardTechnical

99 practiced

Explain the scaled dot-product attention mechanism used in transformers. Provide the mathematical formula Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) V, explain why the 1/sqrt(d_k) scaling is used, and describe multi-head attention intuitively and in terms of dimension splits and concatenation.

HardTechnical

101 practiced

Write a Python function that computes the receptive field size of a single output neuron after applying a sequence of convolutional / pooling layers. The function should accept lists of kernel sizes k_i, strides s_i, paddings p_i, and dilations d_i and return the receptive field and total effective stride. Explain the algorithm and its time complexity.

Sample Answer

Approach (brief): The receptive field (RF) of a single output neuron grows layer-by-layer. If RF_l and stride S_l are the receptive field and total stride up to layer l (input being layer 0: RF_0 = 1, S_0 = 1), then for layer i with kernel k, stride s, dilation d:- total stride S_i = S_{i-1} * s- RF_i = RF_{i-1} + ((k - 1) * d) * S_{i-1}Padding does not change RF size (it shifts coordinates), so p_i is validated but not used for size.

Code (handles lists, validates lengths, supports defaults):

python

from typing import List, Tuple

def receptive_field(k: List[int],
                    s: List[int],
                    p: List[int],
                    d: List[int]) -> Tuple[int,int]:
    """
    Returns (receptive_field, total_stride) for a single output neuron
    after applying layers in order (first list element is first layer applied).
    k,s,p,d must be same length. p is accepted but not used for RF size.
    """
    n = len(k)
    if not (len(s)==len(p)==len(d)==n):
        raise ValueError("k, s, p, d must have same length")
    rf = 1         # receptive field at input for a single pixel
    total_stride = 1
    for i in range(n):
        ki, si, di = k[i], s[i], d[i]
        if ki <= 0 or si <= 0 or di <= 0:
            raise ValueError("kernel, stride, dilation must be positive integers")
        rf = rf + ((ki - 1) * di) * total_stride
        total_stride = total_stride * si
    return rf, total_stride

# Example:
# Two conv layers: 3x3 stride1, then 3x3 stride2
# k=[3,3], s=[1,2], p=[1,1], d=[1,1] -> RF = 1 + (2)*1 + (2)*1*1 = 5? compute:
# Running function gives (5,2)

Key points:- RF formula aggregates the extra receptive span (k-1)*d at the current layer scaled by the cumulative stride from previous layers.- Padding does not change RF size; it affects mapping of output coordinates to input coordinates (offset).Time complexity: O(n) where n is number of layers (single pass). Space: O(1).Edge cases: empty lists -> returns (1,1) if you adjust to handle n=0; validate positive integers; extremely large strides/kernels can overflow integers in constrained environments.

MediumTechnical

88 practiced

Conceptually explain how backpropagation implements the chain rule across a multi-layer network. Using a small MLP (Input → W1,b1 → σ → W2,b2 → L), show the recursive relation for δ_l and the vectorized formula used in implementations.

HardSystem Design

91 practiced

Design a full ML training and deployment pipeline for an image classification model. Include data ingestion, preprocessing/versioning, training (experimentation, hyperparameter tracking), validation, CI/CD for model packaging, canary/capacity-aware rollout, A/B testing, monitoring and retraining triggers. Explain how you'd handle data drift, model rollback, and metric governance in production.

Sample Answer

Requirements & constraints:- Image classification, low-latency serving (<200ms), high throughput (auto-scale), reproducibility, auditable metrics and safe rollouts.

High-level pipeline:1. Data ingestion & storage- Ingest from sources (mobile/web, batch ETL) into object store (S3/GCS) with event metadata (Kafka).- Store lineage + raw hashes in a metadata DB (e.g., Feathr/Delta Lake).

2. Preprocessing & versioning- Use data versioning (DVC/Delta Lake) and schema checks (Great Expectations).- Precompute transforms in Spark/TensorFlow Data Service; save transform graph and tokenizer/augmentation artifacts in artifact store.- Capture dataset snapshot IDs and manifest in ML metadata (MLMD).

3. Training & experimentation- Experiment tracking with MLflow or Weights & Biases: log code commit, dataset version, hyperparams, model weights, metrics, and container image.- Use distributed training (Horovod/PyTorch DDP) on managed infra (K8s or cloud ML).- Automated HPO (Optuna or Ray Tune) with reproducible seeds; store best trials in registry.

4. Validation & approval- Offline validation: holdout test, cross-val, OOD checks, calibration, fairness & explainability (SHAP/Integrated Gradients).- Automated acceptance checks: metric thresholds, confusion matrix, per-class recall, inference cost.

5. CI/CD & packaging- GitOps: code + model artifacts build pipeline (GitHub Actions/Jenkins -> build container with model + runtime).- Model registry (MLflow Model Registry) with stages (Staging, Production).- Automated unit/integration tests: model signature, input/output schema.

6. Deployment strategy- Serve via scalable model server (Triton/Seldon/KFServing) in K8s.- Canary + capacity-aware rollout: route small % traffic to canary using Istio/Linkerd; scale pods with HPA based on CPU/latency and precomputed capacity tests. Gradually ramp using automated metrics gates.

7. A/B testing & experiment analysis- Use feature flags / traffic routing to allocate users to control vs treatment; collect labeled/online metrics.- Statistical analysis platform to compute lift, confidence intervals, and guardrails (min sample size, multiple hypothesis correction).

8. Monitoring & observability- Metrics: accuracy (from periodic labeled samples), latency, throughput, CPU/GPU, memory.- Use Prometheus + Grafana, logging to ELK, tracing (Jaeger).- Data drift & concept drift detectors: monitor input feature distributions (KL divergence, PSI), embedding drift (cosine), and label distribution shifts. Alert on threshold breaches.- Explainability & fairness monitors (per-group metrics).

9. Retraining triggers & automation- Retrain when: significant data/embedding drift, sustained metric degradation beyond SLA, or periodic schedule.- Pipeline orchestrator (Airflow/Kubeflow Pipelines): automatic data snapshot -> preprocess -> train -> validate -> staging deploy for canary.

10. Rollback & safety- Maintain immutable model artifacts and container images. Automated rollback when canary fails gates or production metrics degrade; deploy previous stable model via registry.- Kill-switch: route all traffic to control or older model instantly.

11. Metric governance & audit- Central metric store (Feature store + metrics DB). Define canonical metrics and SLAs; version metric definitions.- Access control and audit logs for model promotions (RBAC, signed approvals).- Periodic reviews and model card generation with declared intended use, limitations, data lineage.

Trade-offs & reasoning:- Using canary + capacity-aware scaling minimizes user impact and avoids overload.- MLflow + model registry ensures reproducibility and controlled promotion.- Automated drift detection balances sensitivity vs false alarms; combine triggers (drift + perf drop) before costly retrain.

This design supports reproducible experimentation, safe automated rollouts, continuous monitoring, and governed retraining/rollback to keep image classifier reliable in production.

HardTechnical

145 practiced

Explain Xavier (Glorot) and He (Kaiming) initialization: derive the intuition and show the formulas for variance-preserving initialization for both uniform and normal distributions. Explain which to use with tanh/sigmoid vs ReLU/LeakyReLU and why.

Unlock Full Question Bank

Get access to hundreds of Neural Network Fundamentals interview questions and detailed answers.

Join thousands of developers preparing for their dream job.