Neural Network Architectures Questions

Broad coverage of modern and advanced neural network architectures, design principles, and components. Candidates should understand core structural elements such as neurons, layers, weights, biases, activation functions, forward and backward passes, and how architecture choices influence learning. Know a range of architecture families including feedforward networks, convolutional neural networks, recurrent neural networks including long short term memory and gated recurrent unit variants, transformer architectures with self attention and multi head attention, vision transformer adaptations, and graph neural networks. Understand inductive biases that make certain architectures appropriate for particular data modalities, trade offs between depth and width, parameter efficiency and computational complexity, and practical considerations such as initialization, normalization, optimization, and scaling strategies. Be able to explain when to choose one architecture over another for a given problem, how to combine or adapt architectures for domain specific needs, and how modern architecture advances address limitations of prior models.

EasyTechnical

80 practiced

Implement a 2D convolution forward operation in Python using NumPy for a single-channel image. Function signature: conv2d(input, kernel, stride=1, padding=0). Input shape: (H_in, W_in). Kernel: (K_h, K_w). Return the output 2D array. Do not use any library convolution helpers. Include handling of padding and stride semantics.

Sample Answer

Approach: We'll explicitly pad the input, then slide the kernel over the padded input using the specified stride, computing elementwise multiplication and sum at each output position. This implementation supports integer stride and symmetric zero padding.

python

import numpy as np

def conv2d(input, kernel, stride=1, padding=0):
    """
    2D convolution for single-channel input.
    input: 2D numpy array shape (H_in, W_in)
    kernel: 2D numpy array shape (K_h, K_w)
    stride: int >=1
    padding: int >=0 (zeros added equally on all sides)
    returns: 2D numpy array of convolved output
    """
    if stride < 1 or padding < 0:
        raise ValueError("stride must be >=1 and padding >=0")
    input = np.asarray(input)
    kernel = np.asarray(kernel)
    H_in, W_in = input.shape
    K_h, K_w = kernel.shape

    # pad input with zeros
    if padding > 0:
        input_padded = np.pad(input, ((padding, padding), (padding, padding)), mode='constant')
    else:
        input_padded = input

    H_p, W_p = input_padded.shape

    # output dimensions (floor semantics)
    out_h = (H_p - K_h) // stride + 1
    out_w = (W_p - K_w) // stride + 1

    if out_h <= 0 or out_w <= 0:
        return np.zeros((0, 0))

    out = np.zeros((out_h, out_w), dtype=input.dtype)

    # flip kernel for convolution (if you want correlation, remove flip)
    k_flipped = np.flip(np.flip(kernel, axis=0), axis=1)

    for i in range(out_h):
        for j in range(out_w):
            y = i * stride
            x = j * stride
            patch = input_padded[y:y+K_h, x:x+K_w]
            out[i, j] = np.sum(patch * k_flipped)

    return out

Key points:- We flip the kernel to implement mathematical convolution; remove flip for cross-correlation.- Time complexity: O(out_h * out_w * K_h * K_w). Space: O(H_in*W_in + padding) for padded input.Edge cases:- Kernel larger than padded input returns empty output (handled).- Non-integer or negative stride/padding raise error.- Works for numpy numeric dtypes; convert types if needed.

EasyTechnical

95 practiced

Implement a forward pass for a two-layer feedforward neural network in Python using NumPy. Input X has shape (batch_size, input_dim). Given weight matrices W1 (input_dim, hidden_dim), b1 (hidden_dim,), W2 (hidden_dim, output_dim), b2 (output_dim,), compute hidden = ReLU(X @ W1 + b1), logits = hidden @ W2 + b2, probs = softmax(logits). Return (logits, probs). Do not use TensorFlow/PyTorch. Include a short docstring and simple shape checks.

Sample Answer

Approach: do matrix multiplies with NumPy, apply bias broadcasting, use ReLU for hidden, and a numerically stable softmax for probabilities. Include simple shape checks to catch mismatched dimensions.

python

import numpy as np

def forward_two_layer(X, W1, b1, W2, b2):
    """
    Forward pass for a two-layer feedforward network.
    Inputs:
      - X: (batch_size, input_dim)
      - W1: (input_dim, hidden_dim)
      - b1: (hidden_dim,)
      - W2: (hidden_dim, output_dim)
      - b2: (output_dim,)
    Returns:
      - logits: (batch_size, output_dim)
      - probs:  (batch_size, output_dim) - softmax probabilities
    """
    # Basic shape checks
    if X.ndim != 2:
        raise ValueError("X must be 2D (batch_size, input_dim)")
    input_dim = X.shape[1]
    if W1.shape[0] != input_dim:
        raise ValueError("W1.shape[0] must match X.shape[1]")
    hidden_dim = W1.shape[1]
    if b1.shape != (hidden_dim,):
        raise ValueError("b1 must have shape (hidden_dim,)")
    if W2.shape[0] != hidden_dim:
        raise ValueError("W2.shape[0] must match hidden_dim")
    output_dim = W2.shape[1]
    if b2.shape != (output_dim,):
        raise ValueError("b2 must have shape (output_dim,)")

    # Forward computations
    z1 = X @ W1 + b1  # (batch_size, hidden_dim) - bias broadcasts
    hidden = np.maximum(0, z1)  # ReLU

    logits = hidden @ W2 + b2  # (batch_size, output_dim)

    # Numerically stable softmax
    logits_max = np.max(logits, axis=1, keepdims=True)
    exp_shifted = np.exp(logits - logits_max)
    probs = exp_shifted / np.sum(exp_shifted, axis=1, keepdims=True)

    return logits, probs

Key points:- Uses broadcasting for biases.- ReLU via np.maximum for efficiency.- Softmax uses logit shifting for numerical stability.- Time: O(batch_size * (input_dim*hidden_dim + hidden_dim*output_dim)). Space: O(batch_size * output_dim) additional.

MediumTechnical

122 practiced

Explain scaled dot-product attention and multi-head attention algorithmically and mathematically. Provide the equations for Attention(Q,K,V), explain the sqrt(d_k) scaling factor, compute time and memory complexity in terms of sequence length L and embedding dim d, and discuss why multiple heads can be beneficial.

HardTechnical

76 practiced

Provide PyTorch code (skeleton) that executes a custom optimizer step with global gradient-norm clipping, learning-rate scaling for gradient-accumulation steps, and support for mixed-precision training using torch.cuda.amp. Include comments indicating where to call clip and scaler operations and how to handle zero_grad and optimizer.step safely.

Sample Answer

Here's a compact PyTorch skeleton demonstrating a custom optimizer step with global gradient-norm clipping, learning-rate scaling for gradient-accumulation steps, and mixed-precision support using torch.cuda.amp. Comments indicate where to call clipping/scaler ops and how to safely handle zero_grad and optimizer.step.

python

import torch
from torch.cuda.amp import GradScaler, autocast

scaler = GradScaler()
optimizer = torch.optim.Adam(model.parameters(), lr=base_lr)
accum_steps = 4  # number of gradient accumulation steps
max_grad_norm = 1.0  # global norm clipping threshold

def train_step(batch):
    inputs, targets = batch
    # scale LR if you prefer scaling per effective batch (optional):
    # optimizer.param_groups[0]['lr'] = base_lr * accum_steps

    with autocast():  # mixed precision forward
        outputs = model(inputs)
        loss = loss_fn(outputs, targets) / accum_steps  # scale loss for accumulation

    # scale and backpropagate
    scaler.scale(loss).backward()

    # every accum_steps, perform gradient clipping and optimizer step
    if (train_step.step_counter + 1) % accum_steps == 0:
        # Unscale gradients before clipping so clipping uses true fp32 grads
        scaler.unscale_(optimizer)  # <-- call before clipping

        # If using DistributedDataParallel, ensure gradients are all-reduced already by DDP
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)  # <-- gradient clipping here

        # Optionally, check for inf/nan gradients:
        # if not torch.isfinite(total_grad_norm):
        #     scaler.zero_grad()
        #     return

        # step optimizer through scaler to handle unscaling/step/opt-state in fp32
        scaler.step(optimizer)  # <-- scaler handles optimizer.step safely
        scaler.update()

        # zero gradients safely after step
        optimizer.zero_grad(set_to_none=True)  # recommended for performance

    else:
        # do not zero grads between accumulation mini-steps
        pass

    train_step.step_counter += 1

# initialize counter attribute
train_step.step_counter = 0

Key notes:- Scale loss by 1/accum_steps to keep learning dynamics consistent.- Always call scaler.unscale_(optimizer) before gradient clipping.- Use scaler.step(optimizer) and then scaler.update() instead of optimizer.step().- Use optimizer.zero_grad(set_to_none=True) after the step; do NOT zero in-between accumulation sub-steps.- In distributed training, use DDP which reduces gradients automatically; ensure clipping uses global norm across processes (use all_reduce on norms if necessary).

MediumTechnical

77 practiced

You have only 1,000 labeled medical images to build a cancer detection model. Describe a step-by-step transfer learning approach using a pre-trained CNN (e.g., ResNet): data preprocessing, augmentation, layer freezing/unfreezing schedule, optimizer and learning rates, regularization, cross-validation, and evaluation procedures to avoid overfitting.

Sample Answer

Start by clarifying objective and class balance (binary cancer yes/no). Then follow this step-by-step transfer-learning plan:

1) Data preprocessing- Standardize image size (e.g., 224x224 for ResNet), maintain aspect ratio with padding.- Apply domain-appropriate normalization using ImageNet mean/std or compute dataset mean/std if feasible.- Convert to float32, ensure consistent channel order; remove corrupted images.

2) Data augmentation (critical for 1,000 images)- Use realistic on-the-fly augmentations: random rotations (±15°), flips (if anatomically valid), small translations, scale jitter, brightness/contrast, elastic deformations if medically acceptable, and Gaussian noise.- Use stronger augmentation during training but not for validation/test.

3) Train/validation splitting & cross-validation- Use stratified k-fold CV (k=5) to estimate generalization; keep an untouched holdout test set (10–15%) if possible.- For each fold, use stratified splits to preserve class ratios.

4) Model setup & layer freezing schedule- Load ResNet pre-trained on ImageNet.- Replace final classifier with a small head: GlobalAveragePooling -> Dense(256, ReLU) -> Dropout(0.5) -> Dense(1, sigmoid).- Phase 1 (feature extraction): freeze all convolutional layers; train only new head for 5–20 epochs until validation loss plateaus.- Phase 2 (gradual fine-tuning): unfreeze last ResNet block (or last N layers) and fine-tune with low learning rate for 10–30 epochs. Optionally unfreeze more layers progressively if validation improves.

5) Optimizer and learning rates- Phase 1: Adam or SGD with momentum; LR = 1e-3 (Adam) or 1e-2 (SGD) for head.- Phase 2: SGD with momentum preferred for fine-tuning (more stable); LR = 1e-4 to 1e-5. Use discriminative LRs (higher for head, lower for earlier layers).- Use cosine annealing or ReduceLROnPlateau to lower LR on plateau.

6) Regularization- L2 weight decay (e.g., 1e-4), dropout in classifier (0.3–0.6), label smoothing (if appropriate), and strong augmentation.- Early stopping based on validation metric with patience tuned to avoid premature stop.

7) Class imbalance handling- Use class weights or focal loss if positive cases are rare; augment minority class more if safe.

8) Evaluation & metrics- Use clinically relevant metrics: AUROC, sensitivity/recall, specificity, precision, F1; report confidence intervals via bootstrap across folds.- Calibrate probabilities (Platt or isotonic) if outputs used for decision thresholds.- Perform per-patient aggregation if multiple images per patient.

9) Robustness checks & best practices- Visualize saliency/Grad-CAM to ensure model focuses on plausible regions.- Run sanity checks: training with shuffled labels (should fail), evaluate on external/temporal holdout if available.- Freeze random seeds and log experiments; use test set only once.

This pipeline maximizes data efficiency, reduces overfitting via staged fine-tuning, heavy but realistic augmentation, regularization, and robust cross-validation and evaluation tailored to clinical needs.

Unlock Full Question Bank

Get access to hundreds of Neural Network Architectures interview questions and detailed answers.

Join thousands of developers preparing for their dream job.