Technical Foundation and Self Assessment Questions

Covers baseline technical knowledge and the candidate's ability to honestly assess and communicate their technical strengths and weaknesses. Topics include fundamental infrastructure and networking concepts, operating system and protocol basics, core development and platform concepts relevant to the role, and the candidate's candid self evaluation of their depth in specific technologies. Interviewers use this to calibrate how technical the candidate is expected to be, identify areas for growth, and ensure alignment of expectations between product and engineering for collaboration.

MediumTechnical

41 practiced

Compare A/B testing and champion-challenger strategies for evaluating models. Design a simple A/B experiment for a recommender system that includes: metric selection, traffic split, significance test, and minimum sample size considerations.

Sample Answer

Comparison:- A/B testing: randomized, online experiment comparing control (A) vs treatment (B). Best for causal inference, measures end-to-end user impact, detects UX/regression effects, needs randomized routing and monitoring for novelty/temporal effects.- Champion–challenger: continuous evaluation framework where a current “champion” serves traffic and one or more “challengers” run in parallel (can be offline or shadow) to compare performance before promoting. Safer for risk-averse launches, supports gradual rollout and many variants, but if challengers run shadow (no user exposure) you only get predicted/offline metrics unless you route live traffic.

A/B experiment design for a recommender system

1) Goal & metrics- Primary metric: gross conversion uplift relevant to business — e.g., click-through rate (CTR) on recommendations OR relative increase in purchases per user (conversion rate) depending on business. Choose one primary to avoid multiple-testing penalties. Example: Recommendation CTR per session.- Secondary metrics: engagement (time on site), revenue per session, downstream conversion (purchase rate), negative signals (unfollow, report, bounce).- Safety guardrails: latency, error rate, CPU/memory.

2) Traffic split & randomization- Split: 50/50 for initial full-power test (A=control recommender; B=new model). If risk-averse, use 10/90 ramp (10% treatment) then 50/50 after smoke checks.- Unit of randomization: user-level (persist assignment via cookie or user ID) to avoid cross-contamination and carryover. Ensure consistency across sessions for a user during experiment.

3) Significance test- For CTR (binary event per impression/session): use two-sample proportion (z) test or logistic regression controlling for covariates. If metric aggregated per-user (clicks/sessions), use bootstrap or t-test on per-user aggregated rates to account for heavy tails.- Predefine alpha = 0.05 (two-sided) and optionally adjust for multiple looks with O’Brien–Fleming or Pocock if doing interim analyses. Use one primary metric only for hypothesis decision.

4) Minimum sample size (example calculation)- Suppose baseline CTR = 4% (p0 = 0.04). Detect minimum relative uplift = 10% (absolute p1 = 0.044). Alpha=0.05, power=0.8.- Using two-proportion sample size formula: n_per_group ≈ [ (Z_{1-α/2}√(2p̄(1−p̄)) + Z_{power}√(p0(1−p0)+p1(1−p1)))^2 ] / (p1−p0)^2 where p̄=(p0+p1)/2.- Plugging numbers gives roughly n_per_group ≈ 60k–80k users (approximate; compute exact with tool).- If unit is impressions or sessions, recalc using baseline event rate per session. Account for expected % of eligible users, retention, and any QA cuts (e.g., bots).

5) Monitoring & validity checks- Run pre-checks: balance of covariates (device, geography), randomization QA, no large throughput/latency regressions.- Monitor secondary and guardrail metrics continuously; stop early if negative safety signals.- Post-hoc: compute incremental lift, confidence intervals; do heterogeneity analysis (by segment) and check novelty decay over time.

6) Champion–challenger workflow variant- Start with champion–challenger in shadow for 2–4 weeks: run challengers in parallel, compare predicted CTR and offline ranking metrics. If promising, run a small-sample live A/B (10% traffic) before full 50/50 rollout. This reduces risk and provides both offline and online evidence.

Reasoning: choosing user-level randomization avoids contamination; primary metric selected ties to business outcome; significance and sample-size ensure statistical validity; monitoring guardrails protect UX and system health.

MediumTechnical

47 practiced

Describe how you would version datasets and model artifacts for reproducible ML experiments. Cover storage choices, metadata (schema, provenance), hashing, dataset lineage, and trade-offs between storage cost and reproducibility.

Sample Answer

I’d implement a reproducible versioning system combining immutable storage for raw/derived datasets, a metadata/catalog service, and a model registry. Key components and choices:

Storage- Raw/large datasets: object storage (S3/GCS) with lifecycle rules; store as immutable objects (date-stamped prefixes) or in a data lake format (Parquet + partitioning).- Versioned sliced datasets / small snapshots: use Git-like data versioning (DVC, LakeFS, or Delta Lake) to track diffs efficiently.- Model artifacts: binary blobs (wheel/torchscript/saved_model) in object storage + entries in a model registry (MLflow, ModelDB, or custom DB).

Metadata & Schema- Central metadata catalog (e.g., Data Catalog, Hive Metastore) storing schema (column types, nullable), statistics (row counts, checksums), and semantic tags.- Enforce and store schema hashes and compatibility checks for downstream jobs.

Provenance & Lineage- Record lineage as directed graphs: dataset versions → transformation jobs (with job code hash, container image, config) → derived dataset/model. Use tools like Pachyderm, OpenLineage, or custom metadata tables.- For each training run capture: input dataset version IDs, exact preprocessing code commit + diff, environment (Docker image hash), dependency versions, hyperparameters, RNG seeds, and output model artifact ID.

Hashing & Integrity- Compute content hashes for files/blobs (SHA-256) and dataset-level Merkle-style hashes for directories/partitions. Store hashes in metadata to detect drift or corruption.- Hash training artifacts: model binary hash + serialized model config; include hash in model registry entry.

Trade-offs: storage cost vs reproducibility- Full snapshotting of every intermediate (highest reproducibility) is expensive. Use delta/storage-efficient formats (columnar + partitioning), and store only checkpoints for expensive-to-recompute steps.- Keep raw immutable data forever; for derived/intermediate artifacts apply retention policies (retain N latest + those tied to production deployments). Archive older artifacts to cheaper storage (Glacier).- For very high reproducibility, store container images and data snapshots; for cost-sensitive projects, store dataset recipes (deterministic ETL code + small seed samples) plus hashed identifiers so full recompute from raw is possible.

Operational practices- Automate capture: CI/CD or training orchestration must automatically record metadata and push to registry.- Provide reproducibility-first templates: standardized experiment spec (data_version, code_commit, env_image, params) that can re-run an experiment end-to-end.- Monitoring & audits: periodic checksum validation and lineage queries to answer “which models used feature X version Y?”

This approach balances practical storage costs with reproducibility by combining immutable raw data, efficient delta/versioning for intermediates, strict metadata/provenance capture, and retention policies tailored to business risk.

EasyTechnical

33 practiced

Compare full-batch gradient descent, stochastic gradient descent, and mini-batch gradient descent. Discuss convergence behavior, noise properties, and memory trade-offs when training deep neural networks.

Sample Answer

Full-batch, stochastic (SGD), and mini-batch gradient descent trade off convergence speed, noise, and memory—important for training deep nets.

Definitions (brief)- Full-batch GD: compute gradient on the entire training set each update.- Stochastic GD: update per single example (or label).- Mini-batch GD: update on a small batch (typical 32–1024 examples).

Convergence behavior- Full-batch: Deterministic updates and smooth convergence toward the true gradient, good for stable final convergence but expensive per step. Can get stuck in shallow local minima or plateaus; slower wall-clock time when dataset large.- SGD: High-frequency noisy updates; often converges faster in wall-clock time and helps escape poor local minima due to noise, but requires careful decreasing learning rate or schedulers to settle.- Mini-batch: Balances both — smoother than SGD, cheaper per step than full-batch. Batch size affects convergence: small batches introduce helpful noise and generalization benefits; very large batches approach full-batch behavior and may require learning rate scaling.

Noise properties- Full-batch: Minimal stochastic noise (only from numerical issues), more precise gradient estimate.- SGD: Maximum noise — gradient variance high; can act as implicit regularizer improving generalization.- Mini-batch: Controlled noise level proportional to 1/batch_size. Enables stable use of vectorized hardware while preserving some regularizing noise.

Memory and computational trade-offs- Full-batch: Highest memory (must load full dataset/model activations for some ops), high per-update compute; impractical for large datasets.- SGD: Lowest memory per update, but inefficient on modern GPUs due to poor parallelism; higher I/O overhead.- Mini-batch: Best practical compromise — good GPU utilization, manageable memory, and efficient I/O. Choose batch size to fit device memory and tune learning rate (e.g., linear scaling rule) and optimizers (Adam adapts better to smaller batches).

Practical guidance- Use mini-batches (32–256) as default for deep learning; scale batch size up with learning rate and warmup if you need throughput.- Use SGD with momentum or adaptive optimizers; apply learning-rate schedules and regularization to control convergence/noise trade-offs.

EasyTechnical

49 practiced

Write a Python generator function that yields mini-batches from an input dataset represented as two NumPy arrays X and y. The generator should support shuffle per epoch and configurable batch_size and yield tuples (X_batch, y_batch). State any assumptions about memory.

Sample Answer

To create mini-batches from NumPy arrays with optional shuffling per epoch, we can write a simple generator that yields (X_batch, y_batch). Assumptions: X and y are NumPy arrays with matching first-dimension length and fit in memory (generator doesn't duplicate the whole dataset). Batches are created using integer indexing (np.take) which returns copies for safety; if you want views and know your arrays are contiguous you can slice directly.

python

import numpy as np
from typing import Iterator, Tuple

def minibatch_generator(X: np.ndarray,
                        y: np.ndarray,
                        batch_size: int = 32,
                        shuffle: bool = True,
                        drop_last: bool = False) -> Iterator[Tuple[np.ndarray, np.ndarray]]:
    """
    Yields (X_batch, y_batch) tuples.
    - X, y: arrays with same first dimension n
    - batch_size: size of each minibatch
    - shuffle: whether to shuffle indices each epoch
    - drop_last: if True, drop final smaller batch
    """
    if X.shape[0] != y.shape[0]:
        raise ValueError("X and y must have the same number of samples")
    n = X.shape[0]
    indices = np.arange(n)
    while True:  # infinite generator; caller typically breaks after desired epochs
        if shuffle:
            np.random.shuffle(indices)
        for start in range(0, n, batch_size):
            end = start + batch_size
            if end > n and drop_last:
                break
            batch_idx = indices[start:end]
            yield np.take(X, batch_idx, axis=0), np.take(y, batch_idx, axis=0)

Key points:- Works for classification/regression pipelines and model.fit loops.- Infinite generator pattern lets caller control epoch count; to run one epoch, iterate n_batches = ceil(n/batch_size) times.Time complexity: O(n) work per epoch (plus O(n) for shuffle). Space: O(batch_size) per yielded batch (plus O(n) for the indices array).Edge cases:- batch_size <= 0 -> raise or validate.- n < batch_size -> yields one batch (or none if drop_last=True).- Non-contiguous arrays: np.take returns safe copies.

HardTechnical

42 practiced

Evaluate three approaches for secure inference: running inference inside a Trusted Execution Environment (TEE), using homomorphic encryption, and performing client-side encryption/tokenization. Compare their threat models, performance overhead, and practicality for real-time services.

Sample Answer

Situation: As an ML engineer evaluating secure inference for production real‑time services, consider three approaches: Trusted Execution Environments (TEEs), Homomorphic Encryption (HE), and client‑side encryption/tokenization.

Threat model:- TEE: Protects confidentiality/integrity of model and data against a compromised OS/hypervisor and some physical attacks (if attestation and secure hardware are intact). Does not protect against side‑channel attacks unless mitigations are applied. Requires trust in hardware vendor.- Homomorphic Encryption: Strong cryptographic protection — the server never sees plaintext inputs or intermediate results; adversary model assumes standard crypto hardness. Does not require trusted hardware but assumes correct implementation and no leakage from ciphertext size/timing.- Client‑side encryption/tokenization: Shifts trust to client; server only sees tokens or transformed features. Threat model depends on transformation: deterministic tokens leak equality/frequency; noninvertible tokenization (secure hashing + salts) reduces leakage. Server could still infer from features or model outputs.

Performance overhead:- TEE: Moderate overhead — context switches, memory limits (e.g., SGX EPC size) and reduced parallelism can slow throughput ~1.5–5× depending on workload and batching. Low latency possible if model fits enclave and I/O is optimized.- HE: High overhead — fully homomorphic or even leveled HE multiplies latency and computation by 10^2–10^6 depending on circuit depth and ciphertext packing. Large neural nets are currently impractical for strict real‑time (ms–tens of ms) without heavy algorithmic compression.- Client‑side: Minimal server overhead (same inference cost) but adds client compute and latency for preprocessing. Overall end‑to‑end latency can be low if client is powerful; mobile clients may struggle.

Practicality for real‑time services:- TEE: Most practical compromise for many real‑time systems where hardware support exists (cloud SGX, Nitro enclaves). Good for medium‑sized models, allows leveraging optimized inference libraries with moderate changes. Requires operational handling of attestation and enclave lifecycle.- HE: Best for highest privacy guarantees but currently suitable for low‑complexity models (logistic regression, small decision trees) or when latency is not strict. Research active (CKKS for approximate arithmetic helps), but production real‑time adoption remains limited.- Client‑side: Highly practical when clients can perform preprocessing or hold model locally (on‑device inference). Works well for privacy‑sensitive features; also enables split computing (partial preprocessing client, rest server). Limitations include heterogeneous client capability and potential for client compromise.

Recommendations:- For sub‑second, high‑throughput services: prefer TEE plus application‑level mitigations (side‑channel hardening, attestation). Combine with model watermarking and rate limits.- For highest cryptographic guarantees where latency can be tolerated: use HE for small models or batched offline processing.- For edge/mobile scenarios: prefer client‑side tokenization or on‑device inference; consider hybrid: client extracts features, server runs model in TEE.- Always layer protections: transport TLS, rigorous access controls, monitoring, and privacy‑preserving logging.

Unlock Full Question Bank

Get access to hundreds of Technical Foundation and Self Assessment interview questions and detailed answers.

Join thousands of developers preparing for their dream job.