Hands On Projects and Problem Solving Questions

Discussion of practical projects and side work you have built or contributed to across domains. Candidates should be prepared to explain their role, architecture and design decisions, services and libraries chosen, alternatives considered, trade offs made, challenges encountered, debugging and troubleshooting approaches, performance optimization, testing strategies, and lessons learned. This includes independent side projects, security labs and capture the flag practice, bug bounty work, coursework projects, and other hands on exercises. Interviewers may probe for how you identified requirements, prioritized tasks, collaborated with others, measured impact, and what you would do differently in hindsight.

EasyTechnical

72 practiced

Write a minimal Dockerfile (in text) to run a PyTorch training script 'train.py' with GPU support. Requirements: use an official PyTorch CUDA runtime image, install Python packages from 'requirements.txt', set WORKDIR, copy code, and set CMD to run 'python train.py --config config.yaml'. Explain any runtime flags required to enable GPU access.

Sample Answer

Use an official PyTorch CUDA runtime image, install requirements, set WORKDIR, copy code, and run the script. Minimal Dockerfile:

dockerfile

FROM pytorch/pytorch:2.2.0-cuda11.8-cudnn8-runtime

# Set working directory
WORKDIR /app

# Copy requirements and install Python packages
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY . .

# Default command to run training
CMD ["python", "train.py", "--config", "config.yaml"]

Runtime flags to enable GPU access:- Modern Docker (>=20.10) + NVIDIA Container Toolkit: run container with --gpus all (e.g. docker run --gpus all -it --rm imagename). This exposes all GPUs; you can limit with --gpus '"device=0,1"'.- Older setups / legacy: use --runtime=nvidia (requires nvidia-docker2) and ensure nvidia-container-toolkit is installed.- For multi-GPU control you can also set NVIDIA_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES env vars inside the container.Make sure host has NVIDIA drivers and the NVIDIA Container Toolkit installed.

HardSystem Design

57 practiced

Design an automated compression and validation pipeline: given a trained model, run quantization/pruning/distillation transforms, automatically validate accuracy, latency, and robustness on holdout and adversarial checks, and gate promotion to production with rollback on failure. Describe components, tests, and how to maintain lineage of compressed artifacts.

Sample Answer

Requirements:- Functional: apply quantization/pruning/distillation transforms automatically to a trained model and produce deployable artifacts.- Validation: measure accuracy (holdout), latency (target hardware), robustness (adversarial / distributional shift).- Safety: gating to production with automated rollback and audit trail / lineage.

High-level architecture:- Orchestrator: Airflow/GitHub Actions/Kubeflow Pipelines triggers on new trained model.- Transform stage: containerized jobs that produce variants (quantized INT8, structured-pruned, distilled smaller net) using standardized toolchains (PyTorch FX + torch.quantization, ONNX + TensorRT, OpenVINO, pruning libs).- Benchmarking stage: - Functional tests: run holdout eval, compare key metrics to baseline (top-1/precision/recall). - Latency/profiling: run hardware-specific microbenchmarks (p99, throughput) on representative infra (GPU/CPU/edge) using perf harness; measure memory. - Robustness tests: run adversarial attacks (FGSM, PGD via Adversarial Robustness Toolbox), corruption suite (ImageNet-C), and synthetic distribution shifts. - Regression tests: smoke tests for I/O, deterministic inference.- Decision engine: compare per-variant results against policy (thresholds, relative drop allowed, latency budget). Produces “approve/reject/needs manual review”.- Promotion and deployment: if approved, push artifact to model registry (MLflow/ModelDB) and artifact store (S3) with signed metadata; trigger canary rollout on K8s with traffic shaping.- Monitoring & rollback: runtime monitors (Prometheus + custom metrics) for accuracy drift, increased errors, latency spikes; automatic rollback if SLA breaches or anomaly detection triggers.

Lineage & provenance:- Store immutable artifacts in object storage with content-addressable IDs (SHA256). Save: - source model ID and git commit for training code - dataset snapshot hash / manifest and evaluation seed - transformation pipeline spec (operator versions, hyperparameters) - hardware/driver stack used for benchmarking - full result metrics and logs- Persist metadata in model registry (MLflow/Atlas) and a metadata DB for queries. Generate a model card and reproducible CI blueprint (container image digest, pipeline DAG).- Reproducibility: store container image digests and IaC (Terraform/K8s manifests). Provide “re-run” button in UI to reproduce compression and validation.

Example tests & gating rules:- Accuracy: new_model.acc >= baseline.acc - 1% absolute OR F1 degradation <= 0.5% relative- Latency: p99 <= target_p99 OR throughput >= target_throughput- Robustness: adversarial success rate increase <= 2x baseline; corruption mCE <= threshold- If any rule fails, mark variant rejected; if all fail, halt promotion and notify owners with artifact + logs.- Canary: route 5% traffic to new model for 24 hours; require no SLA violations and monitored metric stability before full promotion.

Trade-offs:- Strict thresholds reduce regression risk but may block beneficial compression. Use staged policy (automatic small drops allowed + manual review for borderline cases).- Hardware-aware transforms maximize gains but increase validation matrix size; mitigate via prioritized target hardware profiles.

This pipeline ensures automated, auditable compression with robust validation, safe promotion, and full lineage for compliance and reproducibility.

EasyBehavioral

65 practiced

Describe a hands-on AI project you built or contributed to. In your answer include: your role and responsibilities; a high-level architecture diagram described in text (components and data flow); the main tools, frameworks and infra you chose; key design decisions and trade-offs; how you measured impact and what you would do differently in hindsight. Be specific with component names, storage choices, and at least one non-trivial implementation challenge you solved.

Sample Answer

Situation: At my last company I led development of an AI-powered customer support summarization and routing system for enterprise customers (6-month project).

Task: I was the AI engineer responsible for model selection, training pipeline, deployment, and production monitoring.

Action:- Role & responsibilities: designed architecture, implemented data pipeline, fine-tuned models, built inference service, set up monitoring and cost controls. Coordinated with product and infra teams.- High-level architecture (text diagram): - Ingest: Kafka topics ← customer chat logs (stored raw in S3) - ETL: Airflow jobs → preprocess, anonymize, store features in Parquet on S3 and metadata in PostgreSQL - Training: EC2/GPU cluster managed by SageMaker (PyTorch + HuggingFace) reading Parquet from S3 - Model Registry: MLflow for versions and metrics - Serving: TorchServe on EKS behind Nginx + autoscaling; cache recent responses in Redis; async long-running summarization via Celery + SQS - Observability: Prometheus + Grafana (latency/throughput), ELK for logs- Tools & infra: AWS S3, SageMaker, EKS, Kafka, Airflow, PyTorch, HuggingFace Transformers, MLflow, TorchServe, Redis, PostgreSQL, Prometheus/Grafana.- Key design decisions & trade-offs: - Chose fine-tuning a 3B-parameter Transformer (HuggingFace) for control over summarization quality vs. using a smaller distilled model to save cost. Decision: start with 3B for quality, then distill to 300M for inference. - Use S3+Parquet for scalable, columnar storage and fast read in training; PostgreSQL for relational metadata and audit logs. - Real-time routing used a lightweight BERT-based classifier to meet 100ms p95 latency; heavy summarization was async.- Non-trivial implementation challenge solved: - Training instability and OOM on multi-GPU for long documents. Solution: implemented gradient accumulation + mixed-precision (Apex/torch.cuda.amp), dynamic bucketing of sequence lengths, and custom data sampler to balance document length — reduced memory footprint by 40% and stabilized training, enabling batch size effective 64.- Measurement & impact: - Metrics: Rouge-L/ROUGE for summarization, classifier F1 for routing, p95 latency, cost per 1k requests. - Results: summarization Rouge-L +12 points vs baseline, routing F1 0.92, average p95 latency 85ms for classifier, cost reduced 3x after distillation and autoscaling. Customer satisfaction NPS for support automation improved by 8 points in pilot.- What I'd do differently: - Invest earlier in dataset curation and active learning to reduce label noise (we later used co-teaching and human-in-the-loop). - From day one implement model distillation and quantization pipelines to shorten time-to-cost-effective inference. - Add continuous evaluation with synthetic adversarial examples to catch regressions.

Result: Delivered a production system that improved automation accuracy and reduced operator load; the architecture allowed iterative model improvements with controlled inference cost.

EasyTechnical

71 practiced

Design unit and integration tests for a data preprocessing function that normalizes timestamps, imputes missing values, and encodes categorical features. Provide example test cases you would write with pytest in prose: deterministic fixtures, edge cases (all missing, unseen categories), which components to mock, and what to assert in CI to prevent regressions.

Sample Answer

Start with deterministic fixtures- Create small pandas DataFrame fixtures in pytest using fixed seeds and explicit values (no random generation). Include rows with multiple timestamp formats ("2021-01-01T12:00:00", "01/02/2021 13:00"), NaNs in numeric and categorical columns, and rare/unseen category labels. Use pytest fixtures that return these DataFrames.

Unit test cases (focused, fast)1. Timestamp normalization:- Input: mixed formats + timezone-aware and naive timestamps.- Assert: output column is pandas.Timestamp dtype (or datetime64[ns, UTC] if you convert), all values normalized to UTC, and original order preserved.- Edge: all timestamps missing → function should return column of NaT and not crash.

2. Numeric imputation:- Input: numeric column with some NaNs and known distribution.- Assert: imputed values equal the chosen strategy (mean/median/fill_value). If mean, compute expected mean from fixture and check equality within tolerance.- Edge: all missing → strategy should use configured fallback (e.g., 0 or raise). Test both behaviors.

3. Categorical encoding:- Input: nominal column with seen categories and one unseen category not in the training mapping.- Assert: seen categories map to indices or one-hot columns as expected; unseen category maps to an explicit "unknown" bucket or raises if design requires.- Edge: all categories missing → encoding should produce all-zero one-hot or an "unknown" indicator per design.

4. Composed preprocessing unit (small end-to-end on fixture):- Run normalization → imputation → encoding sequentially.- Assert per-column invariants: no NaNs in numeric columns after imputation, categorical dtype matches expected, shapes match, and deterministic outputs.

Integration tests (pipeline-level)- Use a temp CSV fixture with multiple rows to simulate realistic data; run the entire preprocessing function/pipeline and compare the resulting DataFrame to a saved "golden" DataFrame stored in tests (committed artifact).- Assert: column order, dtypes, sample row values, and a checksum (e.g., MD5 of CSV output) match the golden file to catch regressions.

Components to mock- External resources: if preprocessing consults a remote lookup (e.g., timezone service, external vocab), mock those HTTP calls using responses or requests-mock and return deterministic data.- Randomness: if imputation uses stochastic methods (e.g., sampling), patch numpy.random or random to fixed seed or mock to deterministic output.- Heavy dependencies: if conversion uses database or cloud storage, replace with local/temp file fixtures.

What to assert in CI- All unit tests pass.- No change in golden-file checksum (fail CI if checksum differs).- pandas.testing.assert_frame_equal for golden comparisons with strict check_dtype=True and check_like=False; allow tiny numeric tolerances using rtol/atol where appropriate.- Coverage threshold for preprocessing module (e.g., >=90%).- Lint/static checks for nullable usage and type hints if required.

Additional best practices- Parametrize pytest tests to cover multiple impute strategies and encoding schemes.- Keep golden files small and human-readable; regenerate with a dedicated test script when intentional changes occur and require a review step on CI.- Document expected behavior for edge cases in docstring so tests reflect agreed design decisions.

MediumSystem Design

69 practiced

Design a CI/CD workflow for ML using GitHub Actions or Azure DevOps that runs unit tests and data validation on PRs, trains a small model for integration validation, builds a serving Docker image, pushes to a registry, and deploys to staging. Describe gating, artifact immutability, approval steps before production, and caching strategies to speed CI.

Sample Answer

Requirements:- Run unit tests + data validation on PRs- Train a small model for integration validation- Build immutable serving Docker image, push to registry- Deploy to staging automatically; require approval for production- Use GitHub Actions or Azure DevOps; optimize with caching

High-level workflow (applies to GitHub Actions or Azure DevOps pipelines):1. PR Checks (Gated) - Trigger: PR opened/updated - Steps: - Checkout + restore caches (pip/conda, wheel cache, dataset slice) - Run lint + unit tests - Run data validation (Great Expectations / custom checks) on a sampled dataset - Train a tiny “smoke” model (short epochs, small dataset) on CPU or small GPU; run basic metrics (accuracy, loss) - If any step fails, block merge (gated). - Runners: GitHub Hosted or self-hosted GPU pool for quicker training. - Cache: pip/conda packages, ML framework wheels, preprocessed sample dataset, and prebuilt Docker layers.

2. Merge → CI (Artifact creation, Immutable) - Trigger: merge to main - Steps: - Full unit/integration tests - Full data validation on production-like dataset - Train a candidate model (longer run on GPU cluster); log parameters/metrics to MLflow or similar; store model as immutable artifact (use DVC/MLflow + artifact store like S3 or Azure Blob with content-addressable path). - Build serving Docker image embedding model version (tag with sha + model artifact id), run integration tests against container (health checks / inference smoke tests). - Push image to container registry (Azure ACR or GitHub Container Registry). Image tags must be immutable (use content-addressable tags / digest). - Publish pipeline artifacts: model metadata, model artifact id, image digest.

3. Deploy to Staging (Automated) - Trigger: successful CI and image pushed - Steps: - Deploy image digest to staging K8s cluster (ArgoCD/Flux or Azure App Services / AKS with Helm) - Run end-to-end tests and canary traffic validation, performance and resource checks. - Collect telemetry and rollback if errors.

4. Production Promotion (Manual Approval + Gating) - After staging tests pass, create a release that includes immutable artifacts (model id, image digest). - Require manual approval in pipeline (Azure DevOps Environments approval or GitHub Environments protection) from ML engineer and Product/Compliance owners. - After approval, deploy the same immutable image and model artifact to production (no rebuilds).

Key design choices & reasoning:- Gating on PRs prevents bad code/data from merging.- Training a small model on PRs provides integration-level validation without full compute cost.- Immutable artifacts (model stored with content-addressable IDs; image pushed and referenced by digest) ensure reproducible deployments.- Manual approvals and environment protections before production guard against accidental promotions.- Use MLflow/DVC to track lineage and tie code commit → model id → image digest.

Caching strategies to speed CI:- Dependency cache: package manager cache (pip, conda) between jobs.- Docker layer cache: use registry-backed build cache (GitHub Actions cache or Azure Pipeline cache) or buildx with remote cache.- Dataset cache: store preprocessed sample datasets and features in cache or S3 to avoid repeated preprocessing.- Model/cache for pre-trained backbones: cache large pretrained weights (transformers) to avoid re-download.- Job-level artifact cache: reuse compiled wheels or C extensions.- Use matrix builds selectively; run expensive GPU jobs only on merge (not every PR) or use lightweight smoke training in PRs.

Example GitHub Actions notes:- Use workflow_dispatch and environment protection for prod deployment.- Use actions/cache for pip/conda and buildx cache for Docker.- Use artifacts/upload-artifact for model metadata.

Security / Ops:- Sign images, scan images (trivy) before pushing.- Limit secrets and use managed identities.- Monitor deployment metrics and enable automatic rollback thresholds.

This design balances fast feedback during PRs, reproducibility via immutable artifacts, and safe promotion to production with approvals.

Unlock Full Question Bank

Get access to hundreds of Hands On Projects and Problem Solving interview questions and detailed answers.

Join thousands of developers preparing for their dream job.