Java or Python Programming for Test Automation Questions

Strong programming skills in Java or Python with expertise in OOP principles (inheritance, polymorphism, encapsulation, abstraction), exception handling, collections, file I/O, and functional programming concepts. Ability to write clean, well-structured, maintainable code with appropriate design patterns. Understanding of common libraries and utilities for test automation.

HardSystem Design

66 practiced

Design a data-versioning system integrated with your test automation and model training pipelines that ensures reproducibility: content-addressable storage for files, manifest files per snapshot with file-level checksums, lightweight checkout for tests (partial snapshots), retention policies, and CI hooks to pin data versions. Explain how it speeds up tests and preserves provenance.

Sample Answer

Requirements and constraints:- Reproducibility: exact file-level provenance for any snapshot used in training or tests- Efficient test runs: lightweight partial checkouts for small test subsets- Integrity: content-addressable storage (CAS) + per-file checksums in manifests- Governance: retention policies, pinning via CI to freeze datasets used in runs- Integration: CI/CD and training/test pipelines, cloud/on‑prem storage

High-level architecture:1. Content-Addressable Storage (CAS): immutable object store (S3/GCS or on‑prem blob store) where objects are keyed by SHA256(content). Objects are deduplicated and immutable.2. Snapshot Manifests: for each dataset snapshot produce a manifest JSON: - snapshot_id (hash of manifest) - list of files: {path, size, sha256, cas_key, logical_tags} - metadata: creator, timestamp, schema, lineage (parent snapshot ids), retention_policy, dataset_version3. Index / Metadata DB: small relational DB (Postgres) mapping snapshot_id -> manifest, logical names, pins, access-control and retention policy states.4. Lightweight checkout service: client lib + server that resolves manifests, fetches only requested file entries (by path/glob/tags), streams from CAS, verifies checksum, caches locally.5. CI Hooks & Pinning: CI job step that, on successful test/train, records pinned snapshot_id and test-run id in metadata DB and attaches pin TTL or permanent pin.6. Retention Manager: background service enforces retention policies (respect pins), garbage-collects unpinned CAS objects.

Data flow:- Ingestion: ingest pipeline stores each file in CAS (returns cas_key), computes sha256 and records entry in manifest. Manifest hashed to produce snapshot_id and pushed to metadata DB.- Training/Test: pipeline requests snapshot_id (or logical name). Lightweight checkout can request partial subset (e.g., tests request only 1% stratified sample) using manifest filters; files streamed from CAS, verified.- CI: as part of test, CI pins the snapshot_id used for that run; promotes to permanent if release.

How it speeds up tests:- Partial snapshots: test harness requests only small subset of files (paths/tags), avoiding full dataset transfer.- Deduplication via CAS reduces network and storage costs: incremental snapshots only store new objects.- Local caching of CAS objects across CI agents prevents repeated downloads.- Verification up front (checksums) avoids flaky tests due to corrupted data.

How it preserves provenance:- Manifests are content-hashed, versioned, and immutable; any training or test run references snapshot_id which deterministically maps to exact file checksums and CAS object keys.- Lineage fields link parent snapshots and preprocessing transforms; manifests can embed commit hashes for preprocessing code or transformation DAG ids.- CI pinning ensures retention of data used for a given model version; combined with model artifact pins (model binary + snapshot_id) yields full reproducibility.

Scalability and optimizations:- Shard CAS by hash prefix; use CDN or regional caches for heavy-read workloads.- Use range/parallel downloads and streaming for large files; support chunked objects with chunk-level checksums to enable parallel fetch and partial-file checkout.- Store small metadata in DB, large manifests in CAS and reference by manifest hash for atomicity.- Use GC with reference counting from manifests + pins to avoid accidental deletion.

Trade-offs:- Using CAS + manifests adds operational complexity vs simple filesystems, but gives strong guarantees and dedupe.- Strong consistency for manifests requires careful transactional publishing; use MVCC patterns in metadata DB.- Retention GC policy must balance cost vs reproducibility; CI pins mitigate accidental GC.

Example manifest snippet (JSON):{ "snapshot_id": "sha256:...", "created_by": "ingest-service@2025-11-01T12:00Z", "files": [ {"path":"train/0001.csv","size":12345,"sha256":"...","cas_key":"sha256:...","tags":["classA"]}, {"path":"meta/schema.json","size":234,"sha256":"...","cas_key":"sha256:..."} ], "parents":["sha256:..."], "retention":"90d"}

This design gives deterministic reproducibility (manifest -> checksums -> CAS objects), fast, cost-effective testing via partial checkouts and caching, and operational controls (pinning, retention, provenance linkage) needed for robust ML lifecycle.

MediumTechnical

79 practiced

Implement a memory-efficient iterator in Python (generator) or Java (Stream/Iterator) that parses and yields records from a very large newline-delimited JSON (ndjson) file. Include how you'd unit test parsing errors and how you'd test backpressure or slow consumers.

Sample Answer

Approach: use a streaming generator that reads the file line-by-line (constant memory), parses each JSON line lazily with json.loads, yields parsed records or structured errors. For robustness, allow a configurable error handler (skip, raise, collect). To simulate backpressure, consumer can sleep or process slowly; tests use threading/queues or asyncio to ensure iterator doesn't preload data.

Python implementation:

python

import json
from typing import Callable, Iterator, Optional, TextIO

def ndjson_stream(file: TextIO,
                  error_handler: Optional[Callable[[int,str], None]] = None) -> Iterator[dict]:
    """
    Lazily yield parsed JSON objects from an open file (ndjson).
    error_handler(line_no, raw_line) -> None: called on parse error.
    """
    for i, raw in enumerate(file, start=1):
        raw = raw.rstrip("\n")
        if not raw:
            continue
        try:
            yield json.loads(raw)
        except json.JSONDecodeError as e:
            if error_handler:
                error_handler(i, raw)
            else:
                raise

Unit testing parsing errors:- Provide temp files with valid and invalid lines.- Test three behaviors: 1) default: expect JSONDecodeError raised on bad line. 2) custom handler that appends errors to a list and iterator continues; assert collected errors and yielded valid records. 3) empty/whitespace lines are skipped.

Example pytest snippet:

python

def test_error_handler(tmp_path):
    p = tmp_path / "data.ndjson"
    p.write_text('{"a":1}\n{bad json}\n{"b":2}\n')
    errors=[]
    with p.open() as f:
        out = list(ndjson_stream(f, lambda ln,raw: errors.append((ln,raw))))
    assert out == [{"a":1},{"b":2}]
    assert errors[0][0] == 2

Testing backpressure / slow consumers:- Ensure generator reads only on iteration by wrapping file with a proxy that tracks read calls.- Use a SlowConsumer that sleeps between next() calls; assert file.read was called incrementally (not fully loaded).- Alternatively, use threading + queue: producer thread iterates generator and puts items on a bounded queue; consumer sleeps to simulate slow processing and ensure producer blocks on queue.put (demonstrates backpressure handling when integrated into pipeline).

Key points:- Memory O(1) aside from single-line parse; time O(n) overall.- Edge cases: very long lines (may need streaming JSON parser), encoding issues, Windows CRLF.- For production ML pipelines consider batching, backoff, and monitoring parsing error rates.

MediumTechnical

93 practiced

You have a stratified sampler that builds training batches to address class imbalance. Describe unit and integration tests to ensure stratification is correct: tests for rare classes, very small datasets, deterministic sampling with a seed, and behavior when a class has fewer examples than the batch requires.

Sample Answer

Approach summary:- Test at unit level that the sampler returns batches with the requested per-class counts (or follows policy: oversample-with-replacement, allow smaller batch, or raise). - Integration tests run sampler end-to-end with a DataLoader-like loop to validate across many batches: coverage of rare classes, reproducibility with seed, and behavior on tiny datasets.

Unit tests (pytest-style) — key cases and assertions:

1) Rare class appears in every epoch (oversample policy)- Create dataset with classes = {0:100, 1:1} and batch_size=10 with stratified proportions [0.9,0.1] (so class 1 target per batch ≈1).- Assert: in each batch, count(class1) >= 1 (or equals 1 if strict), and no IndexError.

python

def test_rare_class_included_oversample():
    dataset = make_dataset({0:100,1:1})
    sampler = StratifiedSampler(dataset, class_props={0:0.9,1:0.1}, batch_size=10, oversample=True)
    for batch in sampler.batches(n_batches=20):
        assert batch.count_class(1) >= 1

2) Very small dataset (smaller than one batch)- dataset size 5, batch_size=8. Policy: allow smaller final batch or pad by oversampling.- Assert behavior matches policy and no crashes.

python

def test_small_dataset_allows_partial_batch():
    dataset = make_dataset({0:3,1:2})
    sampler = StratifiedSampler(..., allow_partial=True)
    batches = list(sampler.batches(n_batches=1))
    assert len(batches[0]) == 5

3) Deterministic sampling with seed- Fixed dataset and seed should produce identical sequence of batches across runs.

python

def test_seed_reproducibility():
    ds = make_dataset({...})
    s1 = StratifiedSampler(ds, seed=42)
    s2 = StratifiedSampler(ds, seed=42)
    assert list(s1.batches(10)) == list(s2.batches(10))

4) Class has fewer examples than required per batch (no-oversample policy)- dataset {A:2, B:100}, batch targets require 4 of A. If oversample=False -> expect an error or fewer As per policy.- Test both policies.

python

def test_insufficient_examples_raises_or_truncates():
    ds = make_dataset({'A':2,'B':100})
    with pytest.raises(ValueError):
        StratifiedSampler(ds, class_props={'A':0.5,'B':0.5}, batch_size=8, oversample=False).batches(1)
    s = StratifiedSampler(ds, ..., oversample=True)
    batch = next(s.batches(1))
    assert batch.count_class('A') == 4  # oversampled

Integration tests- Large simulation: run for many batches, compute empirical per-class frequency and compare to target proportions within tolerance (e.g., ±2%).- Ensure coverage of all examples including rare ones across epoch (no starvation).- Test with DataLoader pipeline (collate_fn) to ensure indices -> examples mapping preserved.

Edge checks and metrics to assert- No IndexError or infinite loops.- Per-batch class counts equal target or within allowed rounding rules.- Across epoch: variance of class frequency small, and every example either seen or appropriately repeated per policy.- Random seed affects ordering but not class proportions.

Why these tests:- They validate correctness (counts), robustness (tiny datasets, rare classes), reproducibility (seed), and policy compliance when data is insufficient. Include both deterministic unit tests and stochastic integration checks with statistical tolerances.

MediumSystem Design

72 practiced

Design a CI test harness for ML pipelines that runs unit tests, deterministic integration tests, and performance benchmarks. Requirements: support parallel execution, resource limits (CPU/memory/GPU), caching of intermediate artifacts, reproducible containerized environments, and clear artifact storage for test results. Sketch the architecture and a scheduling policy for tests.

Sample Answer

Requirements (clarify): run unit tests, deterministic integration tests, and performance benchmarks; parallelism; enforce CPU/Memory/GPU limits; cache intermediate artifacts (datasets, preprocessed features, model checkpoints); reproducible containerized environments; store test artifacts/results with provenance.

High-level architecture:- Client/API: submit test job + manifest (type: unit/integration/benchmark), resources, dependencies, container image hash.- Orchestrator/Scheduler: central controller that queues jobs, enforces policies, dispatches to runners.- Runner pool (Kubernetes cluster): each runner is a Pod with a single-job lifecycle using the provided container image (immutable image or image+entrypoint). Use Kubernetes resource requests/limits and device plugins for GPUs.- Shared artifact store: object storage (S3) for datasets, caches, model checkpoints, and test results. Use content-addressable keys (hashing) for cache lookup.- Metadata DB: stores job metadata, provenance (git commit, image digest, random seeds), test status.- Cache service: index mapping input hashes -> artifact locations; supports TTL and invalidation.- CI UI / API: show logs, metrics, pass/fail, benchmark time series, diff vs baseline.

Data & flow:1. Developer submits manifest pointing to commit, test type, resource hints, seed, and dependencies.2. Orchestrator computes deterministic job key (commit + manifest + seed). If cache hit for integration (same preprocess/model) return cached artifact or skip steps.3. Scheduler assigns to runner matching resource constraints and GPU availability.4. Runner pulls image (image digest), mounts dataset object store via volume or streaming, runs tests; unit tests run quickly, integration tests run deterministic pipelines with fixed seeds and mocked non-deterministic services; benchmarks run under controlled load and system isolation.5. Runner uploads logs, artifacts, and metrics to artifact store and updates metadata DB.

Scheduling policy:- Priority tiers: unit (high, short), integration (medium), benchmark (low, long).- Bin-packing + fair-share: prefer packing small short jobs to utilize leftovers; long GPU benchmarks scheduled on dedicated GPU nodes to avoid interference.- Resource-aware backpressure: reject or queue if cluster lacks GPU/ram; autoscaler triggers node pool scaling for GPU jobs.- Cache-aware scheduling: co-locate jobs with required cached artifacts when possible (label nodes with hot caches), or prefetch to node-local cache.- Quotas per-team and per-commit to avoid resource starvation.- Preemption rules: preempt benchmarks for urgent integration runs; low-preemptibility for unit tests.

Reproducibility & determinism:- Use immutable container images by digest; record random seeds, library versions, GPU driver versions, and hardware info in metadata.- Run deterministic integration tests by freezing data slices, mocking external services, and using deterministic ML ops (fixed random seeds, deterministic ops flags).- Snapshot environment with a lightweight image-builder (CI job to rebuild image) and store image digest in metadata.

Performance benchmarks:- Run multiple trials, collect percentiles and resource usage metrics (CPU/GPU utilization, memory, throughput, latency).- Store time-series in monitoring (Prometheus + long-term TSDB) and compare against historical baselines; fail on regression thresholds.

Trade-offs:- Strong determinism may require disabling some performance optimizations (non-deterministic cuDNN ops). Provide opt-in flags.- Prefetching caches improves latency but increases storage and complexity.- Using Kubernetes simplifies resource limits and autoscaling but adds orchestration complexity.

This design ensures parallel, resource-constrained, reproducible ML CI with caching and clear artifact provenance.

MediumTechnical

116 practiced

Discuss trade-offs between using mocks/stubs versus real components in integration tests for ML systems. Cover aspects of speed, confidence, flakiness, maintenance, and examples such as feature stores, model registries, and GPU inference. Provide guidance for when to prefer each approach.

Sample Answer

Start by framing the goal: integration tests should give confidence that components work together under realistic conditions while remaining fast, stable, and maintainable. Below I compare mocks/stubs vs real components across key dimensions and give practical guidance with ML examples.

Speed- Mocks/stubs: Fast — no network, DB, or GPU warmup. Good for CI unit/integration gates.- Real components: Slower — data retrieval, feature recomputation, model load and GPU initialization increase test time.

Confidence- Mocks/stubs: Lower end-to-end confidence because they can’t catch integration issues (schema drift, serialization, auth, performance).- Real components: Higher confidence: surface mismatches between feature store schemas, registry metadata, or inference container behavior.

Flakiness- Mocks/stubs: Generally less flaky (deterministic), easier to debug.- Real components: More flakiness from network hiccups, transient cloud quotas, non-deterministic hardware timing (GPU memory), and external service rate limits.

Maintenance- Mocks/stubs: Higher maintenance when interfaces evolve — tests must mirror realistic behavior or risk false positives; but faster to run and simpler to version-control.- Real components: Lower test-code churn for API contracts, but require infra upkeep (test instances of feature stores, registries, GPU availability) and cost.

Concrete examples- Feature store: Mocking feature responses is fine for model logic tests. For schema/feature-engineering regressions, run periodic tests against a real staging feature store to catch drift and freshness issues.- Model registry: Use lightweight stubs to simulate model metadata for fast CI. Use real registry integration tests when validating deployment pipelines, version resolution, or artifact downloads.- GPU inference: Mock CPU-based inference or small synthetic models for most CI. Use real GPU-backed integration tests (nightly or pre-release) to validate performance, memory, and CUDA compatibility.

Practical guidance (when to prefer each)- Prefer mocks/stubs when you need fast, deterministic checks in CI, are validating business logic, or when real infra is expensive/slow.- Prefer real components when validating contracts, data freshness, serialization, deployment pipelines, or performance characteristics (latency, memory).- Hybrid strategy: keep most tests mocked for CI speed; add a targeted suite of integration tests against real staging infra (run less frequently: nightly, pre-merge for release, or on-demand). Use chaos/contract tests to simulate failures and validate retries/timeouts.- Version and isolate: use pinned test fixtures, dedicated staging resources, and orchestration (containers, Terraform) so real-component tests are reproducible and less flaky.

Summary rule: use mocks for speed and determinism; use real components to validate contracts, performance, and systems behaviour. Combine both with a risk-driven cadence: fast gates with mocks, deeper confidence with scheduled real-component tests.

Unlock Full Question Bank

Get access to hundreds of Java or Python Programming for Test Automation interview questions and detailed answers.

Join thousands of developers preparing for their dream job.