Netflix SDET (Mid-Level) Interview Preparation Guide

Situation & goal (brief)
I’d design a single-pane Automation ROI & Health dashboard that surfaces coverage, reliability, speed, cost, and business impact so teams can prioritize automation investment and reduce manual effort.

Key metrics (definitions)

• Automation coverage: % of testable requirements or user journeys automated (by feature/component).
• MTTD (mean-time-to-detect): avg time from regression-introducing commit to test detection.
• Test-run latency: median and P95 time from pipeline start to test completion.
• Percent flaky tests: % of tests with non-deterministic outcomes in last 30/90 days.
• Maintenance cost per test: estimated dev hours * hourly rate per test per month.
• Deflection rate: number of manual test executions avoided / total manual attempts pre-automation.

Data sources

• CI/CD systems (Jenkins/GitHub Actions/GitLab) for run times, pass/fail, timestamps.
• Test management (TestRail, Zephyr) for requirement mapping and manual baseline.
• Source control (git) for commit metadata and blame.
• Issue tracker (Jira) for bugs tied to tests.
• Time tracking / engineering staffing tool for maintenance hours.

Visualizations & layout

• Top row: KPIs (coverage, deflection, maintenance $) with trend sparkline.
• Heatmap by component: coverage vs. flakiness (red = high flake, low coverage).
• Time series: MTTD and test-run latency (median + P95).
• Flaky test table: name, failure rate, rerun ratio, owner.
• Cost breakdown: maintenance cost per suite and ROI curve (cost vs. manual hours saved).

Alert thresholds & rules

• Percent flaky tests > 5% org-wide or >10% for critical suites -> P1 alert.
• MTTD > 48 hours for failures linked to master -> escalate to release lead.
• Test-run P95 latency increase > 2x baseline -> performance ticket for infra.
• Coverage drop > 5% for a component in 7 days -> assign automation owner.

Concrete actions & ownership

• On flaky alert: assign to test owner; create triage ticket within 24h; quarantine to “unstable” group if >3 consecutive failures.
• On MTTD breach: SDET on-call investigates pipeline gating and adds fast-fail smoke tests.
• On latency breach: infra SDET audits parallelism, container sizing, and shard strategy.
• Coverage regression: feature dev + SDET pair to add missing tests in next sprint; track via Jira epic.
• Monthly ROI review: QA manager reviews deflection and maintenance cost; reprioritize suites with low ROI (< 6 months payback).

Why this works
Metrics map directly to developer/tester behavior, are actionable (owners + SLAs), and balance technical health with business ROI so automation becomes measurable and investable.

Direct answer: GPU-based training steps add real, ML-specific sources of nondeterminism on top of every general-purpose cause covered throughout this topic, non-deterministic GPU kernel execution order, floating-point accumulation differences across hardware, and CI-vs-local GPU availability/driver differences, so the detection/reduction/isolation approach needs to explicitly separate "is this ordinary CI flakiness" from "is this ML-specific numerical nondeterminism" before applying a fix.

Structured elaboration

Step-by-step detection: (1) first apply the GENERAL flakiness triage from earlier in this topic, is this CI-only (environment difference) or does it also occur locally under similar conditions (a genuine, reproducible numerical issue)? (2) if CI-only, check whether local and CI environments actually have COMPARABLE GPU hardware/drivers, a common and easily-overlooked root cause is CI running on different GPU hardware (or CPU-only) than local development, which can produce genuinely different floating-point results, not flakiness in the traditional sense at all, but a real environment mismatch that happens to look like flakiness; (3) if it reproduces under matched conditions, suspect GPU-specific nondeterminism sources (see below) rather than the general root-cause taxonomy this topic otherwise covers.

GPU/ML-specific nondeterminism sources: many GPU kernel operations (certain reduction operations, some convolution implementations) are NOT guaranteed to produce bit-identical results across runs even with the SAME inputs and SAME seed, because the order floating-point operations are summed/executed in can vary with GPU scheduling, and floating-point addition is not strictly associative; this is a genuinely different category of nondeterminism than anything, since it isn't a bug in the test or a race condition to eliminate, it's an inherent property of some GPU operations that must be explicitly worked around, not merely detected and fixed the usual way.

Tooling to reduce it: most major ML frameworks expose an explicit "deterministic mode" flag (forcing deterministic kernel implementations where available, at some performance cost) that should be enabled for TEST runs specifically (even if disabled in production training for performance); combined with fixing all relevant seeds (framework RNG, NumPy, Python's own random, and CUDA's own RNG state specifically, a seed source easy to forget since it's separate from the framework's main RNG).

CI strategies: run ML/GPU tests on CI hardware that MATCHES production/development GPU type where feasible (avoiding the CPU-vs-GPU or different-GPU-generation mismatch entirely); where matched hardware isn't feasible, explicitly tag and treat GPU-dependent tests as a SEPARATE tier with wider tolerance bounds (see below) rather than holding them to the same bit-exact standard as CPU-only tests.

Rewriting tests to be deterministic or resilient: for tests asserting on TRAINING OUTCOMES (loss value, accuracy), replace exact-equality assertions with TOLERANCE-based assertions (assert abs(actual_loss - expected_loss) < tolerance), since even with determinism mode enabled, some ML numerical operations retain minor, expected variation across hardware; choosing the tolerance itself needs to be informed by ACTUAL measured variation (run the same test many times, even under determinism mode, and measure the real spread) rather than an arbitrary guess, the same measurement-first discipline covered for timeout calibration elsewhere in this topic. For tests that don't need to verify exact numerical correctness (testing that TRAINING RUNS without crashing, for instance, rather than testing the exact resulting loss value), separate that concern into its own test that doesn't carry the numerical-tolerance question at all.

Worked example: a training-step test asserting final_loss == 0.4231 (exact equality) flakes intermittently in CI. Investigation confirms CI runs on a different GPU generation than local development. Enabling the framework's deterministic-mode flag and fixing all RNG sources (framework, NumPy, CUDA) reduces but does NOT eliminate variation entirely (a genuine property of some GPU operations, not a bug to fully eliminate); running the test 50 times under deterministic mode measures an actual observed spread of the final loss value within ±0.0015 of the target. Replacing the exact-equality assertion with abs(final_loss - 0.4231) < 0.003 (a measured, justified tolerance with margin, not an arbitrary round number) eliminates the flakiness while still meaningfully verifying training correctness.

Trade-offs & pitfalls: enabling deterministic mode typically has a real PERFORMANCE cost (some deterministic kernel implementations are measurably slower than their nondeterministic counterparts), a genuine trade-off worth being explicit about rather than treating determinism as a free correctness upgrade; for a large training-step test, this could meaningfully slow the test suite, worth weighing against just adopting appropriately-measured tolerance-based assertions instead, which don't require the performance cost of full determinism mode at all.

Direct answer. At scale, Screenplay's advantage over POM is compositional: Tasks and Interactions recombine across actors and flows the way functions recombine in ordinary code, while POM's per-page classes tend toward duplicated orchestration logic once enough flows share partial overlap - but that advantage costs a steeper learning curve and a smaller tooling/community ecosystem, so "at scale" is a necessary condition, not a sufficient one.

Structured elaboration, across the named dimensions:

• Expressiveness: Screenplay reads as user intent ("Actor attempts to X"); POM reads as page interaction ("call this page's method"). At scale, the former holds up better because a business-flow-level bug report ("checkout fails for expired cards") maps directly onto a Task name, not a chain of page-method calls a reader has to reconstruct.
• Code reuse: Screenplay's Interactions/Tasks compose (a Task can be built from smaller Tasks reused across many flows); POM's reuse mechanism is page-object composition, which works well within one page but gets awkward across a flow that spans many pages.
• Maintainability: both are maintainable if disciplined; Screenplay tends to concentrate change in one Task when a business rule changes, while a page-spanning POM change can require touching several page classes at once.
• Composition: Screenplay is built around composition from the start (Abilities + Tasks + Questions); POM has no first-class composition primitive beyond object composition of page/component classes, so teams invent their own conventions for it.
• Learning curve: POM wins decisively - most engineers already know "a class per page." Screenplay requires learning four new concepts (Actor/Ability/Task/Question) before writing a useful test.
• Tooling ecosystem: POM is supported natively or trivially by every framework's examples/tutorials; Screenplay's tooling (e.g. Serenity/JS, Boa Constrictor) is real but smaller, so hiring and onboarding lean toward POM by default.

Worked example. Screenplay preferable: a large e-commerce platform with distinct actor types (guest, registered customer, wholesale buyer) all performing overlapping-but-not-identical checkout Tasks - the Task/Actor model lets WholesaleBuyer and RegisteredCustomer share AddItemToCart and ApplyPromoCode Tasks while differing only in PayOnAccount vs PayWithCard. POM preferable: a marketing site with one actor type and mostly independent, single-page flows (contact form, newsletter signup) - there is no cross-flow composition benefit to buy, so POM's lower ceremony wins outright.

Trade-offs and pitfalls. The failure mode in practice is adopting Screenplay for its reputation rather than its actual fit: a team with one actor type and simple flows that adopts Screenplay anyway spends real ramp-up time on Actor/Ability/Task vocabulary for zero compositional payoff, since there is nothing complex to compose.

Direct answer

GROW is a four-stage, question-led coaching structure: Goal (what success looks like), Reality (the current state), Options (possible paths forward), and Way forward (specific commitments). Applied to a 1:1 with someone who wants to grow a specific skill, it turns a vague aspiration into a concrete next step, and the same question-led habit also works inside a work review, not only a scheduled conversation.

Walking through the four stages

• Goal. Get specific: "What would 'better at this' actually look like, concretely, and how would you know it happened?"
• Reality. Surface the current state without judgment: "Tell me about a recent situation where this was hard, what made it hard?"
• Options. Generate paths rather than prescribing one: "What could you try next, and who or what could help?"
• Way forward. Get a specific, small commitment: "Which one thing will you actually do before we talk again, and what support do you need from me?"

Socratic questioning is the companion technique that runs through all four stages: instead of stating the answer, ask a question that leads the person to notice the gap themselves ("what did you expect to happen there, versus what actually happened?"). It works well when there's time to let someone arrive at the insight; it works poorly when someone is genuinely blocked and just needs the direct answer.

Extending this into reviewing someone's work

The same question-led approach makes a review of someone's work (code, a document, a design, an analysis) constructive rather than purely corrective. Concrete techniques: a review template that separates "must fix" from "worth considering" from "just for your awareness," so feedback doesn't read as one undifferentiated pile of criticism; annotated examples that show a better version alongside the original with a short reason, not just a comment naming the problem; and a Socratic question left in the review itself ("what happens here if this is empty?") instead of stating the bug outright, when the goal is teaching and there's no urgency forcing a direct fix.

Worked example

In a 1:1, a mentee said they wanted to get better at making structural decisions independently instead of always checking first. Goal: they described what "independent" would look like in practice (making a defined class of calls without asking). Reality: walking through a recent case, they could explain their reasoning but hadn't trusted it enough to act without confirmation. Options: they proposed trying it on a low-stakes decision first and reviewing the reasoning after the fact rather than before. Way forward: they committed to making the next reversible decision on their own and bringing the reasoning to the following session, with an explicit offer of support if it went wrong.

Trade-offs and pitfalls

A common mistake is treating GROW as a rigid script and marching through all four stages regardless of what the person actually needs that day. A stronger approach holds the structure loosely: skip Reality if it's already obvious, compress stages under time pressure, and know when the moment calls for direct answers instead of more questions, especially if something is safety-critical or urgent. Inside reviews specifically, overusing Socratic questions when someone is genuinely stuck can read as withholding rather than teaching, so it's worth pairing questions with a clear direct answer once the teaching moment has been made.

Approach overview (SDET lens)

I’d treat environment parity as testable, observable, and codified. Goal: dev ≈ staging ≈ prod in behavior that matters to tests while acknowledging acceptable scale differences.

Configuration-as-code

• Store all infra and app config in Git (Terraform/CloudFormation, Helm/Kustomize, env-specific values files).
• Enforce PR reviews and CI linting (terraform validate, kubeval).
• Example: single Helm chart with values-{dev,staging,prod}.yaml and a CI job that diffs rendered manifests.

Feature flags

• Use a flag service (LaunchDarkly, Unleash). Flags default-off in prod; tests toggle flags via API.
• SDET writes automated tests that exercise both flag-on and flag-off paths and include rollback checks.

Scale differences & testing

• Model scale via synthetic load: smaller instance counts but same resource types and autoscaling rules.
• Use contract tests, integration tests in CI with staging-like config, and periodic canary runs against prod-like clusters using synthetic traffic.

Surface parity gaps automatically

• Build a parity-check pipeline:
  • Compare rendered manifests and effective env variables across environments; report diffs.
  • Health and feature probes that call endpoints and flag API; fail CI on critical mismatches.
  • Runtime telemetry checks: alert if service versions, middleware, or schemas differ.
• Expose results in PRs and Slack, and create ticket when parity score falls below threshold.

Why this works

• Codifies intended state, tests behavioral differences, and automates detection so engineers get actionable signals before user impact.

Situation & goal
I’d act as the SDET owner for the failing test: determine why it’s flaky, reproduce it, and decide whether to fix the test, harden environments, or file a product bug.

1) Analyze historical runs

• Query CI test history (buildkite/jenkins/github actions): failure rate, time-of-day, branches, re-run outcomes.
• Collect artifacts: failing logs, screenshots, heap dumps, network traces, test timestamps.
• Look for patterns: correlated infra changes, recent code commits, or resource spikes (CPU/memory) reported by monitoring.

2) Reproduce locally / in CI

• Run the test repeatedly with same seed/config and increased verbosity; add retries and isolation (single test file, clean DB).
• Reproduce in a dedicated CI job with increased logging, tracing, and environment parity (same container image, DB snapshot, feature flags).
• If timing-related, introduce controlled delays or use time mocks to reproduce.

3) Root-cause classification & collaboration

• If logs/stack show assertion or API mismatch -> likely product bug: create a clear bug with reproduction steps and traces.
• If failures vary by timing, order, or shared state -> test logic issue: flaky assertions, improper waiting, race conditions; I’d propose fixes (explicit waits, deterministic mocks, reset state).
• If failures align with environment metrics (network timeouts, DB connection limits) -> environment instability: collaborate with SRE to pin versions, increase resources, or stabilize network.
• Work with developer to pair-debug, share reproducer, and land either a test fix or product patch. Add an interim test quarantine or stricter CI gating until resolved.

4) Preventive steps

• Add better assertions/logging, deterministic fixtures, and CI flakiness dashboards; introduce non-flaky guidelines and run long-running stress to catch timing issues early.

Direct answer

Monitor the business-critical user journeys, not just individual endpoints: track success rate for flows like login, checkout, or search, run synthetic checks that replay those flows continuously, and compare the new release's error and journey-success rates against the previous release's baseline using a statistical comparison rather than a fixed threshold, so a noisy small sample doesn't page someone unnecessarily.

Structured elaboration

What to monitor post-release

• Journey-level success rate for the flows that matter to the business (not raw endpoint 200/500 counts alone, since a flow can "succeed" at the HTTP layer while still failing the user, e.g. a checkout that returns 200 but never charges the card).
• Synthetic canaries: scripted runs of critical flows on a schedule, independent of real user traffic, so you get signal even during low-traffic windows.
• Client-side signal where relevant: crash-free session rate, JS error rate, rage-clicks/dead-clicks if you have RUM (real user monitoring) instrumentation.
• Canary/cohort comparison: route a percentage of traffic to the new release and compare its error rate against the stable baseline cohort, rather than comparing today's absolute number to a fixed historical number.

How to alert without triggering on noise
A newly-deployed canary usually serves far fewer requests than the baseline, so comparing raw percentages directly is misleading: a small sample can show a "higher" error rate purely from chance. Use a two-proportion significance test (comparing whether two observed rates are different by more than sampling noise would explain) before paging, and require the difference to be both statistically significant and above a minimum practical size (e.g., at least a few percentage points), so a "significant" but trivial 0.01% difference doesn't fire.

Feeding it back into testing
Every regression that monitoring catches but tests missed is a test-escape: the incident review should produce a new automated test (unit, integration, or synthetic) that would have caught it, and the monitoring signal that caught it (which journey, which metric) should be reviewed to see if it needs to trigger earlier in the pipeline (e.g., as a pre-production canary gate) next time.

Worked example

Suppose a release goes to a 2,000-request canary cohort while the stable baseline serves 10,000 requests in the same window. Baseline shows 40 errors, canary shows 25 errors.

Pooled error rate across both cohorts:

Two-proportion z-statistic:

Plugging in: standard error $\approx 0.001798$, so $z \approx \frac{0.0085}{0.001798} \approx 4.73$. A z-score that far from zero (well above the ~1.96 threshold for a 95% confidence two-sided test) means this isn't noise: the canary's error rate is a genuine regression, not sampling variance, and it should page. If the canary had instead shown, say, 3 errors out of 2,000 (a rate of 0.15%, actually below baseline), the same test would correctly stay quiet even though 3 is a "different number" from 40.

Trade-offs & pitfalls

• Comparing raw percentages without accounting for sample size is the single most common cause of false canary alarms early in a rollout, when the canary cohort is still small.
• Journey-level monitoring requires actual investment in defining and maintaining synthetic scripts; if they aren't kept in sync with real user flows, they silently stop testing what matters.
• Overly sensitive significance thresholds (alerting on any statistically significant difference, however tiny) reintroduce the alert-fatigue problem from a different angle; pair the significance test with a minimum effect-size floor.
• A test-escape process without an actual ticket/tracking loop back to the test suite tends to catch the same class of regression repeatedly.

Direct answer
Line and branch coverage answer "was this code executed," not "would a wrong answer have been caught," so a suite can reach 100% of both while never noticing a broken boundary condition. Three additional metrics close that gap: mutation score (did the suite actually notice when the code was deliberately broken), boundary-condition coverage (were the specific edge values exercised, not merely the surrounding code path), and property-assertion coverage (were the declared invariants actually checked by a test, not just described in a specification document).

Structured elaboration

• Mutation score: the fraction of deliberately injected code mutants (small, systematic changes like swapping a relational operator or a boundary constant) that the suite "kills," meaning at least one test fails against the mutated code. $\text{mutation score} = \frac{\text{mutants killed}}{\text{total mutants}}$. This measures the STRENGTH of the assertions, not whether the code ran.
• Boundary-condition coverage: the fraction of boundary values identified by boundary value analysis (testing the values immediately below, at, and immediately above each bounded input) that appear as an explicit test input, distinct from line coverage since a single typical-case test can reach 100% of a bounded function's lines while touching none of its actual boundary values.
• Scenario coverage: the fraction of enumerated business use-case scenarios exercised, relevant whenever the same code path serves multiple scenarios with different correctness expectations that line coverage cannot distinguish between.
• Property-assertion coverage: the fraction of declared invariants or properties (from property-based or contract testing) that have at least one test actively checking them, versus properties that exist only in a specification document with nothing enforcing them.

Worked example (executed): why 100% line and branch coverage cannot distinguish a weak suite from a strong one

def is_adult(age):
    return age >= 18

mutants = {
    'age>18': lambda age: age > 18,
    'age<=18': lambda age: age <= 18,
    'age>=17': lambda age: age >= 17,
    'age>=19': lambda age: age >= 19,
    'age==18': lambda age: age == 18,
    'not(age>=18)': lambda age: not (age >= 18),
}

def mutation_score(suite):
    killed = set()
    for age in suite:
        original = is_adult(age)
        for name, mutant in mutants.items():
            if mutant(age) != original:
                killed.add(name)
    return killed, len(killed) / len(mutants)

for suite in ([17, 20], [17, 18, 19]):
    killed, score = mutation_score(suite)
    print(f"suite={suite} killed={sorted(killed)} score={score:.3f}")

This single-expression function has exactly one branch; any input reaches 100% line and branch coverage. Six standard mutation operators applied to it (relational-operator replacement and boundary-constant replacement): age > 18, age <= 18, age >= 17, age >= 19, age == 18, and not (age >= 18). Running the harness above against two different test suites:

weak_suite fails to kill age > 18 and age >= 19, both of which only diverge from the original function exactly at age = 18 or age = 19, values the weak suite never tests. strong_suite, derived directly from boundary value analysis of the single threshold at 18, kills every mutant. Both suites reach identical line and branch coverage; mutation score is the metric that actually distinguishes them.

Instrumenting this in practice
Track mutation score and boundary-coverage percentage as first-class metrics alongside line and branch coverage, not as a replacement for them, and gate merges on a minimum mutation score specifically for high-risk modules rather than project-wide, since mutation testing is computationally expensive (it reruns the full suite once per mutant). Feed both numbers, per module, into the same dashboard that already reports line and branch coverage, trended over time rather than as a single snapshot, so a module whose boundary coverage or mutation score silently drops, for example because a boundary-focused test was deleted during an unrelated refactor, becomes visible on that dashboard before it causes an incident, rather than being caught only in hindsight.

Trade-offs & pitfalls
Mutation testing's computational cost (a full suite re-run per mutant) means it is typically run on a schedule or targeted at high-risk modules rather than on every commit, a real operational trade-off rather than a flaw in the metric itself. An "equivalent mutant," a mutant that is semantically identical to the original code despite a textual change (for example, replacing a multiplication by 1 with the bare value), can never be killed no matter how strong the suite is, and a team chasing 100% mutation score without accounting for equivalent mutants wastes effort chasing an unreachable target. Boundary-condition coverage also inherits whatever gaps exist in the underlying boundary value analysis, since it depends on someone having enumerated the boundaries by hand first; unlike mutation testing, it does not discover boundaries you failed to anticipate on its own.

Situation & goal (brief)
I’d build a self-healing locator system that minimizes flaky UI tests by collecting rich element fingerprints, trying deterministic fallbacks, and using ML similarity when heuristics fail — with safe validation and observable metrics before auto-applying fixes.

Data to collect per element

• DOM attributes: id, name, class list, aria-* attributes, tag, input type
• Full and relative DOM path (CSS / XPath) with indices
• Visual hash: perceptual hash (pHash) of cropped element snapshot
• Text features: exact text, normalized nearby visible text (siblings/parent)
• Layout context: bounding box, relative positions, z-index
• Stable metadata: page route, semantic roles, dataset-* attributes
• Historical stability score and previous locator attempts

Fallback strategies

• Deterministic: alternate selectors (by id → by attributes → by aria → by text)
• Structural: nearest stable ancestor + relative path
• Proximity heuristics: find element near stable label text or within same form
• Visual: match pHash within tolerance
• ML: embedding-based similarity across attribute/text/visual vectors

Heuristic vs ML

• Heuristics first: fast, explainable, low-risk; use rules with priority/weights
• ML next: models (Siamese / approximate nearest neighbor) for ambiguous cases using combined embeddings; good for dynamic UIs and A/B variants
• Always attach confidence score and explanation (which features matched)

Validation of auto-fixes

• Two-step: dry-run in staging/parallel run — attempt alternate locator and run targeted assertions (clickability, text, post-action state)
• Canary tests: apply to small percentage of CI runs + monitor error rates
• Shadow mode: record proposed fixes and outcomes without changing baseline

Metrics & decision rules

• Confidence thresholds: auto-apply if model confidence > 0.95 AND heuristic consensus; require human review if 0.7–0.95; block and alert if < 0.7
• Stability signals: element historical failure rate, frequency of selector churn, visual drift magnitude
• KPIs: reduction in test failures, time-to-repair, false-positive rate of auto-fixes

Risks & safeguards

• Risk: incorrect mapping causing wrong actions (e.g., clicking wrong button) → safeguard: verify unique side-effect free assertions and pre/post-state checks
• Risk: drifting model bias → periodic human audits, rollback capability, immutable change logs
• Security/privacy: redact sensitive screenshots; store hashes not raw images where necessary
• Governance: require approvals for auto-change thresholds, provide UI for reviewers, and integrate audit trail in PRs

I would implement this incrementally: start with extended fingerprints + deterministic fallbacks, add shadow ML scoring, then enable gradual auto-apply with monitoring and human-in-the-loop governance.

Clarify goals & constraints

• Fast triage of environment-related test failures (CI runs, staging)
• Low noise; cost-aware retention for long-term trend analysis
• Integrate with existing CI (Jenkins/GitHub Actions) and observability stack (Prometheus, Grafana, ELK/Vector, Jaeger/OTel)

What to collect — Infrastructure

• Metrics: host CPU, memory, disk I/O, network latency/packets, pod/container restarts, node pressure, kube-scheduler latency
• Logs: node kubelet, container runtime, cloud provider events, auto-scaler events
• Traces: control-plane traces for provisioning/scale events (sampling low)

What to collect — Application / Test harness

• Metrics: app request latency/histograms, error rates (4xx/5xx), DB connection pool usage, queue/backlog depth, feature-flag states
• Logs: structured application logs (JSON) including timestamps, log-level, component, stack traces
• Traces: full request traces with spans for DB, cache, external calls (instrumented via OpenTelemetry)

Correlation strategy

• Inject a unique correlation_id/test_run_id into each test execution and into CI environment variables; propagate via HTTP headers, message attributes, log MDC, and trace context (traceparent).
• When a test fails, query:
  • Traces by trace_id or test_run_id to see call path
  • Logs filtered by test_run_id and time window
  • Metrics during the test window (use Prometheus queries: increase(), rate() with time range)
• Provide a “Test Run” dashboard that aggregates logs, traces, and metrics for that test_run_id.

Retention & cost controls

• Keep high-resolution metrics and traces for 7–14 days (for fast triage); downsample metrics to 1m/5m and store aggregated for 90 days.
• Store full logs and traces for failed runs and selected smoke runs for 30–90 days; auto-delete verbose logs for successful runs after 7 days.
• Use sampling and dynamic sampling: full traces when test_run_id has failures or when error rate crosses threshold; otherwise 0.1–1% sampled.
• Archive critical long-term aggregates to cheap storage (S3 Glacier) and retain alerting/SLI data longer.

Operational practices

• Alert on environment signals during test windows: spike in pod restarts, node OOMs, increased provisioning times.
• Add runbook links in CI failure output with linked dashboard filtered by test_run_id.
• Regularly review retention policy vs cost and adjust sampling/aggregation thresholds.