Automation Frameworks and Tools Questions

Focuses on automation frameworks and tooling used for testing and experiment validation. Topics include browser automation frameworks, test organization patterns such as the page object model, handling synchronization and flaky tests, automated API testing and request validation, test data and fixtures, test runners and assertion libraries, continuous integration and continuous delivery integration for automated test execution and reporting, and strategies to keep automation maintainable and reliable. Candidates should be familiar with common tools and trade offs between end to end tests and faster unit or integration tests.

EasyTechnical

32 practiced

Write a simple automated API test in Python using requests and pytest. The test should: 1) POST to /api/login with JSON body {"username": "test", "password": "pass"}; 2) assert status code 200; 3) assert response JSON contains a non-empty "token" field. Show a small pytest fixture for base_url and the test that uses it.

Sample Answer

**Approach**- Use a pytest fixture to provide base_url.- Use requests to POST JSON, assert 200 and that response JSON contains a non-empty "token".- Keep test simple, readable, and reliable (timeout, explicit headers).

**Code**

python

import requests
import pytest

@pytest.fixture
def base_url():
    # Could be overridden by CLI/env in real suites
    return "http://localhost:5000"

def test_login_returns_token(base_url):
    url = f"{base_url}/api/login"
    payload = {"username": "test", "password": "pass"}
    # POST with small timeout to avoid hanging tests
    resp = requests.post(url, json=payload, headers={"Accept": "application/json"}, timeout=5)

    # Assert status code is 200
    assert resp.status_code == 200, f"Unexpected status: {resp.status_code}, body: {resp.text}"

    # Parse JSON and assert token exists and is non-empty
    data = resp.json()
    assert "token" in data, f"'token' missing in response: {data}"
    assert isinstance(data["token"], str) and data["token"].strip() != "", "Token is empty or not a string"

**Notes / Best practices**- Parameterize base_url for CI; add negative tests (invalid creds).- Consider retry/backoff or mocking for unit tests to avoid dependence on external service.

HardTechnical

30 practiced

Design how to integrate performance testing (k6 or JMeter) into CI/CD so that performance regressions are detected before release. Include: test scheduling (PR vs nightly vs pre-release), thresholds and gating policies, automated comparison vs baselines, canary releases with perf checks, scaling test load generators in CI, and automated rollback triggers when thresholds are exceeded in production or canaries.

Sample Answer

**Approach summary**I would implement a multi-gate performance pipeline that catches regressions early (PR), verifies stability daily, and validates pre-release and canary deployments with automated rollback triggers.

**Test scheduling**- PR: lightweight smoke k6 script (30s–2m) exercising critical paths to detect obvious regressions fast.- Nightly: fuller JMeter/k6 suites with representative traffic patterns and baseline comparison.- Pre-release: long run (hours) including endurance and spike tests against release candidate.- Canary: run real-traffic-shaped k6 scenarios against canary instances for 10–30% traffic window.

**Thresholds & gating**- Define SLOs (p95 latency, error rate, throughput, CPU/mem). Example gate: fail if p95 latency > baseline * 1.2 or error rate > 0.5%.- Enforce PR gates as warnings (block merge if repeated failures) and pre-release/nightly as hard gates (block deploy).

**Automated comparison vs baselines**- Store baseline metrics in time-series DB (Prometheus/Grafana or Influx). After each run compute delta and statistical significance (e.g., Mann–Whitney U for latencies).- Report: baseline, current, % change, confidence, and trend. Fail gate when breach and statistically significant.

**Canary releases with perf checks**- Deploy canary (10–20%): run short k6 tests plus monitor production metrics for N minutes. Automate decision: - Pass: promote - Fail: halt and rollback

**Scaling test load generators in CI**- Use containerized k6 cloud or distributed JMeter with Kubernetes Job that autos-scales load-generator pods. Use Kubernetes Horizontal Pod Autoscaler and artifactized test plans to spawn required VUs.

**Automated rollback triggers**- Integrate monitoring alerts (Prometheus Alertmanager) with CI/CD (ArgoCD/Spinnaker). If canary or production crosses thresholds, trigger automated rollback pipeline and open incident with logs, test artifacts, and links to replay tests.

**Example tools & flow**- k6 for CI-friendly scripts, JMeter for complex protocols, Prometheus+Grafana for metrics, Elasticsearch for logs, GitHub Actions or Jenkins to orchestrate, Kubernetes for scaling generators, Spinnaker/ArgoCD for canary and rollback.

I’d codify thresholds, baselines, and gating in pipeline templates, run frequent validation, and iterate thresholds to reduce noise while ensuring regressions block releases.

EasyTechnical

24 practiced

Compare Selenium WebDriver, Playwright, and Cypress as browser automation tools. For each, mention architecture (client-server vs in-process), cross-browser capability, headless support, network/request interception/mocking, parallelization, test authoring language(s), and typical use-cases where one is preferred over the others.

Sample Answer

**Overview (QA perspective)**

Below compares Selenium WebDriver, Playwright, and Cypress on key QA-relevant dimensions to help choose the right tool per testing need.

**Selenium WebDriver**- Architecture: Client-server (WebDriver protocol, remote drivers); language bindings communicate with browser drivers.- Cross-browser: Excellent — Chrome, Firefox, Edge, Safari, IE (via drivers).- Headless: Supported via browser-specific headless modes.- Network/request interception/mocking: Limited natively; usually requires proxy tools (BrowserMob, mitmproxy) or browser devtools integration.- Parallelization: Mature (Selenium Grid, Selenium Grid 4) and cloud providers.- Languages: Java, Python, C#, JavaScript, Ruby, etc.- Typical use-cases: Broad cross-browser regression, legacy browser support, enterprise environments, integration with CI and language-specific test stacks.

**Playwright**- Architecture: In-process-ish (driver + browser bundled; communicates over WebSocket) — designed for automation across Chromium, Firefox, WebKit.- Cross-browser: Strong (Chromium, Firefox, WebKit/Safari parity).- Headless: First-class support.- Network/request interception/mocking: Built-in, powerful request routing and response mocking.- Parallelization: Good — built-in test runner supports worker-level parallelism; can run browsers in parallel.- Languages: JavaScript/TypeScript, Python, Java, .NET.- Typical use-cases: Reliable cross-browser end-to-end tests with modern web apps, network mocking, flaky-test reduction, CI pipelines.

**Cypress**- Architecture: In-process (runs in the browser with a Node process controlling it); not WebDriver-based.- Cross-browser: Supports Chromium-based and Firefox; Safari support limited (WebKit experimental).- Headless: Supported (headless Chrome/Firefox).- Network/request interception/mocking: Built-in intercept/stub capabilities (good for stubbing XHR/fetch).- Parallelization: Supported via Cypress Dashboard or CI orchestration; less mature than Selenium Grid but effective.- Languages: JavaScript/TypeScript.- Typical use-cases: Fast, developer-friendly front-end E2E and component tests for modern SPAs where deep DOM access and time-travel debugging help; best when team uses JS stack and targets supported browsers.

Choosing: use Selenium for widest browser/legacy coverage; Playwright for modern multi-engine parity and network control; Cypress for fast, developer-friendly JS-centric E2E testing.

HardTechnical

28 practiced

Describe how to integrate automated security testing into a CI pipeline, covering static analysis (SAST), dependency scanning, and dynamic application security testing (DAST). For each stage explain tool options, false-positive handling, developer feedback loops, gating policy suggestions, and how to prioritize security findings in the same workflow as functional test failures.

MediumTechnical

24 practiced

Propose a test pyramid for a SaaS application and justify the percentages or counts of unit, integration, and end-to-end tests. Include target CI run-times for each layer on a PR, and tactics to enforce these targets without compromising coverage (examples: smoke suite, test selection, test-impact analysis).

Unlock Full Question Bank

Get access to hundreds of Automation Frameworks and Tools interview questions and detailed answers.

Join thousands of developers preparing for their dream job.