Mobile Application Testing Questions

Clarify goals & constraints

• Target: high-risk user segments, release cadence (e.g., daily smoke, nightly full), budget cap (dollars/hour).
• Success metric: % of users covered vs cost per run.

Device selection (coverage strategy)

• Use telemetry + analytics to pick top N devices by active users, crash rate, OS split. Prioritize:
  • Top 80% of installs (by device model + OS patch)
  • Plus 2–3 representative low-market-share devices (fragmentation and edge cases)
• Example: Android: Pixel 6 (12), Samsung A13 (13 & 12) ; iOS: iPhone 13 (15) + iPhone 8 (14)
• Limit OS versions to: current major, previous major, and the minimum supported version.

Concurrency & scheduling

• Set pool size based on budget: max_parallel = floor(budget / hourly_device_cost)
• Tiered schedules:
  • Quick smoke (high concurrency, short): CI-triggered Appium suite on top 3 devices
  • Full regression (lower concurrency, night): broader matrix
• Use queuing with priority (smoke > regression > exploratory).

Session reuse

• Reuse device sessions for sequential tests to avoid boot time:
  • Group tests per app-install lifecycle; keep app installed between related tests
  • Use Appium capabilities: set noReset=true for grouped suites, then full reset for integrity checks.

Test sharding & parallelization

• Shard by test file/class (not single test) to reduce flakiness and setup cost.
• Use deterministic hashing of test IDs -> assign to worker slots to balance duration.
• Instrument test durations historically to create balanced shards.

Failover & flaky devices

• Maintain device health metrics; mark device unhealthy after X failures.
• Automatic retry policy:
  • First retry on same device (quick transient), second retry on alternate device with same OS.
  • If flaky > threshold, quarantine device and notify ops.
• Collect logs, screenshots, and Appium server traces for root-cause.

Cost vs coverage trade-offs

• Use risk-based testing: run full matrix only for release-candidates; run reduced matrix for feature branches.
• Employ simulated devices/emulators for lower-cost smoke tests; reserve real devices for hardware-specific and compatibility tests.
• Use sampling: run heavy tests on subset, expand on failure or scheduled full runs.
• Track cost-per-bug and adjust matrix: drop low-yield devices.

Implementation notes

• Orchestrator: CI pipeline (GitHub Actions/Jenkins) + custom scheduler that queries device-farm API (e.g., AWS Device Farm/Sauce Labs) to allocate slots, pass Appium capabilities, collect results.
• Telemetry feedback loop to continuously refine device list and shard weights.

This strategy balances risk, cost, and speed while keeping tests reliable and actionable.

Recommendation (summary)
I’d use a hybrid: ~60% emulators/simulators, 25% cloud device farm, 15% small on‑prem real‑device lab. This balances cost, speed, coverage, and fidelity for 30 configurations on a limited budget.

Test coverage plan

• Emulators (60%): run unit, fast UI smoke, and most automated regression suites across all 30 configs nightly. Use CI-parallelization for OS versions, screen sizes, CPU classes.
• Cloud device farm (25%): run pre-release compatibility matrix tests and flaky/intermittent automation on 8–10 high-priority real models (popular OS versions, different OEM skins, varied network profiles).
• Real-device lab (15%): keep 4–5 devices representing top market share, low-end hardware, and critical OEMs for manual exploratory, performance, battery, sensor, and camera tests; run final regression before release.

Trade-offs

• Cost: emulators cheapest; cloud incurs per-minute/device fees; real lab has upfront device + maintenance.
• Coverage: emulators give breadth; cloud adds breadth + better fidelity; real lab gives highest fidelity for edge issues.
• Fidelity: sensors, BLE, camera, thermal and manufacturer bugs require real devices.

Rationale & metrics
Prioritize devices by telemetry (install base, crash/ANR rates). Measure success by defect escape rate, test flakiness, and release cycle time; adjust mix quarterly.

Brief approach
I’d design Appium automation to be reliable, fast, and CI-friendly: pick devices, set deterministic capabilities, enable parallel runs, prefer device-clouds for scale, and integrate tests into CI with reporting and artifact capture.

Real devices vs emulators

• Real devices: better for gestures, camera, network variability, flaky-device bugs (use for release/regression).
• Emulators/simulators: faster, cheaper, good for early dev and unit-level UI tests.

Desired capabilities (example, Java)

DesiredCapabilities caps = new DesiredCapabilities();
caps.setCapability("platformName", "Android");
caps.setCapability("platformVersion", "12.0");
caps.setCapability("deviceName", "Pixel_5_API_31");
caps.setCapability("automationName", "UiAutomator2");
caps.setCapability("app", "/path/to/app.apk");
caps.setCapability("noReset", true);

Parallel execution strategies

• Run multiple Appium server instances bound to different ports and devices.
• Use Selenium Grid or Appium Grid (Appium 2 + drivers) to route sessions.
• In test framework (TestNG/JUnit) parameterize device configs and use containerized workers.

Device clouds

• BrowserStack / SauceLabs simplify device matrix, provide stable VMs, network sim, and logs.
• Use cloud-specific capabilities (e.g., "browserstack.user") and rely on their parallelism and reporting.

CI integration

• Package tests in build pipeline (Jenkins/GitHub Actions/GitLab). Steps: start emulator or connect cloud credentials → run tests → collect logs/screenshots/video → fail build on regressions → publish reports (Allure/JUnit) and artifacts.
• Use environment variables for secrets and conditional stages to run smoke vs full matrix.

Best practices

• Keep tests deterministic, use retries and implicit waits sparingly, use accessibility IDs, shard tests by feature for parallelism, and run device-cloud nightly full-matrix.