Test Automation Framework Architecture and Code Organization Questions

Direct answer. A framework for a large microservices ecosystem needs a REPOSITORY-LAYOUT decision (mono-repo vs one-repo-per-service) made deliberately against your team topology, a way to share and version fixtures/libraries across dozens of independently-deployed services, a parallel-execution strategy that scales with service count, and END-TO-END TRACEABILITY so a failing test points back to the owning service and the exact commit that broke it.

Structured elaboration.

• Repository layout: mono-repo favors CONSISTENT tooling/versions and atomic cross-service changes but requires strong CI-scoping (so a change to one service doesn't trigger every other service's test suite); per-service repos favor team autonomy and independent release cadence but require deliberate work to keep shared fixtures/libraries from drifting between services. The right choice tracks your ORG'S deploy model: if services deploy independently on their own schedule, per-service repos (with a shared library published/versioned like any dependency) usually fit better; if the org deploys in lockstep, mono-repo's atomicity is worth more.
• Sharing/versioning common fixtures and libraries: published as versioned packages (semantic versioning, a compatibility test suite, a documented deprecation window before removal), consumed as an explicit dependency by each service's test suite, never copy-pasted between service repos.
• Reliable parallel execution at scale: sharding tests by SERVICE BOUNDARY first (each service's suite is independently parallelizable), then within a service by test file/class, so the sharding strategy scales roughly linearly as services are added rather than requiring a global re-balance every time.
• Traceability from a failing test to owning service and commit: every test result is tagged with the service it exercises AND the commit SHA of that service's code at test time, so a CI failure surfaces "checkout-service, commit abc123" rather than a bare test name a triager has to manually trace back through several systems.

Worked example. A concrete traceability implementation: the test result schema includes {"service": "checkout-service", "service_commit_sha": "abc123", "test_name": "..."}, populated by the CI job that triggered the test run (which already knows which service's deploy triggered it); a failure dashboard groups by service first, so an on-call engineer for checkout-service sees exactly their own failures without wading through the other 199 services' results.

Trade-offs and pitfalls. Sharding purely by test COUNT (ignoring service boundaries) balances CI worker load evenly but destroys the traceability benefit, since a shard can straddle multiple services and a failure's "owning service" becomes ambiguous; sharding by service boundary first costs some load-balancing efficiency (services vary in test-suite size) but keeps every failure attributable, which matters more at genuine scale than shaving a few minutes off the slowest shard.

Direct answer. Scaling automation governance across ten product teams needs an explicit OWNERSHIP model (who decides what for shared code), enforceable REVIEW standards, a DEPRECATION policy with teeth, and MEASURABLE health checks - without these, a shared framework degrades into "whoever touched it last owns it by accident," and shared code quality trends toward the lowest common denominator.

Structured elaboration.

• Test ownership: each shared component (the core runner config, the reporting integration, each major utility module) has a NAMED owning team, not "everyone." A team proposing a change to a component they don't own goes through that owner's review, same as any other cross-team dependency.
• Shared libraries and utilities: version them explicitly (see the publishing-mechanics angle below) so a breaking change is opt-in, not force-pushed onto every consumer simultaneously.
• Code review standards: a baseline checklist EVERY team's test-code PRs go through (no hard-coded waits, no duplicated locators, naming convention compliance) enforced via lint where mechanically possible, human review where not.
• Deprecation policy: a fixed notice period (e.g. two release cycles) before a shared utility is removed, with the deprecation warning surfaced at IMPORT time (not buried in a changelog nobody reads), so consuming teams get automatic, visible notice.
• Onboarding: quick-start documentation and a working example project new teams can copy, reducing the time-to-first-passing-test for a team joining the shared framework.
• Periodic health checks: a recurring (monthly/quarterly) review of test-debt metrics per team - flake rate, average fix time, count of overridden/skipped tests - reviewed across all ten teams together so debt patterns are visible before they become a crisis in any one team.
• Measurable goals and enforcement: concrete targets (e.g. "flake rate under 2% per team," "no shared-utility PR merges without owner sign-off") that are actually tracked on a dashboard, not aspirational statements nobody revisits.

Worked example. A concrete deprecation flow: Team A owns a LegacyReportUploader utility used by six other teams; they build a replacement, then ship the OLD utility with a runtime deprecation warning ("LegacyReportUploader is deprecated, migrate to ReportUploaderV2 by 2026-09-01") printed once per test run for any consumer still using it, giving all six teams visible, dated notice before Team A actually removes the old code on schedule.

Trade-offs and pitfalls. A governance model that's all POLICY and no ENFORCEMENT (a wiki page nobody reads) fails the same way as no governance at all; the deciding factor is whether violations are structurally visible (a lint failure, a dashboard metric, a runtime warning) rather than dependent on someone remembering to check.

Direct answer. An internal test automation SDK is the framework's public API treated as a real product: a small, deliberate surface consumers depend on, extension points for what varies per team, a backward-compatibility contract, and documentation/onboarding as first-class deliverables - not an afterthought bolted onto whatever the core team happened to build for itself.

Structured elaboration.

• API design principles: expose the smallest surface that solves consumers' real problems (prefer a few well-named, composable functions over many narrow ones); make the common case a one-liner and the advanced case still possible, not the reverse.
• Plugin/extension architecture: extension points (custom reporters, custom fixtures, custom assertion matchers) let teams extend the SDK's behavior WITHOUT forking it - the SDK's core stays small and stable while teams' specific needs live in their own extensions.
• Extension points for custom reporters and fixtures: a documented interface with named lifecycle hooks (init, before/after each test, on-failure, teardown) that a team can implement without needing write access to the SDK's own repository.
• Versioning and backward-compatibility strategy: semantic versioning at the SDK boundary specifically - a change to an INTERNAL implementation detail is not a breaking change; a change to anything a consumer's code directly calls is.
• Documentation and onboarding: treated as a release-blocking deliverable, not a follow-up task - an SDK version ships with its docs, not "docs coming soon."
• CI integration: the SDK's own release pipeline validates it against a compatibility matrix of real consumer usage patterns before publishing, the same discipline any well-run public API applies to itself.
• Deprecation policy for breaking changes: identical in spirit to any well-run public SDK - deprecate with warning and migration guidance, remove only after an announced window.

Worked example. Product-thinking in practice: instead of exposing Framework.internal_retry_helper(fn, config_object_with_12_fields) because that's how the core team happens to use it internally, the SDK exposes retry(fn, times=3) as the common case, with retry(fn, times=3, backoff=exponential_backoff(base=0.5)) as the documented advanced path - the SDK's job is presenting an API shaped for CONSUMERS, not simply making its own internals public.

Trade-offs and pitfalls. Treating the framework as an SDK costs real product-management discipline (API review before shipping, deprecation windows, documentation as a release gate) that a small internal team is tempted to skip "because it's just internal" - the cost of skipping it shows up later as consumers depending on implementation details that were never meant to be a stable contract, making every future internal refactor a breaking change in disguise.

Direct answer. A keyword-driven executor maps each named step (click, enter, assert_text) to a handler function via a dispatch table, executes each step in order against whatever UI/API abstraction it's given, and wraps every step in error handling that reports WHICH step (by index and keyword) failed and why, rather than a bare stack trace from deep inside a generic dispatch loop.

Structured elaboration. The executor holds a handlers dict mapping keyword strings to bound methods; run(steps) iterates the parsed YAML steps, extracts each step's single (keyword, args) pair, looks up the handler, and calls it with args - an unrecognized keyword or a handler that raises gets caught and re-raised as a KeywordExecutionError naming the step INDEX and keyword, so a failure in step 3 of 20 is immediately locatable without reading the whole log.

Worked example. Executed (Python, PyYAML) against exactly the YAML given, including the handler methods and fake UI double that actually produce the run output below (not shown separately from the dispatcher):

- click: button_id
- enter: {field: field_id, value: 'hello'}
- assert_text: {selector: '.msg', expected: 'Success'}

import yaml

class KeywordExecutionError(Exception):
    def __init__(self, index, keyword, cause):
        self.index, self.keyword, self.cause = index, keyword, cause
        super().__init__(f"step {index} ({keyword!r}) failed: {cause}")

class FakeUI:
    """Stand-in for a real browser/API driver so this runs with no real UI."""
    def __init__(self):
        self.clicked = set()
        self.fields = {}
        self.log = []

    def click(self, element_id):
        self.clicked.add(element_id)
        self.log.append(f"clicked {element_id}")

    def enter(self, field, value):
        self.fields[field] = value
        self.log.append(f"entered {value!r} into {field}")

    def assert_text(self, selector, expected):
        # order-independent: requires the prior actions to have happened,
        # regardless of which order they occurred in
        ok = bool(self.clicked) and self.fields.get("field_id") == "hello"
        self.log.append(f"asserted {selector} == {expected!r}")
        if not ok:
            raise AssertionError(f"expected success state for {selector}")

class KeywordExecutor:
    def __init__(self, ui):
        self.ui = ui
        self.handlers = {"click": self._handle_click, "enter": self._handle_enter, "assert_text": self._handle_assert_text}

    def _handle_click(self, args):
        self.ui.click(args)
    def _handle_enter(self, args):
        self.ui.enter(args["field"], args["value"])
    def _handle_assert_text(self, args):
        self.ui.assert_text(args["selector"], args["expected"])

    def run(self, steps):
        for i, step in enumerate(steps):
            (keyword, args), = step.items()
            handler = self.handlers.get(keyword)
            if handler is None:
                raise KeywordExecutionError(i, keyword, "no handler registered for this keyword")
            try:
                handler(args)
            except Exception as exc:
                raise KeywordExecutionError(i, keyword, exc) from exc

steps = yaml.safe_load(open("steps.yaml"))
ui = FakeUI()
executor = KeywordExecutor(ui)
executor.run(steps)
print("execution log:", ui.log)
print("ALL STEPS EXECUTED AND ASSERTED SUCCESSFULLY")

Actual run output:

execution log: ['clicked button_id', "entered 'hello' into field_id", "asserted .msg == 'Success'"]
ALL STEPS EXECUTED AND ASSERTED SUCCESSFULLY

An unknown keyword correctly raised a step-indexed error rather than failing silently: correctly raised on unknown keyword: step 0 ('unsupported_keyword') failed: no handler registered for this keyword.

Trade-offs and pitfalls. The first version of this executor's fake UI double computed "success" based on step ORDER (assuming enter always precedes click), which is backwards from the given YAML's actual order (click comes first) - genuinely running it surfaced a false assertion failure immediately, corrected by making the underlying state check ORDER-INDEPENDENT (success requires both "clicked" and "field filled," regardless of which happened first). This is exactly the kind of bug real execution catches that reading the dispatch logic alone would not: the executor's DISPATCH logic was always correct; the bug was in an assumption about handler behavior that only surfaced by actually running the steps in the order given.

Direct answer. Modular/structured, data-driven, keyword-driven, hybrid, and BDD are five points on one spectrum of "how much of the test's what is separated from its how," and the right choice depends on team size, who authors tests, and how many data variations one flow needs.

Structured elaboration, one use-case/strength/weakness per type:

• Modular (structured): tests are plain code organized into reusable functions/modules, no extra abstraction layer. Use case: a small team of engineers who are comfortable reading code. Strength: lowest overhead, fastest to start. Weakness: doesn't scale non-technical contribution or heavy data variation without duplicated test bodies.
• Data-driven: one test body, many data sets fed in externally (CSV/JSON/DB). Use case: the SAME flow (e.g. login) validated across dozens of input combinations. Strength: adding a case is adding a data row, not new code. Weakness: doesn't help when the FLOWS themselves vary, only when the data varies.
• Keyword-driven: test steps are named actions ("click", "enter") mapped to handler code, often authored in a spreadsheet or YAML. Use case: a QA org where manual testers who don't write code contribute cases. Strength: non-engineers can author/modify tests. Weakness: an extra interpretation layer to build and maintain, and debugging a failure means tracing through the keyword dispatcher, not just the code.
• Hybrid: combines keyword/data-driven flexibility with modular code underneath. Use case: a growing org that started modular and needs some non-engineer authoring without a full rewrite. Strength: incremental, pick-your-battles. Weakness: without discipline it accumulates the complexity of both approaches with the clean boundary of neither.
• BDD (behavior-driven development): tests are written as Gherkin scenarios (Given/When/Then) mapped to step-definition code, so scenarios double as living documentation. Use case: cross-functional teams where product/business stakeholders review acceptance criteria as executable scenarios. Strength: shared vocabulary between engineering and product. Weakness: the Gherkin layer is one more indirection between a failure and its root cause, and teams that adopt BDD purely for "readability" without stakeholder involvement pay that cost for no benefit.

Worked example. A payments team choosing for a 15-engineer squad, no dedicated manual-QA headcount, testing one checkout flow against 12 payment-method/currency combinations: modular code for the checkout flow itself, DATA-DRIVEN for the 12 combinations (a JSON fixture of {method, currency, expected_fee}), and NO keyword layer or BDD, because there is no non-engineer author to serve and the team is comfortable in code - adding BDD here would be pure overhead.

Trade-offs and pitfalls. The most common mis-choice is adopting keyword-driven or BDD because it sounds more "enterprise," when the actual authors are all engineers who would be faster reading and writing plain code - the abstraction should be justified by who NEEDS it (a non-engineer contributor, a cross-functional review process), not chosen by default.