Legacy Modernization and Architecture Evolution Questions

Direct answer

Offline migration takes the system down, copies the data in one bulk pass, and brings the new system up, which is simple and gives you a clean consistency guarantee at the cost of real downtime. Online migration keeps the system live throughout, using change data capture, dual-write, or a parallel run to keep the target caught up with ongoing writes, which avoids downtime at the cost of real complexity and a period where you have to actively manage consistency between two live systems. For 3TB with continuous writes, the choice mostly comes down to how much downtime the business can actually tolerate, since 3TB is well within the range where either approach is technically feasible.

Structured elaboration

Offline (bulk cutover): take the system offline (or read-only), copy all the data in one pass, validate it, then bring the new system up and resume writes there. The consistency story is simple: there's a single, unambiguous point in time where you switched, and no risk of the two systems drifting because only one is ever accepting writes. The cost is straightforward too: however long the copy and validation takes is downtime (or a read-only window) the business has to absorb.

Online (CDC, dual-write, or parallel-run): the legacy system keeps taking writes throughout, and one of two different mechanisms keeps the new system caught up. CDC reads the source database's own change log (binlog, WAL, or equivalent) and replays those changes into the new store; it is the more reliable option because the log is a durable, ordered record of every write, so a lagging or restarted CDC process can always resume from where it left off without losing anything. Dual-write instead has the application write to both the legacy and the new store synchronously, in the same request path; it avoids needing change-log access at all, but it is the riskier mechanism, because the two writes are not atomic. If the second write fails after the first succeeds, the two stores silently diverge with no durable record of what should have happened, and the application code has to carry compensating logic (retries, a reconciliation queue, or an outbox pattern, where the pending second write is saved as a row in the same local transaction and a background process reliably delivers it afterward) to catch that case. In practice, teams default to CDC whenever the legacy system exposes a usable change log, and reach for dual-write mainly when it doesn't, accepting the added application-level complexity and atomicity risk as the price of not needing log access. Either way, the migration is managing an eventually-consistent system for its duration: there's always some lag between the two sides, and you need reconciliation to catch drift, plus a real cutover moment where you decide the new system has caught up enough to trust with live writes.

The decision factors:

• Downtime tolerance. If the business can accept a maintenance window (a batch system, an internal tool, off-peak hours for a consumer product), offline is simpler and safer precisely because it avoids the complexity online migration takes on. If downtime is unacceptable (a customer-facing transactional system with global users in different timezones, so there's no universally quiet window), online is the only real option.
• Data drift risk. Offline has none, by construction. Online has some, proportional to CDC lag and the duration of the migration window, which is why online migrations need active reconciliation, not just a one-time comparison.
• Complexity budget. Online migration is meaningfully more work to build and operate correctly; if the team doesn't have the capacity or experience to run CDC and reconciliation reliably, an offline approach executed well can beat an online approach executed poorly.

Worked example

For a system with 3TB of transactional data and continuous writes: 3TB can realistically be copied in a matter of hours with modern bulk-transfer tooling, which puts offline migration's downtime cost in the "a maintenance window, not a multi-day outage" range. If this is an internal reporting system with a well-understood low-traffic window (say, a Sunday at 3am), offline is the simpler, lower-risk choice: schedule the window, copy the data, validate with a full checksum comparison (feasible at this size), cut over.

If instead this is a customer-facing order system serving users across time zones with no acceptable downtime window, the same 3TB data volume now calls for online migration: CDC captures ongoing writes during a bulk load, reconciliation runs continuously to catch drift, and the cutover happens once reconciliation has been clean for a defined period, accepting the added operational complexity because the downtime constraint leaves no other option.

Trade-offs and pitfalls

The mistake to avoid is choosing online migration by default because it sounds more sophisticated, when the actual downtime constraint doesn't require it; the added complexity of CDC, drift monitoring, and a more elaborate cutover procedure is a real cost that should only be paid when the downtime tolerance genuinely demands it. The opposite mistake, choosing offline for a system that actually cannot tolerate the downtime because the team underestimated how long the copy and validation would really take, is just as damaging and harder to walk back once the maintenance window has started.

Direct answer

A full rewrite wins when the legacy system's structure fights every incremental change so hard that the accumulated cost of working around it, in time, in bugs, in the parts of the team's attention it consumes, exceeds a realistic (padded) estimate for building it again from what you now know. What has to go right: the team genuinely understands the system's current behavior well enough to not silently drop requirements, the business can tolerate a real delivery gap, and there's a credible plan for the parts of the estimate that are hardest to predict, data migration and the long tail of edge cases nobody remembers exist until they break in production.

Structured elaboration

The risks a rewrite introduces that incremental work avoids:

• The estimate is a guess dressed as a plan. You cannot fully know what a legacy system does until you've read every path through it, and if you could do that cheaply, you probably wouldn't need a rewrite. Rewrites reliably underestimate the "long tail," the 20% of behavior that's undocumented, weird, and only matters for edge cases, because that's exactly the part that's hardest to discover in advance.
• All-or-nothing delivery. An incremental effort can stop, ship partial value, and reassess. A rewrite typically can't ship real value until most of it is done, which means the business is exposed to the full cost of the effort before seeing any of the benefit, and a rewrite that stalls at 80% delivers zero value for the investment made.
• Data migration risk concentrates at the end. Incremental approaches migrate data piece by piece, catching problems early on a small blast radius. A rewrite often defers data migration to a single cutover at the very end, which is exactly when the team has the least remaining slack to absorb a surprise.

The risks incremental work introduces that a rewrite avoids:

• Running two systems for longer than planned, with the ongoing cost and the risk of the migration simply never finishing.
• The seam itself becoming a source of bugs, translation errors at the boundary between old and new that a from-scratch rewrite wouldn't have to deal with.
• Slower overall delivery of the target end state, since incremental work is deliberately paced to be safe rather than fast.

Concrete mitigations for the rewrite risks: build characterization tests against the legacy system's actual behavior before writing a line of the replacement, so the estimate is grounded in observed behavior rather than assumed behavior; plan the data migration and cutover as a first-class, staged piece of the project rather than a final step; and set an internal checkpoint (not a public commitment) partway through where the team honestly reassesses whether the estimate is holding, with permission to fall back to a hybrid approach if it isn't.

Worked example

A retrospective example: a team decided a legacy inventory system's coupling to a proprietary rules engine made incremental extraction impractical, so they chose a rewrite. What made it work: they spent the first month purely on characterization testing against the legacy system, capturing behavior for every product category and edge case they could enumerate, before writing any new code. That testing surfaced a rounding rule for one product category that looked like a bug but turned out to be a deliberate accommodation for a regulatory requirement in one region, exactly the kind of hidden logic a rewrite risks silently dropping. Because they'd captured it in a test before starting, the new system preserved it correctly, and the team could point to a passing test suite, not just confidence, when they cut over.

An architectural decision that limited scale, brought up in the same retrospective: the original system had used a single shared table for all regions' inventory data, which made the regulatory rounding rule and a dozen similar region-specific rules invisible in the schema and discoverable only by reading application code path by path. The rewrite's replacement schema made region-specific rules an explicit, first-class concept, directly because the team had been burned by how hard the old shape made those rules to find.

Trade-offs and pitfalls

The single biggest risk-reducer for a rewrite is treating "we understand the current behavior" as a deliverable to prove, via characterization tests against real behavior, rather than an assumption to proceed on. Teams that skip this step and start writing new code based on what they believe the system does, rather than what it's actually observed to do, are the ones whose rewrites silently change behavior and discover it in production months later.

Direct answer

An anti-corruption layer (ACL) is a translation boundary you deliberately put between a legacy system and a new one so that the new system's domain model never has to bend to accommodate the legacy system's quirks. It is not just an adapter that converts data formats; its job is to protect the new model's integrity by absorbing all the legacy system's inconsistencies, missing fields, and outdated assumptions on the legacy side of the boundary, so nothing about the old system's design leaks into the new one.

Structured elaboration

Concretely, an ACL is responsible for:

• Translation: converting the legacy system's data shapes, field names, and units into the new system's domain model, not the other way around.
• Mapping semantic gaps: the legacy system might represent a concept the new system does not have an exact equivalent for (a status enum with legacy-only values, a field that means two different things depending on another field). The ACL is where you decide how those map, once, in one place, instead of every consumer inventing its own interpretation.
• Isolation: consumers on the new side never call the legacy system directly or see its raw shapes. If the legacy system changes (or if you eventually replace it), only the ACL has to change.

The benefit is that your new services get to have a clean domain model that reflects how the business actually works today, not how a fifteen-year-old system happened to represent it. The cost is that the ACL itself becomes a piece of infrastructure someone has to own, test, and keep in sync as both sides evolve, and a badly maintained ACL can become exactly the kind of tangled legacy code it was meant to prevent.

Worked example

Say a legacy order system represents order status as an integer code (0, 1, 2, 9) where 9 means "cancelled" but also gets reused for "refunded" depending on an unrelated flag elsewhere in the record, a real and common kind of legacy inconsistency. A new order service wants a clean OrderStatus enum: PENDING, CONFIRMED, SHIPPED, CANCELLED, REFUNDED.

The ACL sits at the boundary and does the translation:

def translate_legacy_status(legacy_code: int, refund_flag: bool) -> str:
    if legacy_code == 9 and refund_flag:
        return "REFUNDED"
    mapping = {0: "PENDING", 1: "CONFIRMED", 2: "SHIPPED", 9: "CANCELLED"}
    return mapping[legacy_code]

Every consumer on the new side calls the ACL and gets back a clean OrderStatus, never the raw integer code or the refund flag. If the legacy system later adds a sixth status code, only this one function needs to change.

To validate this kind of adapter, the concrete test strategy is a contract test against a fixture of known legacy inputs paired with their expected new-model outputs, covering every legacy value (including the ambiguous ones like the reused code 9) and every combination the ACL has to disambiguate, not just the happy path. That test suite is what tells you the ACL is behaving correctly before any real traffic depends on it, and it is what catches the translation silently breaking if someone touches the mapping later.

Trade-offs and pitfalls

The main pitfall is letting the ACL grow into a second copy of legacy logic instead of a thin translation boundary: if it starts encoding business rules of its own rather than just mapping shapes, you have created a new piece of legacy code, not protected against the old one. The other common mistake is skipping the ACL for "just this one caller" because it seems faster, which reliably ends with the legacy system's quirks leaking into the new domain model through that one exception, and everyone downstream having to account for it.

Direct answer

These are four different tools for four different situations, and interviewers who use them interchangeably are usually testing whether you actually know the difference. A strangler fig is for gradually replacing a legacy system's functionality behind a routing seam while it stays live. An anti-corruption layer is for protecting a new system's domain model when it has to talk to a legacy system it is not replacing (or not replacing yet). A facade is for simplifying and unifying how callers interact with a messy legacy system, without migrating anything or translating between two different domain models at all. A full rewrite is for when the legacy system cannot be safely peeled apart at all. The decision comes down to whether the system can be decomposed into independently extractable pieces, and whether it needs to keep running the whole time. A fourth axis matters just as much: whether you are actually trying to replace anything at all, or you just want a safer, simpler interface onto a legacy system you have no plan to migrate away from, which is what a facade alone is for.

Structured elaboration

A useful way to separate them:

• Strangler fig answers "how do I replace this system's functionality over time without a big cutover." It assumes the system can be broken into pieces that can move independently, and it is a migration strategy, not a permanent architecture. Use it when the legacy system is large but decomposable, and downtime is not acceptable.
• Anti-corruption layer answers "how do I integrate with this system without its bad decisions becoming my bad decisions." It does not assume you are replacing anything; you might be integrating with a legacy system permanently (a partner's system you do not control) or temporarily (as one piece of a larger strangler effort, where the ACL sits between the parts still on legacy and the parts already moved). Use it any time a system you do not fully trust the shape of has to feed a system whose domain model you want to keep clean.
• Facade answers "how do I make a messy legacy system safer and simpler to call, without replacing or translating anything." Unlike an anti-corruption layer, it does not have to reconcile two different domain models, it is a single unified interface placed in front of a system you are not migrating away from, at least not yet, so callers stop depending directly on the legacy system's tangled internals. Use it when the goal is purely to make what already exists safer to call, or as the seam a later strangler-fig effort will route traffic through once you do decide to replace what is behind it.
• Full rewrite answers "the incremental approach is not viable here." Use it when the legacy system's capabilities are so tightly coupled that there is no seam to strangle along, when the code is small enough that a rewrite is genuinely cheaper than untangling it, or when the business can tolerate a real code freeze while the rewrite happens. It is also sometimes the right call for build-versus-buy reasons that have nothing to do with technical coupling: if a vendor product now does what the core subsystem does, replacing rather than incrementally modernizing can be the faster and cheaper path, provided the migration and switching costs are honestly priced in.

In practice they combine rather than compete: a strangler-fig migration typically starts by placing a facade in front of the legacy system to create a single seam, then uses an anti-corruption layer at that seam to protect the already-migrated parts from the still-legacy parts as pieces move across it.

Worked example

A team replacing a core subsystem (say, pricing) weighs the options (a facade alone is ruled out early, since the goal is to actually move pricing off the legacy system, not just make it safer to call):

• Strangler: pricing logic touches a dozen call sites across the codebase, but each call site is independently identifiable, so they can move pricing behind a facade and migrate call sites one at a time. This is the default choice given decomposability.
• ACL alone (no strangler): they decide pricing itself will stay on the legacy system for now, but a new checkout service needs pricing data. Rather than have checkout speak the legacy pricing format, they add an ACL so checkout's domain model stays clean, with no plan yet to replace pricing itself.
• Full rewrite / buy: they discover pricing logic is deeply entangled with tax and discount logic in ways that resist any clean extraction, and a commercial pricing engine now covers the requirements. They rewrite (replace) rather than strangle, accepting a scoped migration project with a defined cutover instead of an open-ended incremental one.

What would make the team abort a strangler approach midway and fall back to one of the other two: discovering that the "independent" call sites actually share hidden mutable state that makes partial migration unsafe, or finding the timeline slipping so far that the cost of running two systems is exceeding the cost a rewrite would have been from the start.

Trade-offs and pitfalls

The trap is picking strangler fig by default because it feels lower-risk, without checking that the system is actually decomposable; forcing a strangler approach onto tightly coupled logic produces years of a half-migrated system with all the maintenance cost of two systems and none of the safety benefit, because the seam itself becomes unreliable. The opposite trap is reaching for a full rewrite out of frustration with legacy code, when a narrower ACL would have solved the actual integration problem at a fraction of the cost and risk.

Direct answer

Duplicated and out-of-order outputs right after a scheduler migration is a classic sign that the new orchestrator's delivery guarantees or retry behavior differ from the legacy cron system's, even though nothing about the downstream pipelines changed. The triage priority is stop the bleeding first (pause or gate the affected pipelines so bad data doesn't propagate further), gather evidence while the problem is still fresh and reproducible, then find the actual root cause before declaring it fixed.

Structured elaboration

1. Stop further damage first. Before investigating root cause, decide whether to pause the affected pipelines or gate their downstream consumers, since every additional run while the bug is active compounds the cleanup required later. This is a judgment call between availability and correctness, but for a modernization release specifically, the safer default is usually to pause, given that the whole point of the migration was not to introduce new correctness problems.
2. Gather evidence while it's fresh. Capture the exact job run logs, timestamps, and any retry or scheduling metadata from the new orchestrator for the affected runs, since orchestrator-level state (queued, retried, duplicate-triggered) is often the first thing to get cleaned up or rotated out of logs, and it's the evidence most likely to actually explain what happened.
3. Identify the root cause, focusing on what changed. Common causes of this exact symptom after a scheduler migration: the new orchestrator's retry policy differs from the legacy cron's (retrying a job that actually succeeded but whose success signal was slow, causing a duplicate run), a change in how job triggers handle overlapping runs (the legacy cron might have simply skipped an overlapping run, while the new orchestrator queues or parallelizes it), or a timezone or scheduling-window difference causing jobs to fire in an unexpected order relative to their dependencies.
4. Remediate, which for duplicated output usually means both fixing the trigger behavior going forward and cleaning up the duplicated data already produced, a data-repair task distinct from the pipeline-configuration fix itself.
5. Prevent recurrence by adding a check (an idempotency guarantee at the pipeline level, or a monitoring alert on unexpected duplicate run IDs) so this specific failure mode is caught immediately if it happens again, rather than relying on downstream analytics to notice something looks wrong.
6. Communicate throughout, not just after resolution: stakeholders consuming the affected analytics need to know their data may be unreliable for the affected window as soon as that's known, so they don't make decisions based on corrupted numbers while the investigation is still underway.

Worked example

Immediately after migrating a scheduler from legacy cron to a new orchestrator, downstream pipelines show duplicated and out-of-order outputs:

• Stop the bleeding: the team immediately pauses the three downstream pipelines showing anomalies, accepting a delay in fresh data rather than risking further corrupted output while the cause is unknown.
• Evidence gathering: pulling the new orchestrator's run logs for the affected jobs reveals two of them were triggered twice within a short window, and the orchestrator's own metadata shows both triggers came from the same scheduled entry, pointing at a configuration issue rather than a pipeline-code bug.
• Root cause: investigation finds the new orchestrator's default retry-on-timeout behavior triggered a second run because a job's completion signal was slower than the orchestrator's timeout threshold, even though the first run had actually completed successfully; the legacy cron system had no such retry behavior, so this exact failure mode never existed before the migration.
• Remediation: the orchestrator's timeout threshold is increased to comfortably exceed the job's normal completion time, and the duplicated output is identified and removed from the affected downstream tables via a targeted cleanup script, run only against the specific affected time window identified from the orchestrator logs.
• Prevention: an idempotency key is added to the pipeline's output-write step, so even if a duplicate trigger happens again for any reason, the second run is a safe no-op rather than a second write, and a monitoring alert is added specifically watching for duplicate run IDs from the orchestrator.
• Communication: downstream teams are notified within the first hour that the affected time window's data may be unreliable, well before the root cause investigation concludes, so they don't act on it in the meantime.

Trade-offs and pitfalls

The trade-off in the "stop the bleeding" decision is availability against correctness: pausing pipelines means stale data for consumers in the short term, but continuing to run a pipeline with an unknown active bug risks compounding the cleanup, and for a modernization release specifically, the more conservative choice is usually right. The pitfall to avoid is fixing the immediate symptom (removing the duplicated rows) without finding and fixing the actual trigger-behavior difference that caused it, which just means the same bug recurs on the next scheduling edge case that happens to hit the new orchestrator's different retry or overlap-handling behavior.