Legacy Modernization and Architecture Evolution Questions
Evolving an existing system rather than designing greenfield. Covers the modernization patterns (strangler fig, anti-corruption layers, facades and protocol adapters), choosing between rehosting, replatforming, incremental refactoring and a full rewrite, data migration and coexistence (dual-running, change-data-capture versus bulk cutover, reconciliation and drift), cutover readiness and decommissioning, recovering undocumented behavior from legacy code and stored procedures, instrumenting a migration so you can tell in real time whether it is working, and the organizational and risk management of long migrations across many teams. The scope is the migration itself: not quantifying or prioritizing technical debt, not code-level refactoring craft, not how to decompose a system into microservices, and not cloud migration or deployment and rollback mechanics as topics in their own right.
You're extracting model-serving and feature computation out of a monolith that currently handles both batch scoring and real-time predictions. How do you decide what to pull out first, and how do you know the extraction didn't change any outputs?
Sample Answer
Direct answer
Extracting model serving and feature pipelines from a tightly coupled monolith means deciding what to pull out first based on what's genuinely separable with the least shared state, then proving at each step that the extraction preserved correctness before moving to the next piece, the same incremental discipline as any strangler-fig extraction (pulling one piece out at a time behind a stable interface, verifying it matches the old behavior, then moving to the next), applied specifically to the coupling patterns ML systems tend to have (shared feature computation, tightly coupled batch-and-real-time code paths).
Structured elaboration
- Sequence by coupling, not by size. The component with the fewest hidden dependencies on the rest of the monolith, not necessarily the biggest or most important one, should usually go first, since it's the cheapest way to validate the extraction process itself before applying it to something higher-stakes. Often this means starting with a specific model's serving path that has a well-defined, narrow feature dependency, rather than the shared feature-computation layer that everything else depends on. This same logic decides the batch-versus-real-time question specifically: real-time serving is usually the better first target, not because it's technically simpler, but because a bug in an extracted real-time path is visible within a single request, while a bug in extracted batch scoring can silently run for hours and corrupt an entire scoring run's worth of output before anyone notices; batch jobs also tend to share more surface area with each other and with the feature-computation layer, making a clean extraction boundary harder to draw on the first attempt.
- API and contract design before extraction, not after. Define the interface the extracted service will expose (what features it needs as input, what prediction format it returns) explicitly, so the extraction has a clear target rather than organically figuring out the boundary as you go, which tends to produce a messier interface.
- Testing and verification strategy specific to ML. Beyond standard integration testing, this needs prediction-level comparison: does the extracted service, given the same input, produce the same prediction as the monolith did for that input, within whatever numerical tolerance is appropriate. This is the ML-specific equivalent of a golden-dataset comparison used in any legacy migration.
- Data migration steps for whatever state the extraction needs to bring with it: model artifacts, feature-store data if features are moving too, and any historical data needed for the extracted service's own operational needs (monitoring baselines, for instance).
- Measure correctness throughout the migration, not just at the end. Continuous comparison between the still-monolithic and newly-extracted paths, for as long as both exist during the transition, catches a regression introduced partway through rather than only at a single final validation gate.
Worked example
A legacy monolith handles both batch scoring and real-time predictions with tightly coupled feature-sharing logic between the two:
- Sequencing decision: rather than extracting the shared feature-computation layer first (which everything depends on, making it the highest-risk piece to get wrong), the team extracts the real-time prediction serving path for one specific, lower-traffic model first, since it has the narrowest feature dependency and the smallest blast radius if something goes wrong.
- Boundary and contract design: they explicitly define the extracted service's input contract (which specific features it needs, in what format) before writing the extraction code, discovering during this design step that two of the features it currently reads directly from shared in-process memory would need a real interface (an API call, or a shared feature store) once extracted, a coupling that wasn't obvious until they tried to draw the boundary explicitly.
- Verification: the extracted service runs in shadow mode, receiving the same inputs as the monolith's real-time path and comparing its predictions against the monolith's, for two weeks, with any divergence investigated before the extracted service takes any real traffic.
- Sequencing continues: having proven the extraction pattern and tooling on the lowest-risk model, they apply the same process to the next model, then eventually the shared feature-computation layer itself, using what was learned (including the shared in-process memory issue) to design that extraction more carefully from the start. Batch scoring is deliberately sequenced after the real-time extractions are proven and after the shared feature-computation layer has been pulled out cleanly, specifically because a batch-scoring bug can silently corrupt a full day's output before anyone notices it, versus a real-time bug that's visible within a single request; when it is extracted, the same shadow-mode comparison runs against a full historical batch window rather than a live request stream, comparing the extracted batch job's output file against the monolith's for the same input data before either replaces the other.
- Correctness measured throughout: at each extraction step, prediction-comparison monitoring runs continuously during the shadow and gradual-cutover period, not just as a one-time check, catching a subtle feature-computation timing difference in the third extraction that hadn't shown up in the first two, because it only manifested under a specific data-freshness condition the earlier extractions' feature sets didn't happen to depend on.
Trade-offs and pitfalls
The trade-off is a slower overall migration (many small, verified extractions rather than one larger effort) against the much lower risk of each individual step, which is worth it specifically because the pattern discovered in the worked example, a coupling issue only surfacing once you actually try to draw the extraction boundary, is common enough that starting with the lowest-risk piece consistently pays off in lessons that make later, higher-stakes extractions safer. The pitfall is treating "the code compiles and runs" as sufficient verification for an extracted ML component, when only prediction-level comparison against the original actually confirms the extraction preserved correctness, not just that it produces some plausible-looking output.
A legacy system has real business logic buried inside feature-preprocessing and model code, undocumented and hard to separate from the plumbing around it. How would you pull it apart into something you can actually maintain, without silently changing what it does?
Sample Answer
Direct answer
Pulling real business logic out of feature-preprocessing and model code means separating three genuinely different concerns, data transformation, model inference, and business rules, that a legacy system has tangled together, and doing it in a way that preserves the exact reference outputs the compliance and audit function depends on, so the separation itself never silently changes what the system actually decides.
Structured elaboration
- Discover the entanglement first. Read through the actual code (not documentation, which is often stale) to find where business logic is hiding inside what looks like feature engineering, an if-statement inside a feature transform that's really encoding a business rule, a hardcoded threshold in the model-scoring code that represents a policy decision, not a modeling one.
- Define the target separation. Feature transforms should produce inputs to the model and nothing else; model inference should produce a raw model output and nothing else; business rules should take the model output and apply whatever policy logic (thresholds, overrides, regulatory constraints) turns it into a final decision. This separation makes each piece independently testable, auditable, and changeable, which is exactly what a legacy system that tangled them together made difficult.
- Preserve reference outputs through characterization testing. Before refactoring, capture the system's exact current output for a comprehensive set of inputs, including edge cases the compliance function specifically cares about, and treat matching those outputs as the bar for the refactored version, not "looks approximately right." This is the same discipline as migrating any legacy business logic: the system's current behavior, correct or not, is the thing you're preserving unless a deliberate, separately-approved decision changes it.
- Auditability as a first-class requirement of the new structure. Once business rules are pulled into their own explicit component, they should be independently logged and traceable, so an auditor can see exactly which rule fired and why for a given decision, something that's often impossible to reconstruct when the logic is buried inside a feature-preprocessing function.
- Migrate incrementally, one entangled piece at a time, verifying reference-output parity after each extraction rather than attempting the full separation in one large, hard-to-verify change.
Worked example
A legacy credit-scoring system has business logic embedded directly inside its feature-preprocessing code: a feature-transform function silently caps a specific input value at a regulatory-mandated maximum before feeding it to the model, a piece of compliance logic that's invisible unless someone reads that specific function closely.
- Discovery: a full code review of the preprocessing pipeline surfaces four instances of business logic embedded this way, including the capping example, none of them documented as compliance requirements anywhere outside the code itself.
- Characterization tests are written capturing the system's exact output (including the effects of all four embedded rules) across a representative and edge-case-heavy set of inputs, verified against the compliance team's own records of what the rules are actually supposed to do, which in one case reveals the code's behavior had drifted from the documented policy years ago, a discrepancy the compliance team explicitly decides to preserve (matching current production behavior) rather than silently correct during this refactor, treating any policy fix as a separate, deliberate decision.
- Refactor: feature transforms are rewritten to produce only model inputs, the capping rule and the other three business rules are extracted into an explicit, separately testable
business_rulesmodule that runs after model inference, and each rule logs its own decision trace. - Validation: the refactored system's output matches the characterization tests exactly across the full test set, including the intentionally-preserved discrepancy, before it replaces the legacy code path, and the compliance team signs off specifically on the new auditability (being able to see which rule fired for a given decision) as a genuine improvement over the legacy system's opacity.
Trade-offs and pitfalls
The pitfall this whole approach is built to avoid is treating this as an opportunity to "clean up" logic that looks wrong while you're in there; separating structure and correcting behavior are two different decisions, and conflating them during a refactor is exactly how a well-intentioned modernization effort silently changes a regulated system's behavior without anyone explicitly approving that change. The trade-off is the extra rigor of characterization testing against a system that may have real historical drift from its own documented rules, which takes real time to untangle and get sign-off on, but skipping it trades a small time savings now for a much larger compliance risk later.
You need to move an existing inference or model-serving system onto a new platform without a service interruption users would notice. How would you sequence that cutover and prove the new platform is behaving equivalently before you commit to it?
Sample Answer
Direct answer
Moving an inference platform to a new serving product without a noticeable interruption means treating it like any other high-stakes cutover: validate compatibility and compliance up front, shift traffic gradually while comparing outputs between old and new, and keep an instant rollback available until you've proven parity across enough real traffic to trust the new platform fully.
Structured elaboration
- Discovery and compatibility testing first. Before moving any real traffic, confirm the new platform can actually serve the existing model artifacts (format compatibility, supported runtime versions) and reproduces the same inference behavior on a representative set of inputs. A managed serving product's runtime is rarely bit-for-bit identical to a custom on-prem setup, so this step exists specifically to catch numerical or behavioral drift before it's customer-facing.
- Data residency and compliance checks. Moving to a managed product often means data crosses infrastructure boundaries it didn't before; confirm the new platform satisfies whatever data-residency or regulatory constraints applied to the on-prem setup, since this is a category of failure that doesn't show up in a functional test but can be a real compliance problem.
- Gradual cutover with traffic-splitting. Start with a small percentage of inference requests routed to the new platform (or, better, shadow traffic that doesn't affect real responses), comparing predictions against the on-prem platform's output for the same input before trusting the new platform with any real decision.
- Validate parity before full cutover, not just at the start. Prediction distributions can drift subtly for reasons unrelated to a bug (a difference in floating-point precision, a slightly different preprocessing step), so parity validation should run continuously through the ramp-up, not just as a one-time gate at 1% traffic.
- Rollback procedures kept live throughout. Even after reaching 100% traffic on the new platform, keep the ability to fall back to on-prem serving for a defined grace period, since some failure modes (a rare input distribution, an edge case in preprocessing) may not surface until real production volume has run against the new platform for a while.
Worked example
A team migrating an on-prem inference platform to a cloud-managed model-serving product:
- Discovery: they confirm the managed product supports the model's existing serialization format directly, avoiding a costly model-conversion step that would have introduced its own risk of behavioral drift.
- Compatibility testing: running the same 10,000-request evaluation set against both platforms reveals predictions match within floating-point tolerance for 99.97% of requests; the remaining 0.03% are investigated and traced to a difference in how the two platforms handle a specific rare input shape, which is fixed in the preprocessing step before proceeding.
- Compliance checks: the team confirms the managed platform's regional deployment options satisfy the same data-residency requirement the on-prem setup was built to meet, a check done explicitly rather than assumed, since the managed platform's default configuration would not have satisfied it without an explicit regional setting.
- Gradual cutover: traffic ramps from shadow-only, to 5%, to 50%, to 100% over two weeks, with prediction-parity monitoring running continuously at each stage, and the ramp pausing for three extra days at 50% when a monitoring anomaly (later traced to an unrelated infrastructure issue, not the migration itself) needed investigation before proceeding.
- Rollback readiness: the on-prem platform remains available in a reduced-capacity standby mode for 30 days after reaching 100% traffic, providing a real fallback option rather than a purely theoretical one.
Trade-offs and pitfalls
The trade-off is migration speed against confidence: a managed platform is often adopted specifically to reduce operational burden, but rushing the cutover to capture that benefit sooner risks exactly the kind of subtle behavioral drift the compatibility testing step exists to catch. The pitfall to watch for is validating parity only at the start of the ramp and assuming it holds as traffic scales up; a difference that's invisible at 1% traffic (a rare input type, a load-related timing issue) can become visible only once real production volume and its full input diversity actually hits the new platform.
A machine learning system was built to retrain and serve in batch and now needs to support near-real-time updates and inference instead. How do you get there without breaking what's currently in production?
Sample Answer
Direct answer
Moving from weekly batch retraining to near-online learning and real-time inference is a genuine architecture change, not just a scheduling change, so the migration needs to introduce streaming data ingestion and online feature computation as new, validated components running alongside the existing batch pipeline, cutting over only once the streaming path has proven it produces equivalent (or better-understood, if intentionally different) results.
Structured elaboration
- Streaming data ingestion, built and validated independently first. Before touching model serving, stand up the streaming ingestion path and validate it captures the same events, with the same completeness and correctness, that the batch pipeline currently captures, running it in parallel with the batch pipeline rather than replacing it immediately.
- Online feature computation. Batch feature computation and streaming feature computation are genuinely different engineering problems: a feature that's a simple aggregate over a fixed historical window in batch may need a different implementation (a sliding window, an incrementally-updated aggregate) to compute correctly and efficiently in a streaming context. Validate that online-computed features match batch-computed features for the same underlying data before trusting them.
- Retraining cadence, moved deliberately and separately from serving. The retraining pipeline itself has to move off a purely weekly batch schedule, since "near-real-time updates" means the model, not just the features feeding it, needs to change on a much faster cycle. This doesn't have to mean full online or continuous learning right away, since that introduces its own stability risk (a single bad batch of streaming data corrupting a model that gets retrained and redeployed within minutes, with no chance for a human to catch it first). A safer intermediate step is a much more frequent, still-batch retraining cadence (hourly or event-triggered rather than weekly), gated by an automated check that only promotes a new model candidate to serving after it matches or beats the currently-serving model's validation metrics, rather than deploying automatically the instant training finishes. This is a genuinely separate engineering problem from streaming feature computation, with its own rollout and rollback path, distinct from the serving-side change below.
- Model serving changes. Moving to real-time inference likely means the serving infrastructure itself needs to change (lower-latency serving paths, handling feature staleness or missing features gracefully if the streaming pipeline hasn't caught up yet), which is a separate validated step from both the feature-computation change and the retraining-cadence change.
- Validation throughout, not just at the end. Compare online-computed features and predictions against the batch pipeline's equivalent outputs for the same underlying events, on an ongoing basis during the transition, since a divergence discovered only after full cutover is much more expensive to diagnose than one caught during a parallel-run period.
- Rollback strategy. Keep the batch pipeline capable of running as a fallback until the streaming path has proven reliable across enough real-world conditions (including whatever peak load or unusual traffic patterns weekly batch runs never had to handle in real time), since streaming systems have failure modes (backpressure, out-of-order events) that a batch job simply doesn't encounter.
Worked example
A team modernizing a legacy ML pipeline currently retraining weekly on batch features:
- They build streaming ingestion for the underlying event data first, running it in parallel with the existing batch collection for a month, and validate that the streaming path captures the same event volume and content as the batch path processes, catching a gap early where a specific event type was being silently dropped by the new streaming consumer's schema validation, a bug that would have caused silent feature staleness in production if it had gone straight to serving.
- For online feature computation, they identify which existing batch features can translate directly to a streaming equivalent (simple counts and sums over a rolling window) and which need real redesign (a feature that depended on a full historical batch join, which has no clean streaming equivalent and needs to be reworked or approximated); features are validated one at a time, comparing streaming-computed values against the batch pipeline's values for the same underlying data.
- Retraining cadence: the retraining pipeline itself moves from a weekly cron job to a 4-hour cadence triggered off the now-available streaming feature store, with an automated validation gate that compares each new model candidate against the currently-serving model on a held-out recent window and only promotes it if it matches or improves on the existing model's key metrics, so a bad batch of streaming data can degrade a candidate without ever reaching production.
- Model serving is updated to handle the case where a streaming feature is momentarily stale (the streaming pipeline is a few seconds behind for a specific user) by falling back to the last known-good batch-computed value rather than serving with a missing feature, an explicit design decision made after early testing revealed the naive approach (erroring on a stale feature) caused an unacceptable rate of failed predictions during normal streaming lag.
- The team runs the new streaming inference path fully in shadow mode (computing predictions without serving them) for two weeks, comparing its outputs against the current production model's predictions, which is still retrained weekly during this validation period since the faster 4-hour retraining cadence is only switched on after the streaming ingestion, feature, and inference paths are already trusted on their own, before cutting any real serving traffic over, and keeps the original weekly-batch pipeline runnable as a fallback for a full month post-cutover.
Trade-offs and pitfalls
The trade-off is real engineering investment (rebuilding feature computation for a fundamentally different execution model) against the business value of fresher predictions, which is worth carefully validating is actually needed before committing to this scope of change, since not every use case genuinely benefits from near-real-time freshness enough to justify it. The pitfall that shows up in practice is treating streaming feature computation as a mechanical translation of batch logic, when several features genuinely need to be redesigned rather than ported, and discovering that redesign need late in the migration, after committing to a timeline that assumed a simpler translation, is a common and costly surprise.
That is every published Legacy Modernization and Architecture Evolution question for Machine Learning Engineer so far. Browse the other topics in this category, or practice this one interactively.