Product and Engineering Collaboration Questions

Direct answer

Do both, in sequence, and say so out loud. Contain the problem today with a small, reversible stopgap, then spend one or two days finding the real cause before committing three to four weeks to a refactor. A hotfix is a quick, narrow patch shipped outside the normal release cycle: it answers "stop the bleeding". A refactor restructures code without changing what it does: it answers "stop it recurring". They are not rivals. The decision is what each buys, what each costs, and in what order.

Deciding what is now and what is later

Test every candidate action with four questions:

1. Does it shrink customer harm today (missing conversion rows lead to wrong decisions on dashboards)?
2. Is it small and reversible (hours of work, easy to switch off)?
3. Does it hide the failure? A retry that silently succeeds can mask a growing problem, so every stopgap ships with an alert.
4. Does the refactor depend on knowing the cause? If nobody has reproduced the failure, a refactor is a bet, not a fix.

Today: add a stopgap, meaning a temporary measure that reduces harm without fixing the cause. It has four parts: an automatic retry that is idempotent (running it twice gives the same result, with no duplicate rows), an automated backfill (re-loading the days a failed run missed), an alert when a day's conversion count is missing or far below its usual range, and a visible "data as of" note on the dashboards.

Next one to two days: a time-boxed root-cause hunt. Pull the logs of the recent failures and look for the common trigger. Typical suspects: an upstream schema change (the source team renamed or retyped a column and the load breaks), late-arriving data (yesterday's events land after the nightly job has already run), a memory limit (the job crashes when a day's data is larger than usual), or a rate limit (a source API refuses calls after a quota). Time-boxed means the hunt has a fixed deadline, here two days; if no cause is found by then, we decide with what we have instead of searching indefinitely.

Then decide. If the cause is narrow, the refactor may shrink to a much smaller fix. If it is structural (the design cannot tolerate late data), commit to the refactor and slice it so the riskiest stage ships first, with value landing in weeks one and two instead of week four. For example, if the cause is late-arriving data: week 1 makes the load step re-process the previous three days each night, which removes the most common failure; week 2 adds a check that rejects a day with too few rows; weeks 3 and 4 restructure the rest. After week 1 most of the benefit is already in production.

Worked example (illustrative numbers)

Suppose the pipeline failed on 6 of the last 30 nights, which is 20%. A refactor of three to four weeks covers 21 to 28 nights, so with no stopgap you would expect roughly 4 to 6 more failing nights (21 x 0.2 = 4.2 and 28 x 0.2 = 5.6), each one a hole in the dashboards until someone notices. With retry, backfill and an alert, each failure becomes a visible gap of hours rather than a silent missing day. Product gets its hotfix today in the safe form. Engineering gets its refactor after a two-day diagnosis, with a start date.

Keeping product and customers informed

• Product: a short written note today covering what we know, what we do not know, what we are doing, the time of the next update, and one plain sentence on the trade ("the stopgap shortens the gaps, it does not remove failures"). Then updates at the promised times, even if the update is "no change". Give the refactor milestones: the diagnosis decision date first, then weekly.
• Customers or dashboard users: a banner on affected dashboards ("conversion data for 14 March is delayed, expected corrected by 10:00"), corrected figures after backfill, and a note when the permanent fix lands.
• Support and customer success: a two-sentence script so they say the same thing customers see on the banner.

Trade-offs and pitfalls

• Stopgaps become permanent. Put "remove the stopgap" into the refactor's definition of done.
• Never promise "this will not happen again" from a hotfix.
• A refactor pitched as all-or-nothing loses the argument. Slice it and show early value.
• What would change my call: if failures produce wrong numbers (not just missing ones), pause downstream use of the data first. If diagnosis finds a one-line cause, skip the refactor and say so.

Direct answer

I would not fight the date; I would turn engineering's concerns into evidence, offer ways to hit the date with bounded risk, and, if leadership still decides to go, write the decision and its rollback conditions down and then carry it out fully. Arguing hard before a decision and committing after it is how engineering stays credible with leadership.

Structured elaboration

1. Understand what the date protects. Ask why it is fixed (a public announcement, an event, a contract) and what is flexible: scope, audience, speed of rollout.

2. Bring evidence, not opinion. For each concern: what could fail, how likely, how bad, how we would know. Examples: a load test at the forecast launch peak; a list of untested paths; what happened in the last comparable launch.

3. Offer options that keep the date

I would recommend B, with the exact cap and widening schedule agreed in the decision record.

4. Settle a disagreement about readiness with a test. If product thinks current monitoring is enough and engineering disagrees, replace the argument with a test: replay the last few real incidents against the current alerts, or inject a known failure in a staging environment (a rehearsal copy of the system) and see whether an alert fires and how fast. If it fires, I concede. If not, I ask for the minimum, such as three alerts and one launch dashboard, not a monitoring overhaul. If we still disagree, I escalate once, in writing, as a single yes-or-no question to the executive sponsor (the senior leader who owns the launch and can decide) at a launch-readiness checkpoint (a scheduled meeting shortly before launch where the team reports whether it is ready), with both views on one page.

5. Mobilise once the date is approved. Once leadership has approved a plan that raises operational risk, mobilise rather than re-argue: name an owner for each risk (on-call cover for launch week, runbooks, which are step-by-step instructions for handling a known failure, and a rehearsed rollback, meaning returning to the previous version, practised beforehand), renegotiate scope on the items that add the most risk, and update the roadmaps and dependent teams' plans, telling each what moved. An owner list can be three lines: "Launch-week on-call: Maya. Runbook for payment errors: Luis, due 5 days before launch. Rollback rehearsal: Dev, 3 days before launch." The message to a dependent team might read: "The launch date held, so the reporting feature moves from sprint 14 to sprint 15. You can start the data export work now. I will confirm by Friday whether the API freeze date moves."

6. Document the decision and the rollback conditions. Write a one-page decision record: the decision, who made it, the engineering concern, the risk leadership accepted, mitigations with owners and dates, the rollback triggers (measurable conditions that mean we undo the launch) and who may pull them. An excerpt follows.

Worked example: decision record excerpt (illustrative thresholds)

Decision: launch publicly on the committed date with staged enablement. Decided by: the executive sponsor, recorded after the readiness review. Engineering concern: untested behaviour under launch-day load. Accepted risk: early users may see slower pages.

Rollback triggers (any one, sustained as stated): error rate (the share of requests that fail) above 2% of requests for 10 minutes in a row; checkout response time above 3 seconds for 15 minutes; any confirmed data-integrity or security issue, immediately. Rollback authority: the on-call lead, without waiting for approval. Review: a short review two weeks after launch comparing what happened with what engineering predicted.

Trade-offs and pitfalls

• Staged enablement caps early demand; leadership must accept that cost knowingly.
• A rollback condition nobody is willing to act on is decoration, so name the authority in advance.

What would change my call

If the load test fails even at the forecast peak, or the risk is irreversible (data loss, security), I push to move the date and say so plainly.

Closing the loop

After launch, review with leadership what we predicted against what happened, so the next negotiation starts with a track record.

Direct answer

In the meeting I would not say "no". I would say what I can do by Friday, name the specific harm of the rushed version, and offer a safer path to the same outcome. Then I would write the agreed trade-off down before leaving the room, so the decision and the accepted risk have names on them.

In the meeting

1. Ask what the 48 hours protects (a customer promise, a demo). Understand the goal before arguing about the date.
2. State the risk concretely, in the PM's language: "If we ship this migration unreviewed, it can overwrite customer addresses, and we cannot restore them." A migration is a script that changes existing stored data in bulk. Another line I would say: "I can have Thursday's version ready that only shows what it would change and writes nothing. That gives you something to show on Friday, and the real run follows after review." The words "hard to undo" are the heart of it: a mistake that can be reversed is a cheap risk; one that cannot is expensive.
3. Separate the reversible part from the irreversible part.

What I offer instead

• Option A: in 48 hours, ship the reversible part (for example, read-only or dry-run output: the feature runs and reports what it would change, but writes nothing), behind a feature flag (a switch that turns the feature off without a new deploy).
• Option B: the full feature in about 5 working days with a review and a rollback plan (the exact steps to return to the previous state if it goes wrong).
• Option C: the full feature in 48 hours only if the PM's manager accepts the named risk in writing, with a tested backup of the data first.

I recommend A, then B. If the PM chooses C, I record it and do what I can to reduce harm, not refuse.

Recording the trade-offs

Post a short decision record in the team channel or ticket the same day:

Decision: Ship dry-run address cleanup Thursday behind a flag; real write after review (target 5 working days out).
Context: Sales wants it for Friday's customer call. The write step cannot be undone without a backup.
Options considered: A) dry-run now, write later  B) full in 5 working days  C) full in 48h, no review
Chosen by: PM (what and when), tech lead (risk statement)
Risk accepted: Customer sees a preview, not the cleaned data, on Friday.
Revisit: review at the 5-day mark, owner: tech lead

Trade-offs and pitfalls

• A bare refusal teaches product to stop asking engineers early.
• Do not bury the irreversible risk in jargon; the PM cannot weigh what they cannot picture.
• If the PM overrules me, I still write the record. It protects the relationship as much as the team: everyone later sees the same facts.

Direct answer

I timebox the decision to about an hour of two engineers' time, decide on a few written criteria, and usually land on the middle path: a minimal preparatory refactor sized to the feature, then the feature, with the larger cleanup scheduled separately. The question is never "refactor or ship" in the abstract. It is "how much does this specific code change raise the risk or cost of this specific feature, and of the next one?"

How much time to spend assessing

About 60 minutes, with the tech lead and one engineer who knows the code, plus the product manager (PM) for the first ten. If the answer needs more than that, the unknown itself is the finding, and the next step is a short spike (a timeboxed experiment to learn something, not to ship).

Criteria that decide it

1. Is the refactor in the path of the feature? If the feature must change the messy code, prepare it first. If not, defer.
2. Regression risk: is there test coverage on the code we will touch? None means a change can break things silently.
3. Can it be split? The smallest refactor that makes the feature safe, versus the full cleanup.
4. Cost of waiting: does the next feature compound the problem?
5. Reversibility and date: what does product lose if the feature lands one sprint later?

Facilitating the sprint-planning conversation

When product and design push new work and engineering warns about regressions, I bring a one-page decision aid: options, extra days, regression risk, what product gets, and when. The people in the room are the PM, the designer, the tech lead and whoever owns testing. The PM decides what and when; the tech lead states the risk plainly; the sprint commitment is written after the choice.

Worked example (illustrative numbers)

Feature: a discount-code field in checkout, 3 days. It touches a shared tax-calculation function with no tests. A sprint has 10 working days; 6 are free for this work.

I choose C: it leaves 1.5 days of slack, and the remaining refactor (3.5 days) goes onto the next sprint's plan, visible to product.

Pitfalls

• Assessing for two days to decide a one-day question.
• Using "refactor first" as a default: it delays the customer and often grows past its estimate.
• Using "ship now" as a default: the cost shows up later as slower, riskier work. What would flip my call: a hard external date with no safe slice, in which case I ship with a flag and record the refactor as owed.

Direct answer

I would tell a story where I stopped treating it as "product vs engineering" and got both sides to agree on one shared problem, then traded explicitly. In my example, product wanted to ship new onboarding features as often as every day or two to learn from users (a user-experience goal), while engineering wanted a slower weekly release because each deploy was manual and risky (a developer constraint). A deploy is putting new code into the live product; a rollback is undoing a deploy by returning to the previous version; a hotfix is an urgent fix pushed outside the normal schedule. I got both to look at the same release history, proposed a trade that gave each side what it needed most, and we measured the result after six weeks.

The story, step by step

Situation. A team of 8 engineers, a six-week plan, and two camps. Product wanted a faster signup flow with several user tests. Engineering wanted the last two deploys' rollbacks fixed before anything new went out. The sticking point was deployment cadence (how often code goes to production): product wanted near-daily deploys, engineering wanted one a week because every deploy needed a two-hour manual regression check (a person clicking through the product to confirm that old features still work after the change).

What I did, in order:

1. Asked each side what it was protecting, not what it was demanding. Product was protecting learning speed (every week of delay was a week without user feedback). Engineering was protecting stability and its own evenings (rollbacks happened after hours). Writing both down showed that neither goal was unreasonable.
2. Put one shared fact on the table. I pulled the last ten releases: four needed a hotfix or rollback. That made "slow down" and "ship faster" both partly right, and it moved the argument from opinion to data.
3. Proposed a trade with a price tag. Two engineers spend two weeks automating the smoke tests (a short set of quick checks that the most important paths, such as signup and login, still work) and putting new work behind feature flags (switches that let code ship switched off and be turned on later without a new deploy). Product keeps its fast cadence because code can ship dark (deployed to production but switched off for users) every day, instead of waiting for the weekly release train (the fixed weekly slot at which all finished changes go out together). Engineering controls exposure because a flag can be turned off in seconds. Product also agreed to cut the first release to the first step of the new flow.
4. Made the decision visible and owned. In the planning meeting I said to the product lead: "You keep daily shipping. In return the first release is only step one of the flow, and two of our engineers spend two weeks on the test automation." To the engineering lead: "You get a switch to turn anything off in seconds, and the manual check goes away once the automation covers it. Are you comfortable with that?" I then wrote a half-page note with the trade, the cost, and a checkpoint at week three, and both leads signed off in the planning meeting. I owned running the checkpoint.

Worked example: the numbers in this story

These figures are illustrative placeholders; use your own real ones in an interview.

• Capacity: 8 engineers x 6 weeks = 48 engineer-weeks. The automation pair cost 2 engineers x 2 weeks = 4 engineer-weeks, which is 4 / 48, about 8% of the plan.
• Baseline: 4 of the last 10 releases needed a hotfix or rollback (40%).
• Outcome after six weeks: 1 of 10 needed one (10%), and product shipped three flagged experiments instead of waiting for the weekly train. With only ten releases each side, I presented that as an encouraging but statistically weak signal (a Fisher exact test, a standard small-sample significance test, gives p of about 0.30 for 4 of 10 versus 1 of 10, meaning a gap this size could easily be chance) and kept watching it for another quarter rather than calling it proven.

What I would do differently

I would bring the release-history data to the very first conversation instead of the second, and I would agree on a standing reliability share of capacity up front so we did not have to renegotiate it feature by feature.

Trade-offs and pitfalls

• Declaring a winner ("we do reliability first") satisfies one side and costs the relationship. A trade both can explain to their own team lasts longer.
• Vague outcomes ("things got better") sound like a story you invented. Name the measure before you tell it.
• Skipping the cost of the compromise (the 8% of capacity) makes it sound free and invites the follow-up "so what did you give up?"