Product and Engineering Collaboration Questions

Direct answer

I would not argue "reliability vs features". I would give product leadership a short, honest account of what happened, show the cost of a repeat in terms they track (revenue, customers, a deal or a compliance exposure), and ask for a bounded, time-boxed slice of capacity with a checkpoint where they can stop it. In the illustrative plan below that slice is 34 of 260 engineer-weeks in a quarter (about 13%), so about 87% of capacity stays on features.

Post-incident narrative (the one page I would hand the executive team)

Illustrative figures throughout; swap in your real ones.

• What happened: the checkout service was down for 95 minutes after a release. A database connection limit (the maximum number of simultaneous links the database allows; once reached, new requests are refused) was reached under load, and the rollback (returning to the previous version) was slow because it was manual.
• How long each step took (illustrative): the failure began at 14:00, the alert fired at 14:12, so time to detect was 12 minutes; service was restored at 15:35, so time to restore was 95 minutes from the start (14:00 to 15:35), or 83 minutes from the alert.
• What it cost: at an assumed $40,000 of checkout revenue per hour, direct lost revenue was 95 / 60 x $40,000, about $63,000. It also triggered support tickets and one enterprise customer's request for a service-credit review (a request for a refund or discount that the contract gives customers when uptime falls short).
• Worst case, not just what happened (the tail: a rare but much larger outcome that averages hide): the same failure on the busiest sales day (assume 4 hours and 3x normal traffic) would be 4 x $40,000 x 3 = $480,000 plus contractual and reputational damage. If any customer contract carries an uptime commitment or a regulatory requirement, say which one.
• What it is not: a request to stop features. It is a request for a small, dated, reversible investment with a decision point.

The proposal

Total: 16 + 18 = 34 engineer-weeks out of 20 x 13 = 260, about 13%. At an assumed $4,000 loaded cost per engineer-week (salary plus benefits and overhead, divided by weeks) that is $136,000. A connection-pool alert warns when the database is close to its connection limit, and a dependency review checks which other systems a service relies on and how each could fail.

Honest cost-benefit

$136,000 / $63,000 is about 2.1, so on direct revenue alone the plan pays back only if it prevents roughly two outages of this size. I would say that out loud: "On direct revenue alone this costs about twice what this outage did, so I am not claiming it pays for itself on one incident. I am asking because the same failure on our busiest day would cost several times more, and the cause is still in the system." The case rests on the tail (the $480,000 scenario), service-credit and renewal exposure, and the fact that the same root cause is likely to recur if nothing changes. If leadership will not accept that argument, I would offer the pilot below instead of the whole plan.

The pilot and how we track it

• Pilot: do only the short-term phase on checkout (16 engineer-weeks, about 6% of the quarter), review at week 4, and release the medium-term money only if the checkpoint shows progress.
• Tracking: change failure rate for the service (releases that caused an incident divided by releases), time to detect and time to restore, number of repeat-cause incidents (new incidents with the same root cause as an earlier one), and whether the service met its service-level objective (SLO, the reliability target agreed with the business). Time to restore is the clock time from the start of the failure to recovery, as in the 95-minute example above. Review these in the same weekly meeting that tracks feature delivery, so reliability is not a separate conversation.

What would change my call

• If the root cause was a one-off human error that a single guardrail now prevents, I would shrink the ask to the short-term phase only.
• If another outage lands during the pilot, I would raise the share and say so before being asked.
• If product has a launch date that cannot move, I would keep the plan and trade scope on the feature side rather than drop the guardrails.

Pitfalls

• Presenting reliability as a moral duty; executives respond to risk priced in their own terms.
• Asking for an open-ended percentage with no end date.
• Skipping the cost line: an unpriced ask looks like a blank cheque.

Direct answer

Trust returns through fast, honest, specific communication followed by visible follow-through. I would speak first with facts and a time for the next update, stop the harm for the affected users, then fix the release process and report back on what changed. A regression is a feature or behaviour that worked before and broke after a release. Here it hit a subset of users, so one early job is to identify exactly who.

What I tell them now (first hours)

Product and customer-facing teams need five things: what is broken, who is affected (the cohort: the group of users sharing a trait, here the annual-plan accounts, with a way to check a given customer), what customers can do meanwhile, when the next update comes, and who owns it. Do not guess at a cause or a fix time you cannot defend. Example update:

"14:10. Since the 13:00 release, customers on the annual plan cannot export reports; about a quarter of accounts are affected, list attached. Cause not confirmed. Workaround: use the monthly view. We are preparing a rollback and will update at 15:00 whatever the status. Owner: the on-call engineering lead, reachable in the incident channel."

Reducing customer impact in the short term

1. Roll back (return to the previous release), or switch the change off with a feature flag (a configuration switch that turns one feature on or off without redeploying), if that is safe. Prefer the fastest reversible action.
2. If rollback is not possible, ship the narrowest fix, with a second engineer reviewing it.
3. Give support a macro (a ready-made reply) and the affected-account list for proactive outreach.
4. Repair damaged data for the affected users and confirm it.

What changes so it does not recur

Hold a blameless review (focused on how the process allowed it, not who caused it), and agree a small number of changes, each with an owner and a date (for example, "regression test: owned by the engineer who wrote the change, due Friday"):

• A regression test (an automated test that fails if this exact failure comes back) that reproduces this failure.
• Staged rollout: release to a small share of users first and watch error rates by cohort before widening. For example, 5% of accounts for one day, then 25% for one day, then 100%, moving on only if the error rate stays at or below the old release's rate. Comparing by cohort matters: a 1% error rate overall can hide 4% among annual-plan accounts if they are a quarter of traffic and the rest have none (0.25 x 4% = 1%).
• Automatic rollback criteria agreed beforehand.
• Release notes sent to product and support before release, listing what could affect customers.
• For a model or data change: evaluate performance on segments, not just the overall average, because an average can hide a subset getting worse. For example, a model with 94% accuracy for 90% of users and 64% for the other 10% averages 0.9 x 94 + 0.1 x 64 = 91%, which looks fine while one group is badly served.

Owners and dates for those changes (illustrative): regression test, the engineer who wrote the change, due Friday; staged rollout, the release engineer, live within two weeks; automatic rollback criteria, the on-call lead with product agreeing the threshold, due within two weeks; release notes to support, the product manager, from the next release; segment evaluation for model or data changes, the owner of that model, before its next release.

Rebuilding the relationship

Trust is rebuilt by keeping small commitments. Send the updates at the times you promised, share the review conclusions within a week, and report back two weeks later on whether each change was done. Invite product into the release-readiness check (the short pre-release review of risks, tests and rollback plan) so they see the safeguards and share ownership. A report-back message, two weeks later, might read: "Of the four actions from the review, three are done: regression test added, staged rollout live at 5%, 25%, 100%, release notes now go to support a day ahead. The automatic-rollback threshold is still open and due Friday. Affected accounts: all 14 outreach calls made, no open tickets."

Pitfalls

Over-promising ("this will never happen again"); hiding the affected cohort; blaming a person or team; adding heavy approvals that slow every release when a staged rollout would catch the problem more cheaply.

Direct answer

I would not say yes or no at the first conversation. I would turn "I suspect it will break capacity" into a number: how much extra load, when, against what limit. Then I would recommend a staged launch with explicit gates, so product still gets the feature early while the system is protected. The recommendation is the output of the arithmetic, plus a plan for what to do if the arithmetic is wrong.

What I ask product

• Expected adoption: how many users, how fast, on which days? Is there a marketing event that causes a spike?
• Per-user behaviour: how many requests does an active user make through this feature?
• Deadline logic: is the date fixed by a contract or event, or preferred?
• Flexibility: can the feature launch to a segment first, or with reduced functionality?

What I ask telemetry

• Current peak requests per second (RPS), and growth trend.
• Utilisation at peak: CPU, memory, connection pools (the fixed number of reusable database connections the service can hold open), database load.
• Latency at the 95th percentile (P95: 95% of requests are faster than this) and error rate as load rises.
• The tested saturation point from a load test: the load where latency or errors degrade. If there is no load test, that is the first task.
• Lead time to add capacity (minutes for autoscaling, where the platform adds servers automatically when load rises; weeks for database changes or quota requests).

Assess and decide

Set the rule: peak load stays at or below 70% of tested saturation (the load level where latency or errors start to degrade in a load test). The 70% is a rule of thumb, not a law. The margin covers forecast error, traffic spikes above the daily peak, losing a server during peak, and the fact that systems slow down sharply as they approach their limit because requests start queueing. A team with a slower scale-up or a less certain forecast would pick a lower share; a team that can add capacity in minutes could run higher.

Worked example (illustrative)

Tested saturation is 7,500 RPS, so the ceiling is 7,500 x 0.7 = 5,250 RPS. Today's peak is 4,000 and organic growth will take it to 4,400 by launch. Product forecasts the feature adds 1,200 RPS at full adoption.

At full rollout we would need a saturation point of 5,600 / 0.7 = 8,000 RPS, about 6.7% above the tested 7,500. So the recommendation: launch to 10%, then 25%, then 50% behind a flag, holding each stage long enough to see a full peak day; in parallel, add the capacity (or remove the hot spot found in the load test: a single table, queue or server that takes far more traffic than the rest and hits its limit first) before the 100% step. Time to market is barely affected: users get the feature at stage 1, and the final step waits for roughly one capacity change.

Pitfalls

• Treating a forecast as fact. The gates exist because forecasts are wrong; watch live utilisation at each step.
• Averages hide peaks and the first bottleneck may not be the one you load-tested.
• What would change my call: a load test showing the saturation point is far lower than assumed, or a hard launch date with no capacity lead time, which turns it into a conversation about descoping the feature.

Direct answer

Build one page for the quarter where every piece of work, whether a feature, a reliability improvement or maintenance, is a row with the same columns and is paid for from the same capacity number. Once all three kinds compete for one budget, the conversation changes from "can we squeeze this in?" to "what moves below the line if we add this?". The line is the capacity line on the page: items above it fit in the quarter's capacity and are committed; items below it do not fit and wait. I would use it as the agenda of a fortnightly planning review.

What the page contains

1. Capacity header: engineers x weeks, minus a reserve for leave, on-call and interruptions.
2. Initiative table, one row per item, with these fields: ID, name, type (feature, reliability, maintenance), product goal supported, owner, effort in engineer-weeks, dependencies, risk, and the cost of not doing it.
3. A capacity line: items above it are committed, items below it are not.
4. A decision log: date, decision, who decided, reason. A sample entry: "Week 3. Moved F2 below the line to fit the partner feature. Decided by: product lead, with engineering lead consulted. Reason: partner contract renewal. Reopen if capacity grows."

Worked example (illustrative)

8 engineers for a 12-week quarter is 96 engineer-weeks. Holding back 20% for leave, on-call and interruptions leaves 76.8 plannable weeks (96 x 0.8).

Capacity line: at 76.8 plannable weeks, all seven rows (75 weeks) sit above the line and are committed; a new row is placed below the line unless something above it moves down.

Totals: features 45, reliability 18, maintenance 12, all 75 weeks (45 + 18 + 12), leaving 1.8 unallocated. That is a split of 60%, 24% and 16%. It is a starting point to negotiate, not a rule.

Using it in conversations

• When a request arrives: "It costs 12 weeks. We have 1.8 spare. Which item moves below the line?" The trade becomes visible and belongs to whoever asks.
• For reliability items, bring evidence in product terms (incidents, customer minutes affected) so they compete with features on impact.
• The same page works when requests come from outside engineering, for example product asking a data team for a report. An intake form feeds new rows into the table: requester, the decision or customer need it supports, deadline and why, who uses it, rough size, and triage checks (is it a duplicate of existing work? is the data available? is there a named owner?). Unqualified requests wait. A filled-in example (illustrative): "Requester: PM B. Need: churn report for the renewal review on 14 March. Users: 6 account managers. Size: 3 engineer-weeks. Triage: no duplicate, data available, owner named. Result: becomes row F4 and competes with the rest."

Cadence, participants and decision rules

Fortnightly 45-minute review with the product lead, engineering manager, SRE or tech lead, and a design representative. Rules: anything over 5 engineer-weeks, or anything that displaces a committed item, is decided in the review; product owns feature order, engineering owns the evidence and minimum bar for reliability and security; unresolved ties go to the director of product and director of engineering, and the outcome is written in the decision log.

Presenting trade-offs to stakeholders

Show two or three scenarios, each with what ships, what slips and the risk, and recommend one. Example: adding a 12-week partner feature means moving F2 below the line (its 15 weeks plus the 1.8 spare cover the 12), and the product lead decides whether that is the right trade.

Pitfalls

A page that only lists features; reliability rows without impact evidence; refilling the reserve with scope because it looks idle.

Direct answer (the story in two sentences)

I argued for a two-sprint stabilisation period before launching a partner integration; product leadership shipped on the contracted date instead. I committed to the decision, gathered evidence of what happened, repaired the relationship, and changed how we record risk so the next disagreement started from shared facts.

Situation and what I argued

I was the engineering manager. Leadership had promised a partner a launch date. My team had found that the integration's retry logic could pile up requests when the partner's service slowed: each failed call is retried automatically, so if the partner slows and 1,000 calls fail, three retries each turn that into about 4,000 calls hitting an already struggling service (illustrative). I asked for two sprints to add limits and test failure behaviour. My argument was the risk of cascading failures (one overloaded service making the services that depend on it fail in turn) during launch week, with the cost of a visible outage to the partner relationship. I also warned that a retry storm could cause queued partner orders to be dropped.

Why the decision went the other way

The director of product weighed two things I had underweighted: a contractual date tied to the partner's own marketing, and a belief that we could patch quickly after launch. I had talked only about risk, and had not quantified the cost of delay from their side. Cost of delay means what each week of waiting costs the business. For example, if the partner's launch campaign was worth about $15,000 of contract revenue and credits per week, two sprints (4 weeks) cost about $60,000, and I never put a number like that next to my risk. That was my miss.

How I reacted

I was frustrated, and I said so privately to my manager rather than to the team. I then did what I would want from a peer: I said in the meeting, roughly: "I still think two sprints is safer, and I'll say that once for the record. The decision is made and I'll commit fully. Can we add two things that cost days, not sprints?" The two safeguards were a kill switch (a control that turns the integration off in seconds without a new release) and alert thresholds (pre-agreed limits, such as retry rate above 5%, that notify the on-call engineer). They agreed.

Finding out whether I was right (honestly)

In the first month, retry pile-ups caused two incidents, both contained by the kill switch within minutes. I had been right about that mechanism. I had also predicted data loss, which did not happen, so I was partly wrong. I presented the incident timeline in a blameless review (a post-incident discussion about system causes, not blame), with no "I told you so", and included the cost of delay as product had described it, so the conclusion read fairly.

Rebuilding trust

I met the product director one-on-one, credited her for the date being met, and asked what evidence she would need from engineering to delay a launch in future. That conversation changed the relationship more than the post-incident report did.

Structural changes

1. A launch-readiness checklist where every unmet item names the person who accepts that risk, recorded in a decision log (a shared list of decisions with date, owner and reason). A sample row: "Retry limits not load-tested. Risk accepted by: Director of Product. Review date: two weeks after launch."
2. A short pre-launch risk brief from engineering that includes the cost of delay, written with product.
3. Reliability work scheduled in planning as visible line items, not hidden from product.

What I would do differently

Quantify the cost of delay with product before the debate, and bring a smaller proposal (the kill switch) at the start, not as a fallback.

Pitfalls

A lost-debate story that casts the other side as foolish, or claims total vindication, signals low self-awareness.