DevOps Culture and Delivery Practices Questions

Direct answer

Every change to production automation, infrastructure code (infrastructure described in files, often called IaC, such as Terraform or CloudFormation templates) and runbooks (step-by-step guides for operating or recovering a system) goes through the same route as application code: a pull request, automated checks, a human review, and then an apply performed by a pipeline rather than by someone's laptop. I do this because these changes have a larger blast radius (the amount of the system damaged if the change is wrong) than most application code, and because the pull request becomes the audit trail.

Day to day, in order

1. Branch and pull request. The author describes the intent and the expected effect, and links the ticket or incident.
2. Automated checks. Format and lint, policy checks (for example, no public storage buckets), and a preview of what will change. For infrastructure code that preview is the tool's plan output: a list of the exact resources the change would add, change or destroy, produced without touching production. Terraform and CloudFormation are two common tools that read the infrastructure files and make the real system match them. CloudFormation calls its equivalent a change set (a reviewable list of the resources a stack update would create, modify or replace) rather than a plan. A simplified Terraform-style plan looks like this:

Plan: 1 to add, 1 to change, 0 to destroy.
  + security_group_rule.api_ingress   (new: allow port 443 from the office network)
  ~ database.orders                   (backup_retention_days: 7 -> 14)

The reviewer reads the plan as well as the diff. A line marked for destroy, or a change to something the author did not mention, is where the review slows down.
3. Review. One reviewer from the owning team reads the diff and the plan. Anything with high blast radius (networking, identity, databases, shared automation) needs a second reviewer. The reviewer asks "what is the worst case and how do we undo it?"
4. Apply from the pipeline. To apply is to execute the approved plan against the real system. Credentials for production live in the pipeline, so no one runs changes by hand, and every change is attributable.
5. Runbooks. Reviewed like code. The reviewer should be someone who did not write it, and ideally runs the steps in a non-production environment, since a runbook that has never been followed is a guess.

Why this setup

• A second pair of eyes catches the errors the author cannot see (wrong environment, wrong account).
• The plan output makes reviewers judge the effect, not just the text.
• Pipeline-only apply gives an audit trail and prevents drift: the real system quietly differing from what the files say, which happens when someone edits production by hand and forgets to update the code.

Exceptions

During an incident, the on-call engineer can make the minimum change to stop the harm, with a second engineer on the call as a witness (someone who watches and can confirm what was done), and a pull request raised afterwards for review within one business day. I keep the exception narrow so it does not become the habit.

Pitfall: approval fatigue. If every change needs two reviewers, they rubber-stamp. Scale review to risk.

Direct answer

I would start by measuring, not changing anything: a baseline of deployment frequency, change fail rate, change lead time (commit to production) and recovery time (how long it takes to restore service after a failed deploy), plus a breakdown of where a change spends its time. Then I would shrink batch size and remove waiting, because both frequency and safety improve when changes are small. Crucially, "do not raise the change-fail rate" is a weaker promise than it sounds, so I would raise that with leadership at the start.

The catch hidden in the goal

Illustrative baseline: 60 deploys a month, 10 percent fail, so 6 failed deploys. Double the frequency to 120 deploys and hold the rate at 10 percent: 120 x 0.10 = 12 failed deploys a month. The rate is unchanged, but customers see twice as many failures. To keep the absolute number of bad deploys at 6, the rate must fall to 6 / 120 = 5 percent. I would propose a goal of doubling frequency while reducing fail rate or at least holding recovery time down, and say why.

Step 1: measure first

• The four numbers above, with agreed definitions of "deploy" and "failure".
• A timeline of a typical change: waiting for review, waiting for the build, waiting for an approval board or test environment, waiting for a release slot. The longest wait is usually the first target.

Step 2: organisational changes

• A reviewing norm: first response within a business day, so changes are not stuck in a queue.
• Replace slow change approval boards (committees that approve each release) for low-risk changes with pre-agreed rules; keep human approval for genuinely risky ones.
• Give teams ownership of their own release, including the pager (being on call, the first to be alerted when it breaks), so they feel the cost of a bad release.
• Agree that a failed deploy leads to a blameless review (a post-incident discussion about how the process allowed the failure, not who to blame), so failures are reported.

Step 3: technical changes

• Smaller changes merged to a shared main branch daily (trunk-based development), with unfinished features behind feature flags (switches that hide unfinished code until it is turned on).
• Automated tests and builds fast enough to run on every merge.
• Staged rollouts (sending a change to a small share of users first, then more) and the ability to switch a change off quickly, which keep each failure cheap. Safer releases make a higher frequency tolerable: a staged rollout limits how many users a bad change reaches, and a quick switch-off or rollback shortens how long they are affected, so each failure stays cheap even when there are more deploys.

How I would know it is real improvement, not gaming

• Count rules fixed in advance: a deploy is a change to production code; empty or configuration-only deploys and artificial splitting do not count unless stated.
• Watch the pair: frequency rising while change size distribution, lead time, and absolute failures behave sensibly.
• Cross-check against things people cannot easily fake: customer-visible incidents, support tickets, and a developer survey.
• Review the numbers with the teams, so a jump in frequency prompts the question 'what changed in how you work?'

Sequence over the two quarters

• Weeks 1 to 3: baseline and the wait-time timeline, plus agreeing the counting rules.
• Weeks 4 to 8: the first action is the review norm and replacing approval-board steps for low-risk changes, because they are cheap and remove the longest waits.
• Weeks 6 to 18: faster automated tests, trunk-based development and feature flags, team by team.
• Weeks 14 to 26: staged rollouts and ownership of release and pager, then re-read the baseline at week 26.

Trade-offs: a two-quarter horizon is tight for cultural change; I would sequence the quick wins (review latency, approvals) first and the testing investment alongside.

Direct answer

I would track the five delivery measures from DORA (DevOps Research and Assessment, a long-running research programme on what makes software teams deliver well), taken from dora.dev's current definitions, and read them together rather than one at a time. Three describe speed (throughput: how much change flows through) and two describe stability (instability: how often releases go wrong). Lead time, deployment frequency and recovery time are the throughput group in DORA's grouping; change fail rate and rework rate are the instability group. Version control means the shared history of code, such as Git. Each can be computed from two data sources you already have: a log of production deploys and a record of which deploys caused trouble.

The five measures and how to compute each

"Immediate intervention" means someone had to act at once after the deploy, for example a rollback or an emergency fix. The median is the middle value of a sorted list; the mean is the average.

Pick the denominator once (deploys, not commits or tickets) and write it down, otherwise two teams will report different numbers for the same behaviour.

Worked example (illustrative data, 30-day window)

• 60 production deploys, so deployment frequency is 60 / 30 = 2 per day.
• 6 of those 60 needed immediate intervention, so change fail rate = 6 / 60 = 10 percent.
• Of the 6, two were fixed by switching the previous version back on without a new deploy, and four by shipping a new fix deploy. Rework counts deployments, and only the four new fix deploys are unplanned deployments caused by an incident, so rework rate = 4 / 60 = about 6.7 percent. If your rollback is itself a fresh deploy, count those two as well and rework rate becomes 6 / 60 = 10 percent. The rule has to be written down either way.
• Recovery times for the 6 failures, in hours: 0.5, 1, 1.5, 2, 3, 5.5. The median is 1.75 hours; the mean is 2.25 hours. The median is the better headline because one long outage should not hide the typical case, but I also show the longest one.
• Lead time from raw timestamps: a change committed Monday 09:00 and deployed Friday 15:00 of the same week has lead time of 4 days 6 hours, about 4.25 days. Do that subtraction for every change, then take the median. For ten sampled changes, in days: 2, 3, 3, 4, 5, 6, 7, 9, 14, 40. The median is 5.5 days; the 40-day change is a flag to investigate, not noise to hide.

Healthy versus unhealthy readings

DORA's current pages do not publish fixed performance bands, so I judge by pattern and by trend against our own baseline, not against a borrowed benchmark:

• Frequent deploys, low and steady fail rate, short recovery: healthy. Small changes are flowing safely.
• Deployment frequency rising while fail rate rises too: speed is outrunning safety.
• Rare deploys with a low fail rate: often large batches and heavy approval; low failure may be luck over a small sample.
• Short lead time but long recovery: fast to ship, slow to fix, which usually points at weak monitoring or rollback.
• Rework rate climbing: the team is spending capacity repairing earlier releases.

Illustrative readings (invented numbers to show the shape, not benchmarks): a team at 2 deploys a day, 5 percent fail rate and a 1.75-hour median recovery is in good shape. A team at 1 deploy a week with a 40 percent fail rate and a 3-day median lead time has unstable releases, and a weekly deploy cadence points to large batches rather than small ones, so the instability is plausibly a batch-size problem. Separately, 6 of 60 failing (10 percent) would worry me more if it were climbing month on month.

Pitfalls

• Using any one metric as a target invites gaming (splitting trivial deploys to inflate frequency, or reclassifying incidents).
• Comparing teams: a platform team and a mobile app team have different release constraints. Compare each team with its own past.
• Small samples: 6 failures in a month is a rough signal; look at a longer window before reacting to a swing.

Direct answer

I would spend the first three weeks diagnosing with evidence from our own releases and interviews, so I fix the real driver and not the loudest complaint. Then I would make one change aimed at shared ownership of deployment outcomes, and measure it with a few shared numbers. The clash is rarely about personalities; it is usually about incentives (developers rewarded for shipping, operators for stability) and late handoffs.

Step 1: find the driver (weeks 1 to 3)

• Look at the last 10 to 15 blocked or failed releases and tag each with a cause: late notice to operations, unclear approval, flaky environment, missing runbook (the written steps for operating or recovering a service), untested change, or a real risk someone caught.
• Map one release end to end (a value stream map: each step, who does it, and where it waits). The longest waits are usually handoffs.
• Interview both sides separately, asking "what happens that makes you angry on release day?" and "what would you want from the other team?"
• Test hypotheses: incentives (who is paged, who is measured on what), handoff timing, ownership gaps (nobody owns the deployment config), or environment differences.

Step 2: first change (weeks 4 to 6)

I would choose the single change the evidence points to. If the pattern is "operations hears about it late and developers do not see the consequences", the first change is shared on-call ownership: developers share responsibility for the production effects of their changes, and an operations engineer joins design review (the meeting where a proposed design is critiqued before it is built) for risky changes. In practice: developers join the on-call rota, the rotating schedule of who is paged first when an alert fires, for the services they own. A typical start is one week in four as primary responder with an operations engineer as second line who coaches, and the page goes to the team that shipped the change, not to a central operations desk. Pick one change, not five, so the team can tell what worked.

Step 3: tell whether it is working (weeks 7 to 13)

Use shared metrics, owned jointly: change fail rate (the share of deployments needing immediate intervention), change lead time (commit to production), failed deployment recovery time (how long it takes to restore service after a deployment that failed and needed immediate intervention), plus the count of releases blocked in a week and a short two-question pulse survey (a brief, recurring poll, for example "I trust the other team's changes" for developers or "I trust the other team's operational readiness" for operators, each on a five-point scale). Compare to the baseline from step 1 and review at week 13.

Worked example (illustrative)

Metrics, using a month of data: 40 deployments, 6 needed a rollback or hotfix, so change fail rate = 6 / 40 = 15 percent. Lead time is commit-to-production time, taken as the median over the month.

Tagging 12 releases finds 7 blocked for late notice, 3 for environment differences, 2 for real risk. The first change then targets notice, delivered through the Step 2 mechanism: the operations engineer who joins design review for risky changes means releases above a risk threshold are known to operations at design time, not on release day, and the developers' on-call rota makes the cost of late notice visible to the people who cause it. That is still one change (shared ownership), with notice as the number that tells me whether it works. The baseline is 7 of 12, about 58 percent. Fewer than half as many means 3 or fewer per 12 releases (25 percent), so a good sign at week 13 is 3 or fewer blocked-for-notice releases per 12, with change fail rate not rising above its 15 percent baseline during the quarter and trending down toward a written target (for example 4 of 40 deployments, 10 percent, illustrative), because the brief is to cut deployment failures and a flat rate would not meet it. With about 40 deployments a month, 6 failures versus 4 is within normal month-to-month noise, so I would judge the trend over the quarter, not a single month. A blameless postmortem (a review of an incident that asks how the process allowed it, not who to blame) after each failed release feeds this.

What would change my call: if most causes were environmental flakiness, the first change would be an environment fix and I would involve a platform team (a group that builds shared tooling and environments for product teams) instead.

Direct answer

I would run shared delivery assets (build images, the prebuilt environments your CI jobs run in, and deployment templates, the reusable definitions teams use to ship) as an internal product with a named owner, versioned releases that consumers opt into, and approval effort that scales with risk. The goal is that a change breaks one canary consumer early, never seven teams at once, and that low-risk changes need no meeting.

Why the current setup failed

The shared asset was probably edited in place, with every consumer tracking "latest" (automatically using whatever the newest version is). One merge therefore changed everyone's build at the same moment, with no one checking consumers first. Governance should fix those two things: consumers pin versions, and changes are tested against real consumers before release.

The design

1. Ownership. A small owning team (or named maintainers) is accountable for each asset. A CODEOWNERS file (a repository file that lists who must review changes to which paths) makes this enforceable.
2. Versioning and pinning. Releases use semantic versioning (major.minor.patch, where a major bump signals a breaking change). Consumers pin an exact version (lock their build to, say, build-image:2.3.1 rather than following whatever is newest) and upgrade deliberately, with automated pull requests proposing the bump.
3. Tiered approval by risk.

4. What is tested. A consumer-contract suite (tests that encode what consumers rely on, so a change that would break them fails before release): a representative set of real pipelines (say one per language stack in use) built against the candidate version. If one breaks, the change does not ship. Teams contribute their pipeline as a test case in return for protection.
5. Avoiding the bottleneck. Service level targets for the owners (for example, a first response within one working day), automation for the patch tier, a published backlog, and a rule that consumers can fork (take their own copy of) the asset or override a pin for their own team without permission as long as they accept the maintenance. Track the owner's review queue; if patches wait, add maintainers.
6. Emergency path. A critical security fix can ship with one approver and be reviewed afterwards, with consumers notified.

Worked example: one change through the tiers

A maintainer wants to upgrade the Java runtime in the build image from 17.0.9 to 17.0.10 (a security fix). That is a patch: image version goes from 2.3.1 to 2.3.2, one maintainer approves once the shared suite is green, and automated pull requests propose the bump to the 7 consuming teams. Later, someone wants to drop Python 3.8 from the image. That is a major change: it becomes 3.0.0, needs a short written proposal, gets tried first by 2 canary teams, and both 2.x and 3.0.0 stay available for a stated period (say 8 weeks) so no team is forced to migrate in the middle of a release.

Trade-offs

• Pinning means consumers lag; set a maximum age (for example, no more than two minors behind) and automate the nudges.
• The contract suite costs upkeep; keep it small and owned.
• Tight control reduces breakage but invites shadow copies (private, unofficial duplicates of the asset that teams maintain themselves). If copies appear, treat that as a signal the asset is slow, not as a policy violation.

What would change my call: with only three or four consuming teams, I would skip the tiers and use one reviewer plus canary. Past about twenty, I would add a dedicated platform team. These team counts are judgement calls, not research findings: the real trigger is whether maintainer review queues and upgrade support are consuming more than a part-time rotation can give, which is the signal to watch rather than the head count.