DevOps Culture and Delivery Practices Questions
The principles and engineering culture behind modern software delivery: what DevOps means as a shared-ownership way of working and how to ease development and operations friction, adopting DevOps, infrastructure-as-code and GitOps practices across teams, the software development lifecycle and its methodology trade-offs (waterfall, agile, trunk-based development and branching agreements), engineering velocity and how it is measured without perverse incentives, development standards and how they are rolled out, enforced, excepted and justified with a business case, code review policy, turnaround expectations and handling bypasses, governance of shared delivery assets and changes to production automation, and the role of a platform team in enabling faster, safer delivery. Focuses on why teams work this way and how to drive adoption, not on how to build the machinery. Pipeline and build mechanics, deployment and rollback techniques, infrastructure tooling, reliability practice, and incident handling are covered elsewhere.
You are setting up code review for a team that has none. What goes into the policy (who must approve, what gets checked, how long reviews may take), and how do you keep it from turning into a bottleneck?
Sample Answer
Direct answer
Code review means a teammate other than the author reads a proposed change (a pull request, or PR) before it joins the shared codebase. For a team starting from nothing I would write a short, one-page policy with three parts: who approves, what is checked, and how fast reviews must start. Then I would put the speed controls in place from day one, because a review process with no time expectation turns into a queue within weeks.
Draft policy (the excerpt I would publish to the team)
| Topic | Rule |
|---|---|
| Who approves | One approval from a teammate who is not the author. Changes in sensitive areas (payments, authentication, database schema) also need approval from the named code owner for that area (the person or group officially responsible for that part of the code, listed in a file the hosting tool reads). Authors never approve their own change. |
| What reviewers check | Does it work and is it tested? Is the design understandable and in the right place? Would a new teammate be able to maintain it? Are there security or data-privacy implications? |
| What reviewers do not check by hand | Formatting and style: a formatter (a tool that rewrites code into one standard layout) and a linter (a tool that flags likely mistakes and rule violations) run automatically, so humans spend time on design and correctness. |
| Approval standard | Approve once the change improves the overall health of the codebase, even if it is not how the reviewer would have written it. Style preferences are marked as optional comments ("nit", short for nitpick, meaning a minor point the author may ignore). In plain terms: if the change makes the code better than before and breaks nothing, do not hold it up over taste. This follows Google's published engineering practices. |
| Speed | First response within one business day (Google's guidance is that one business day is the maximum). First response means a real read, not a rubber stamp, and it is not the same as final approval. |
| Size | Aim for changes a reviewer can understand in one sitting; split larger work into steps that each leave the code working. |
Keeping review from becoming a bottleneck
- Automate the mechanical checks (tests, formatting, linting) so a human is never the one catching a missing semicolon.
- Spread the load: rotate or auto-assign reviewers instead of letting the most senior person become the gate.
- Small changes: a short change is reviewed faster and more carefully than a large one.
- Make the wait visible: a weekly look at how long PRs sit before a first response, treated as a team problem, not a ranking of individuals.
- Allow approval with minor follow-ups: a reviewer who trusts the author can approve while leaving small comments to be fixed.
Worked example (illustrative)
A team of 6 engineers each opens about one PR a day, so there are about 6 reviews to do per day. With 5 possible reviewers per PR (everyone except the author) that is 6 / 5 = 1.2 reviews each per day. That is manageable only if changes are small and reviews are shared evenly; if one lead reviews everything, that person faces 6 per day and becomes the bottleneck. In hours, at about 30 minutes per review: 1.2 reviews is 36 minutes a day each, but 6 reviews is 3 hours a day for the lead, which is why work starts to queue.
Pitfalls
- Two required approvers everywhere sounds safer but doubles the waiting for little added protection on routine changes.
- Using review turnaround to rank people encourages hasty approvals.
- A policy nobody can see enforced (no branch protection, the repository setting that blocks merging until the required approvals are in) lasts only until the first deadline.
Your org currently deploys by having CI push to production, and someone proposes pull-based GitOps. What changes in ownership, audit and who can change production, and is the move worth it for a 200-engineer company?
Sample Answer
Direct answer
I would say the move is worth it for a 200-engineer company if most workloads run on Kubernetes (a system that runs applications packaged as containers, which are self-contained bundles of a program and what it needs to run, across many machines) and audit or access control matters, and I would roll it out as a staged pilot, not a big bang. GitOps means the desired state of production lives in a Git repository, and an agent inside the environment (Argo CD and Flux are the common open-source ones) continuously pulls that state and makes reality match it. That matching step is called syncing or reconciling: the agent compares what Git says should run with what is running, and fixes any difference. Today CI pushes into production, so CI holds production credentials. In the pull model CI only builds and proposes a change; production changes itself.
What changes
| CI pushes to production (today) | Pull-based GitOps | |
|---|---|---|
| Who can change production | Anyone with CI permissions or the pipeline credentials | Anyone whose change is merged to the config repo branch, plus a small break-glass group (a few named people allowed to change production directly in an emergency, with every use logged and reviewed afterwards) |
| Credentials | CI holds production access | The agent holds access from inside the environment; CI needs no production credentials (it still needs write access to the config repository) |
| Audit | Pipeline logs, which can be scattered or expire | Git history: who proposed, who approved, when |
| Drift (production differing from what was intended) | Silent until someone notices | Detected; reverted automatically only if self-healing (automatic correction of drift) is switched on, otherwise flagged for a person |
| Rollback | Re-run an older pipeline | Revert a commit |
| Ownership | The pipeline owner owns deploys | Service teams own their config; a platform team owns the agent and conventions |
One caveat on the credentials row: it describes the common in-cluster model (Flux, or Argo CD installed in each cluster). Argo CD can also run as one central instance that holds credentials for many remote clusters; that setup still removes production access from CI, but the central instance becomes a high-value credential store, so it needs the same protection CI used to need.
Is it worth it at 200 engineers?
- In favour: removing production credentials from CI is a real security gain, review of config becomes the control, and audit evidence falls out of Git rather than being assembled by hand for each audit.
- Against: it adds a layer to learn and operate; you need a decision about where config lives (usually a separate repo from application code); secrets (passwords and API keys, for example a database password) need a separate solution because they must not sit in Git in plain text, and Git history is permanent; "merged but not yet synced" is a new state engineers must understand; and if you mostly run virtual machines or serverless functions, the tooling fit is weaker.
My recommendation
Yes if the Kubernetes share is high and there is audit pressure. Run a pilot with two or three willing teams for one quarter. Judge it by whether those teams' change lead time stays about the same, whether manual production changes drop, and whether the platform team can support it without becoming the new gate. If fewer than roughly a third of services are on Kubernetes, I would stay with push and tighten credentials instead; between a third and a half, the audit pressure decides, and above half I would adopt it. These thresholds are judgement calls, not research findings. The reasoning: the agent only manages Kubernetes workloads, so the platform team runs two deployment systems whatever the share, and the question is whether the credential and audit gain justifies running the second one. Below about a third that gain covers too few services to repay the operating cost; above half the pull model becomes the main path and push the exception. Two or three teams for a quarter is enough to see several full release cycles and a few incidents while limiting the damage if the conventions turn out wrong.
What would change my call: a compliance mandate for an audit trail (accelerates), or a team of five platform engineers already stretched (delays).
Many teams depend on shared build images and shared deployment templates, and one team's change just broke several others. Design the governance for changes to shared delivery assets: who approves, what is tested, and how you keep it from becoming a gatekeeping bottleneck.
Sample Answer
Direct answer
I would run shared delivery assets (build images, the prebuilt environments your CI jobs run in, and deployment templates, the reusable definitions teams use to ship) as an internal product with a named owner, versioned releases that consumers opt into, and approval effort that scales with risk. The goal is that a change breaks one canary consumer early, never seven teams at once, and that low-risk changes need no meeting.
Why the current setup failed
The shared asset was probably edited in place, with every consumer tracking "latest" (automatically using whatever the newest version is). One merge therefore changed everyone's build at the same moment, with no one checking consumers first. Governance should fix those two things: consumers pin versions, and changes are tested against real consumers before release.
The design
- Ownership. A small owning team (or named maintainers) is accountable for each asset. A CODEOWNERS file (a repository file that lists who must review changes to which paths) makes this enforceable.
- Versioning and pinning. Releases use semantic versioning (major.minor.patch, where a major bump signals a breaking change). Consumers pin an exact version (lock their build to, say,
build-image:2.3.1rather than following whatever is newest) and upgrade deliberately, with automated pull requests proposing the bump. - Tiered approval by risk.
| Change tier | Example | Approval | Test |
|---|---|---|---|
| Patch | Security fix, no behaviour change | One maintainer, merge when green | Shared test suite |
| Minor | New optional tool in the image | One maintainer plus owning-team review | Suite plus canary consumers (a few volunteer teams that try it first) |
| Major | Removes or renames something | Short written proposal, sign-off from consumer representatives | Suite, canary, and a deprecation window (a fixed period in which the old version still works and is marked for removal) with both versions available |
- What is tested. A consumer-contract suite (tests that encode what consumers rely on, so a change that would break them fails before release): a representative set of real pipelines (say one per language stack in use) built against the candidate version. If one breaks, the change does not ship. Teams contribute their pipeline as a test case in return for protection.
- Avoiding the bottleneck. Service level targets for the owners (for example, a first response within one working day), automation for the patch tier, a published backlog, and a rule that consumers can fork (take their own copy of) the asset or override a pin for their own team without permission as long as they accept the maintenance. Track the owner's review queue; if patches wait, add maintainers.
- Emergency path. A critical security fix can ship with one approver and be reviewed afterwards, with consumers notified.
Worked example: one change through the tiers
A maintainer wants to upgrade the Java runtime in the build image from 17.0.9 to 17.0.10 (a security fix). That is a patch: image version goes from 2.3.1 to 2.3.2, one maintainer approves once the shared suite is green, and automated pull requests propose the bump to the 7 consuming teams. Later, someone wants to drop Python 3.8 from the image. That is a major change: it becomes 3.0.0, needs a short written proposal, gets tried first by 2 canary teams, and both 2.x and 3.0.0 stay available for a stated period (say 8 weeks) so no team is forced to migrate in the middle of a release.
Trade-offs
- Pinning means consumers lag; set a maximum age (for example, no more than two minors behind) and automate the nudges.
- The contract suite costs upkeep; keep it small and owned.
- Tight control reduces breakage but invites shadow copies (private, unofficial duplicates of the asset that teams maintain themselves). If copies appear, treat that as a signal the asset is slow, not as a policy violation.
What would change my call: with only three or four consuming teams, I would skip the tiers and use one reviewer plus canary. Past about twenty, I would add a dedicated platform team. These team counts are judgement calls, not research findings: the real trigger is whether maintainer review queues and upgrade support are consuming more than a part-time rotation can give, which is the signal to watch rather than the head count.
Tell me about a time you led adoption of a new engineering standard or practice (for example mandatory PR checks, linting or a deployment process). What resistance did you meet and how did it turn out?
Sample Answer
Situation. I was an engineering manager for a team of 12 shipping a web service. Over one quarter, 5 of our 40 releases (12.5%) needed a hotfix, and the common thread was changes merged without a second pair of eyes or any automated checks. I proposed required pull request checks: a pull request (PR) is a proposed change; required checks are automated tests (and, in the original proposal, linting) that must pass, plus one approving review, before the change can merge. In the end only the tests and one review became blocking; linting stayed advisory, as described below. (The numbers here are illustrative; use your own in an interview.)
Resistance I met
- Senior engineers: "this slows us down and implies we are not trusted."
- A tech lead worried flaky tests would block urgent fixes.
- Two engineers said linting rules were a matter of taste.
What I did
- Gathered evidence first: listed the five hotfixes and what check would have caught each; three of five would have been caught by tests that already existed but were not run automatically.
- Co-designed the rule set with the tech lead and one skeptic, so they owned it.
- Started advisory for three weeks, fixed the two flakiest tests, and set a review response target the team chose.
- Added a hotfix lane (one approver, review afterwards) to answer the "urgent fix" objection.
- Turned on blocking for tests and one review only; linting stayed advisory until the team agreed a style guide.
Result. In the next quarter 2 of 44 releases (about 4.5%) needed a hotfix, and median time to merge rose by a few hours at first (judged against a before-figure I could only reconstruct roughly from PR timestamps, since I had not measured it) and recovered after we fixed review response time. With counts this small (5 hotfixes, then 2), the drop is encouraging rather than proof; I told the team so and kept tracking it for another quarter. The skeptics later proposed two additional checks themselves.
What I would do differently. I would have measured merge time before the rollout, not afterwards, so I could show the cost honestly instead of estimating it after the fact. I would also have fixed the flaky tests before the advisory phase, since they cost the most goodwill.
Leadership asks you to double deployment frequency in two quarters without raising the change-failure rate. What would you measure first, what organizational and technical changes would you propose, and how would you know you were improving rather than gaming the numbers?
Sample Answer
Direct answer
I would start by measuring, not changing anything: a baseline of deployment frequency, change fail rate, change lead time (commit to production) and recovery time (how long it takes to restore service after a failed deploy), plus a breakdown of where a change spends its time. Then I would shrink batch size and remove waiting, because both frequency and safety improve when changes are small. Crucially, "do not raise the change-fail rate" is a weaker promise than it sounds, so I would raise that with leadership at the start.
The catch hidden in the goal
Illustrative baseline: 60 deploys a month, 10 percent fail, so 6 failed deploys. Double the frequency to 120 deploys and hold the rate at 10 percent: 120 x 0.10 = 12 failed deploys a month. The rate is unchanged, but customers see twice as many failures. To keep the absolute number of bad deploys at 6, the rate must fall to 6 / 120 = 5 percent. I would propose a goal of doubling frequency while reducing fail rate or at least holding recovery time down, and say why.
Step 1: measure first
- The four numbers above, with agreed definitions of "deploy" and "failure".
- A timeline of a typical change: waiting for review, waiting for the build, waiting for an approval board or test environment, waiting for a release slot. The longest wait is usually the first target.
Step 2: organisational changes
- A reviewing norm: first response within a business day, so changes are not stuck in a queue.
- Replace slow change approval boards (committees that approve each release) for low-risk changes with pre-agreed rules; keep human approval for genuinely risky ones.
- Give teams ownership of their own release, including the pager (being on call, the first to be alerted when it breaks), so they feel the cost of a bad release.
- Agree that a failed deploy leads to a blameless review (a post-incident discussion about how the process allowed the failure, not who to blame), so failures are reported.
Step 3: technical changes
- Smaller changes merged to a shared main branch daily (trunk-based development), with unfinished features behind feature flags (switches that hide unfinished code until it is turned on).
- Automated tests and builds fast enough to run on every merge.
- Staged rollouts (sending a change to a small share of users first, then more) and the ability to switch a change off quickly, which keep each failure cheap. Safer releases make a higher frequency tolerable: a staged rollout limits how many users a bad change reaches, and a quick switch-off or rollback shortens how long they are affected, so each failure stays cheap even when there are more deploys.
How I would know it is real improvement, not gaming
- Count rules fixed in advance: a deploy is a change to production code; empty or configuration-only deploys and artificial splitting do not count unless stated.
- Watch the pair: frequency rising while change size distribution, lead time, and absolute failures behave sensibly.
- Cross-check against things people cannot easily fake: customer-visible incidents, support tickets, and a developer survey.
- Review the numbers with the teams, so a jump in frequency prompts the question 'what changed in how you work?'
Sequence over the two quarters
- Weeks 1 to 3: baseline and the wait-time timeline, plus agreeing the counting rules.
- Weeks 4 to 8: the first action is the review norm and replacing approval-board steps for low-risk changes, because they are cheap and remove the longest waits.
- Weeks 6 to 18: faster automated tests, trunk-based development and feature flags, team by team.
- Weeks 14 to 26: staged rollouts and ownership of release and pager, then re-read the baseline at week 26.
Trade-offs: a two-quarter horizon is tight for cultural change; I would sequence the quick wins (review latency, approvals) first and the testing investment alongside.
Unlock Full Question Bank
Get access to all 22 DevOps Culture and Delivery Practices interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.