Role Responsibilities and Success Criteria Questions

Owning a component means acting like its long-term steward, not just its most recent contributor: you know its weak points, you're the first call when it breaks, and you're driving what happens to it next rather than waiting to be told.

What ownership looks like across five areas

• Code quality: maintain a testing bar for the component, actively drive down a known recurring source of bugs, and keep its architecture documented well enough that someone else could reason about a change without asking you first.
• Documentation: maintain a runbook (a step-by-step guide for operating and debugging the component) and an architecture decision record, or ADR (a short document capturing why a significant design choice was made), so knowledge doesn't live only in your head.
• Incident response: be the primary escalation point when it breaks, write a blameless postmortem (an incident review focused on fixing the system rather than blaming the person) after any real incident, and track whether the same class of incident recurs afterward.
• Roadmap planning: proactively flag technical debt or scaling risk before it becomes an emergency, and propose, not just execute, the next quarter's improvements to the component.
• Mentoring: bring at least one other contributor up to speed on the component so it isn't a bus-factor-of-one situation, where only one person understands it well enough to keep it running.

Success metrics to track the impact

A downward trend in repeat incidents for that specific component over time, how long it takes to onboard a second contributor to real changes, PR review turnaround time within the component, and a qualitative signal like fewer escalations bypassing you straight to your manager.

Worked example

Owning a checkout service in a given quarter: write the runbook that didn't exist, add an alert for the exact failure mode that caused the last outage, mentor a teammate through their first on-call shift on the service, and get a scaling proposal you authored onto next quarter's roadmap instead of waiting for someone else to notice the risk.

Trade-offs and pitfalls

Ownership without documentation or mentoring creates a single point of failure (SPOF): if you're the only one who understands the component, you haven't reduced organizational risk, you've just centralized it in yourself. There's also a trade-off against over-scoping ownership into empire-building; a component is still one part of a larger system, and treating every adjacent decision as yours to control slows the team down rather than protecting it.

Success for an SRE (site reliability engineer) supporting AI products means picking a small set of measurable indicators that both engineering and product genuinely trust as the shared truth about system health, then being accountable for keeping them within agreed targets and being the first responder when they slip. The role is not measured by "zero incidents," it is measured by whether availability, correctness, and performance stay trustworthy and visible to both sides at once.

Three SLIs (service-level indicators, meaning measurable signals of how well a system is doing at something)

Availability, measured as request success rate: the percentage of inference requests that return a valid response, not a timeout, server error, or client-visible failure, within the agreed timeout window. Measure it from the API gateway or load balancer logs as successful responses divided by total requests, tracked over a rolling five-minute window for alerting and a rolling thirty-day window for reporting. One detail that is easy to get wrong and quietly corrupts this number: the canary probes described next are replayed against the same serving path, so unless they are tagged at the gateway and filtered out, they land in this denominator and you end up reporting a success rate partly made of requests no user ever sent. Probe traffic is usually healthier and more uniform than real traffic, so it flatters the number rather than harming it, which is why nobody notices. Tag it with a header at the gateway, exclude it from both the availability and the latency SLIs, and report it as its own separate series. It matters to engineering because it isolates infrastructure health from model behavior, and it matters to product because a user who gets no response at all is the fastest way to lose trust, independent of how good the model is.

Correctness, measured as a guardrail and validation pass rate, used as a proxy since there is rarely a live ground truth for open-ended AI output: the percentage of responses that pass automated checks, schema validation, safety and guardrail filters, and agreement with a small fixed set of canary prompts, meaning test inputs whose correct answer is already known (the term borrows from the canary used in coal mines as an early-warning signal), replayed continuously against the live serving path so a silent regression shows up immediately. Measure it by running that canary set on a fixed schedule and logging pass or fail, plus periodically sampling real traffic for human or automated review. It matters to engineering because it catches silent model or prompt regressions a plain uptime check would miss entirely, and it matters to product because a service that is up but confidently wrong is often worse for user trust than a visible outage, since users have no obvious signal to distrust it.

Performance, measured as end-to-end p95 latency (the time within which 95 percent of requests complete) and time-to-first-token for streaming responses: instrumented through distributed tracing at the gateway and the inference service, aggregated per model version and per traffic segment, with the same synthetic-traffic exclusion applied, since a fixed canary prompt set has near-constant input length and therefore near-constant latency and will compress the tail of any percentile it is mixed into. It matters to engineering because latency regressions are often an early warning of a deeper problem before a full outage shows up, and it matters to product because response time directly shapes perceived quality and abandonment for interactive AI features.

Worked example

Picture a weekly dashboard for a customer-support AI assistant, reviewed jointly by the on-call SRE and the product manager: request success rate sits at 99.6 percent against a 99.5 percent target, guardrail pass rate drops from 98 percent to 91 percent right after a prompt change ships, and p95 latency holds steady at 1.8 seconds. Because the drop shows up specifically in the correctness indicator and nowhere else, with availability and latency both unmoved, the SRE pages engineering, and engineering and product then jointly decide whether to roll back the prompt change. That is exactly the kind of call a single blended "AI health" number would have hidden: blended across the three indicators, a 7-point drop in one of them barely moves the aggregate, and the aggregate would not have told anyone which change to reverse.

Trade-offs and pitfalls

Do not confuse an SLI, the raw measured signal, with an SLO (service-level objective), the target threshold set on top of it: a team that reports the number without ever agreeing on a target has no way to say whether it is good or bad. Resist collapsing all three indicators into one blended score for a leadership dashboard, since that hides which pillar is actually failing and takes away both engineering's and product's ability to act on the right one. And correctness proxies like a canary set will drift out of sync with real user-perceived quality as usage patterns shift, so they need periodic re-validation against fresh human review rather than being frozen at launch and trusted forever. Budget the canary set deliberately too: replaying it continuously against the live path costs real inference spend on every run and grows linearly with both set size and replay frequency, so the honest trade is a small set replayed often enough to catch a regression within one deploy cycle, not the largest set you can assemble.

Direct answer

Before the call, do the homework a genuinely curious, well-prepared candidate would do: read what the team has published publicly, read what others have said about working there, and bring specific, current signals to the conversation instead of questions you could have answered yourself with 30 minutes of research.

Public sources to consult

The company's engineering blog, since its topics and frequency reveal what the team is proud of or currently building; public repositories if any exist, for commit cadence and recent activity; the job posting itself, read closely for named tools and responsibilities rather than just the title; review sites like Glassdoor or Blind for team-specific commentary, treated as directional hypotheses rather than fact; and the product itself, used firsthand for rough edges or recent changes.

Harder-to-reach sources

LinkedIn profiles of current team members, where tenure patterns (a burst of very recent joiners, or a lot of very long tenure) both tell you something different; any mutual connections who might give an informal read; and the interviewer's own public talks or writing, if they have any.

Signals to look for

Commit cadence and recency on public repositories, to judge whether something is actively maintained or legacy; recurring themes in open issues, which hint at known pain points; incident history if publicly documented, like a status page's past record; product metrics you can observe externally, like app store ratings and review themes; the pattern of recent job postings, since a burst of hiring for one specialty suggests a recent gap or new initiative; and engineering blog posts, since a post about a migration tells you exactly what's currently being rebuilt.

How to prioritize what to bring up

Favor things that are recent (shows fresh research, not a five-year-old post), specific to this team rather than generic to the company, and genuinely uncertain to you. Save your best, most specific question for something public information alone couldn't resolve, that's what the live conversation is actually for, rather than asking something the job posting already answered.

Worked example

If the engineering blog has a post from two months ago about migrating to a new queueing system, and the job posting separately mentions "own reliability of the payments pipeline," I'd connect the two and ask something like "how has the queueing migration affected the reliability work this role owns, is that mostly finished or still in progress." That shows I read both artifacts and connected them, rather than asking a generic "what's a typical day like" question.

Trade-offs and pitfalls

Over-researching to the point of asking overly narrow or gotcha-sounding questions can come across as trying to catch the team out rather than being genuinely curious. The other pitfall is treating review-site commentary as fully reliable, it's useful for hypotheses to verify live, not conclusions to state as fact in the interview.

Five 90-day metrics should be concrete, cheap to measure with data you already have or can get quickly, and tied to something the team actually cares about, not vanity numbers. I use the SMART framework (Specific, Measurable, Achievable, Relevant, Time-bound) as the filter for whether a proposed metric is real or just an aspiration. Below is the set I would commit to, each with why it matters, the data source and method, and a target range. The ranges are opening positions stated on the assumption of an average existing baseline; I would confirm or renegotiate each one in week two or three against the team's real numbers, and I say that out loud rather than pretending day-one precision I do not have.

1. Shipped, reviewed deliverables tied to the team's top objective. Why it matters: it proves I can navigate the codebase, the review culture, and the release process, which is the precondition for everything else. Source and method: count merged changes in version control, filtered to the objective's tracking label, excluding trivial changes. Target: at least two substantive changes merged by day 45 and six to eight by day 90, and every one of them reviewed by someone other than my onboarding buddy.

2. One validated impact result, with its unit cost. Why it matters: output that nobody can attribute is indistinguishable from busywork. Source and method: an A/B test (a controlled comparison against a held-out group that does not get the change) read from the experimentation platform, paired with the serving or query cost per unit pulled from the cloud billing export. Target: one experiment designed, shipped and read out inside 90 days, with the result reported honestly whether it won or lost, and unit cost held at or below the agreed ceiling rather than traded away for the win.

3. A reliability improvement on something a downstream consumer depends on. Why it matters: the pipeline or service running is not the point, the consumer trusting it is. Source and method: the scheduler or monitoring tool's run history for failure and late-delivery rate over a rolling 30 days, confirmed with the two or three teams that consume the output. Target: cut the failure-plus-late rate on one named dependency by roughly half against its trailing 30-day baseline, and get that confirmed by a consumer in their own words, not just on the dashboard.

4. Finished work with a quality bar attached. Why it matters: counting work in progress is how a quarter disappears with nothing landed. Source and method: the team's tracker, counting only items that reached production and carry a documented accuracy, adoption, or acceptance check. Source of the quality number is whatever the team already records; if it records nothing, establishing that record is itself part of the metric. Target: two to three finished items by day 90, none of them still labelled in review.

5. Adoption of one thing I built, by a named group. Why it matters: a dashboard or tool nobody opens is an output metric pretending to be an impact metric. Source and method: the tool's own usage log for weekly active viewers in the target group, plus a short qualitative check with three of those users on whether it changed a decision. Target: a majority of the named target group using it weekly by day 90, and at least one concrete decision someone can point to. I would avoid leaning on a Net Promoter Score here: it is a loyalty measure built from a single likelihood-to-recommend question scored zero to ten, it needs far more respondents than a ten-person stakeholder group to be stable, and weekly active use plus three honest conversations tells you more at this size.

How the same five slots look in different roles

The slots are deliberately generic because they travel. Slot 1 is a Software Engineer's "did specific, verifiable work land." Slot 2 is the AI Engineer flavor: impact plus inference cost per prediction, not accuracy alone. Slot 3 is the Data Engineer's consumer-benefit framing. Slot 4 is the Data Scientist's way of counting finished models or analyses rather than work in progress. Slot 5 is the BI Analyst's dashboard-adoption framing. A Solutions Architect's version blends slots 3 and 5: pick a specific customer or internal team consuming your architecture recommendations, and track whether that team actually adopted the recommended design, not just whether you delivered a document, confirmed directly with them.

Trade-offs and pitfalls

Committing to a hard numeric target before you have seen the team's real baseline is a common mistake, which is why every range above is explicitly an opening position to be reset in week two or three. The opposite mistake is worse in an interview: refusing to name any number at all reads as hedging, so state the range and state what would move it. A subtler pitfall is picking five metrics that are all about your own output (things you did) with none about downstream impact (things that changed because of what you did); slots 1 and 4 are output, slots 2, 3 and 5 are impact, and a strong set always has at least one of each.

Direct answer

Ask to see the artifacts a team actually uses to run itself, not just a description of its service-level agreements (SLAs, formal commitments about uptime, latency, or accuracy) and service-level objectives (SLOs, the internal targets a team holds itself to, which may be stricter than the external SLA). A team with real operational maturity can produce a dashboard and a runbook on request; a team that's only aspirationally following its targets will describe numbers it can't actually show you.

Dashboards to request

"Could you show me, or describe, the dashboard you look at to decide if the service is healthy right now?" A specific answer naming actual metrics, like 99th-percentile (p99, the value below which 99% of measurements fall) latency or error rate, is a good sign; a vague "we just check if things are broken" is a warning sign.

Runbooks to request

"What does the runbook for your most common alert look like?" Ask to see an actual on-call runbook, or at least hear a specific example of a step-by-step response to a real past incident.

On-call interviews

Ask to speak with, or ask about, whoever is currently on-call: "What's the last page you got, and what did resolving it involve?" "How often does the team get paged outside business hours, and how has that trended?"

Signals the targets are realistic, monitored, and owned

Realistic: the target is specific and has a rationale that traces to something outside the team, like "our client SLA is 300 milliseconds, so we hold an internal SLO of 250 to leave ourselves room to react before we breach the contract," not a round number nobody can justify. Listen specifically for that gap between the two numbers. A team whose internal objective is set exactly at its external commitment learns it has a problem at the same moment its customer does, which defeats the reason the objective and the agreement are two separate numbers in the first place, so hearing an SLO quoted as identical to the SLA is a sign the distinction is nominal. Monitored: the team can show you a live number against the target right now, not just describe it from memory. Owned: there's a named person or rotation responsible when the target is breached, and the team can describe what happens next, who gets paged and what the escalation path is, rather than "someone probably notices eventually."

Worked example

If I ask what the current p99 latency target is and where the team stands against it, and the interviewer immediately pulls up a number and says "target's 250 milliseconds, we've averaged 230 this month, breached it twice after a deploy last week and rolled both back," that's strong evidence of a real, owned SLO. If instead they say "I think we have a target somewhere, let me check with someone," the target exists on paper but isn't operationally alive.

Trade-offs and pitfalls

Pushing too hard on this in an early screening interview can feel adversarial, so calibrate depth to the interview stage and to whether you're talking to the hiring manager or a future teammate. The bigger pitfall is taking a confident verbal answer at face value without asking for the artifact behind it, confidence isn't the same as evidence.